Multi-modal Siamese Network for Entity Alignment
The booming of multi-modal knowledge graphs (MMKGs) has raised the imperative demand for multi-modal entity alignment techniques, which facilitate the integration of multiple MMKGs from separate data sources. Unfortunately, prior arts harness multi-modal knowledge only via the heuristic merging of uni-modal feature embeddings. Therefore, inter-modal cues concealed in multi-modal knowledge could be largely ignored. To deal with that problem, in this paper, we propose a novel Multi-modal Siamese Network for Entity Alignment (MSNEA) to align entities in different MMKGs, in which multi-modal knowledge could be comprehensively leveraged by the exploitation of inter-modal effect. Specifically, we first devise a multi-modal knowledge embedding module to extract visual, relational, and attribute features of entities to generate holistic entity representations for distinct MMKGs. During this procedure, we employ inter-modal enhancement mechanisms to integrate visual features to guide relational feature learning and adaptively assign attention weights to capture valuable attributes for alignment. Afterwards, we design a multi-modal contrastive learning module to achieve inter-modal enhancement fusion with avoiding the overwhelming impact of weak modalities. Experimental results on two public datasets demonstrate that our proposed MSNEA provides state-of-the-art performance with a large margin compared with competitive baselines.
PDFCode
FB15k
DBpedia
UMVM
Results from the Paper
Ranked #7 on
Multi-modal Entity Alignment
on UMVM-oea-d-w-v1
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Multi-modal Entity Alignment | UMVM-dbp-fr-en | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.557 | # 10 | ||
| Multi-modal Entity Alignment | UMVM-dbp-fr-en | MSNEA (w/o surf) | Hits@1 | 0.583 | # 9 | ||
| Multi-modal Entity Alignment | UMVM-dbp-ja-en | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.541 | # 10 | ||
| Multi-modal Entity Alignment | UMVM-dbp-ja-en | MSNEA (w/o surf) | Hits@1 | 0.557 | # 9 | ||
| Multi-modal Entity Alignment | UMVM-dbp-zh-en | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.609 | # 10 | ||
| Multi-modal Entity Alignment | UMVM-dbp-zh-en | MSNEA (w/o surf) | Hits@1 | 0.648 | # 9 | ||
| Multi-modal Entity Alignment | UMVM-oea-d-w-v1 | MSNEA (w/o surf) | Hits@1 | 0.809 | # 7 | ||
| Multi-modal Entity Alignment | UMVM-oea-d-w-v1 | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.800 | # 8 | ||
| Multi-modal Entity Alignment | UMVM-oea-d-w-v2 | MSNEA (w/o surf) | Hits@1 | 0.862 | # 7 | ||
| Multi-modal Entity Alignment | UMVM-oea-en-de | MSNEA (w/o surf) | Hits@1 | 0.788 | # 7 | ||
| Multi-modal Entity Alignment | UMVM-oea-en-de | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.753 | # 8 | ||
| Multi-modal Entity Alignment | UMVM-oea-en-fr | MSNEA (w/o surf) | Hits@1 | 0.699 | # 7 | ||
| Multi-modal Entity Alignment | UMVM-oea-en-fr | MSNEA (w/o surf & w/o iter) | Hits@1 | 0.692 | # 8 |
