论文

RA-MMEA:先评估图像可靠性,再做多模态实体对齐

Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment

精选理由

RA-MMEA 先评估图像可靠性再融合,替代直接拼图片的做法,专治图像噪声拖累实体对齐,代码将开源。

针对多模态实体对齐(MMEA)中图像直接融合会引入噪声、与实体语义错位的问题,论文提出可靠性感知框架 RA-MMEA。框架包含两个模块:DA-VRP 利用实体内多模态依赖估计图像可靠性,SR-VEG 以文本模态编码的语义为条件生成替代视觉表示用于融合。实验显示 RA-MMEA 在该任务上取得 state-of-the-art 结果,代码和结果将开源。

原文 · arXiv cs.AI

Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment

The visual modality, i.e., images, plays a key role in multi-modal entity alignment (MMEA). Existing approaches often directly fuse the image with other modalities to align different entities. Although simple, such strategies overlook the potential noise in the images and their semantic misalignment with corresponding entities, resulting in suboptimal fusion and degraded performance. Addressing this, we propose a novel Reliability-Aware framework for MMEA (RA-MMEA), which assesses visual reliability and adaptively improves unreliable visual representations for robust entity alignment. The core lies in two modules, including dependency-aware visual reliability prediction (DA-VRP) and stability-regularized visual embedding generation (SR-VEG). The former aims to estimate the reliability of an image by leveraging multi-modal dependency within the entity, while the latter focuses on producing alternative visual representation conditioned on semantics encoded in textual modalities for multi-modal fusion. Compared to current methods, RA-MMEA enables more reliable visual representations for modality fusion, thereby improving performance. In extensive experiments, RA-MMEA achieves state-of-the-art results, verifying the importance of reliable visual modality for entity alignment and the effectiveness of RA-MMEA. The code and results will be released.