Multimodal Named Entity Recognition (MNER) is an important research direction in Natural Language Processing, which aims to enhance Named Entity Recognition (NER) performance via additional image information. As posts with multiple images become increasingly common on social media, existing methods primarily focus on single-image scenarios, leaving a significant research gap for multi-image MNER. Current multi-image approaches often fuse holistic image representations with text, a strategy susceptible to noise from irrelevant visual information and which overlooks fine-grained, object-level cues crucial for entity recognition. To address this challenge, we propose a novel Object-level Semantic Alignment Framework. The framework utilizes an object detector to decompose the multiple images into a set of fine-grained candidate visual objects. A semantic alignment module then calculates the semantic relevance of each object to the text, based on which it selects the Top-K most critical visual proofs from the candidates, which are then fed into a multimodal interaction module for deep fusion with the text. Extensive experiments on public multi-image MNER datasets demonstrate that our proposed method significantly outperforms existing baselines. Ablation studies further validate the effectiveness of our object alignment and selection mechanism, providing a novel solution for the MNER-MI task.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Focus on What Matters: Object-Level Semantic Alignment for Multimodal Named Entity Recognition with Multiple Images

  • Peng Fan,
  • Yanli Jin,
  • Yunyu Zhang,
  • Peng Liu,
  • Xianxian Li

摘要

Multimodal Named Entity Recognition (MNER) is an important research direction in Natural Language Processing, which aims to enhance Named Entity Recognition (NER) performance via additional image information. As posts with multiple images become increasingly common on social media, existing methods primarily focus on single-image scenarios, leaving a significant research gap for multi-image MNER. Current multi-image approaches often fuse holistic image representations with text, a strategy susceptible to noise from irrelevant visual information and which overlooks fine-grained, object-level cues crucial for entity recognition. To address this challenge, we propose a novel Object-level Semantic Alignment Framework. The framework utilizes an object detector to decompose the multiple images into a set of fine-grained candidate visual objects. A semantic alignment module then calculates the semantic relevance of each object to the text, based on which it selects the Top-K most critical visual proofs from the candidates, which are then fed into a multimodal interaction module for deep fusion with the text. Extensive experiments on public multi-image MNER datasets demonstrate that our proposed method significantly outperforms existing baselines. Ablation studies further validate the effectiveness of our object alignment and selection mechanism, providing a novel solution for the MNER-MI task.