<p>Grounded multimodal named entity recognition (GMNER) extends multimodal named entity recognition (MNER) by linking textual entities with corresponding visual regions. Existing approaches face two main challenges: (1) they often focus on either global or local visual features, where the former misses fine-grained correspondences and the latter overlooks broader contextual cues; (2) current methods lack unified and semantically aligned representations, limiting cross-modal consistency. To address these issues, we propose a multi-granularity cross-modal alignment framework (MGCAF), which leverages global and local visual features along with textual cues from images, enabling multi-granularity cross-modal alignment and interactions between text and images. Specifically, we introduce a multi-granularity visual feature fusion (MVF) module to balance and integrate global and local visual information and a multi-granularity cross-modal alignment (MCA) module to align multi-granularity visual and textual representations. Notably, GMNER involves large-scale multimodal processing and dense cross-modal interactions, and the proposed MGCAF is inherently suited for parallel and distributed computation, enabling efficient training and scalable inference. Experimental results demonstrate that MGCAF improves both entity recognition and visual grounding, validating its effectiveness for the GMNER task.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MGCAF: a multi-granularity cross-modal alignment framework for grounded multimodal named entity recognition

  • Xiaojia Wu,
  • Lingfeng Liu,
  • Lin Cheng

摘要

Grounded multimodal named entity recognition (GMNER) extends multimodal named entity recognition (MNER) by linking textual entities with corresponding visual regions. Existing approaches face two main challenges: (1) they often focus on either global or local visual features, where the former misses fine-grained correspondences and the latter overlooks broader contextual cues; (2) current methods lack unified and semantically aligned representations, limiting cross-modal consistency. To address these issues, we propose a multi-granularity cross-modal alignment framework (MGCAF), which leverages global and local visual features along with textual cues from images, enabling multi-granularity cross-modal alignment and interactions between text and images. Specifically, we introduce a multi-granularity visual feature fusion (MVF) module to balance and integrate global and local visual information and a multi-granularity cross-modal alignment (MCA) module to align multi-granularity visual and textual representations. Notably, GMNER involves large-scale multimodal processing and dense cross-modal interactions, and the proposed MGCAF is inherently suited for parallel and distributed computation, enabling efficient training and scalable inference. Experimental results demonstrate that MGCAF improves both entity recognition and visual grounding, validating its effectiveness for the GMNER task.