<p>Multi-modal relation extraction (MRE) aims to identify semantic relations between entities in text by leveraging associated visual information. While existing methods have made progress, they face two key limitations: (1) overlooking fine-grained visual relationships between objects that provide critical contextual clues, and (2) failing to utilize external knowledge that could enhance understanding of entities and textual context. To address these challenges, we propose DGF-PT, a novel framework that integrates Visual Relations and Multi-grained Knowledge for MRE. Our approach first constructs a unified multi-modal graph by aligning visual objects and textual tokens through scene graphs, where diverse relations (e.g., spatial, action-based) are explicitly modeled. A relation-aware Transformer then propagates information across the graph while preserving modality-specific and cross-modal interactions. Additionally, we employ large language models (LLMs) to generate entity-level descriptions and sentence-level explanations, which are selectively fused via cross-attention to enrich textual representations. Experiments on the MNRE benchmark show that DGF-PT achieves state-of-the-art performance (85.26% F1), outperforming prior methods by 2.71%.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Visual relation-aware and knowledge-guided multi-modal relation extraction

  • Chunyu Lu,
  • Tianran Chen,
  • Duo Shang,
  • Jun Luo,
  • Xin Hui,
  • Ruhui Shi

摘要

Multi-modal relation extraction (MRE) aims to identify semantic relations between entities in text by leveraging associated visual information. While existing methods have made progress, they face two key limitations: (1) overlooking fine-grained visual relationships between objects that provide critical contextual clues, and (2) failing to utilize external knowledge that could enhance understanding of entities and textual context. To address these challenges, we propose DGF-PT, a novel framework that integrates Visual Relations and Multi-grained Knowledge for MRE. Our approach first constructs a unified multi-modal graph by aligning visual objects and textual tokens through scene graphs, where diverse relations (e.g., spatial, action-based) are explicitly modeled. A relation-aware Transformer then propagates information across the graph while preserving modality-specific and cross-modal interactions. Additionally, we employ large language models (LLMs) to generate entity-level descriptions and sentence-level explanations, which are selectively fused via cross-attention to enrich textual representations. Experiments on the MNRE benchmark show that DGF-PT achieves state-of-the-art performance (85.26% F1), outperforming prior methods by 2.71%.