Multimodal Relation Extraction (MRE) is a task that identifies semantic relations between two entities from image-text pairs. Existing methods often suffer from insufficient extraction accuracy and low cross-modal fusion efficiency due to inadequate auxiliary information. In this paper, we propose a multimodal relation extraction framework KITE-MRE,which integrating large model generation and knowledge graph (KG) to provide abundant additional auxiliary information. Specifically, prompts guide large language models to construct image semantic entity triplets and generate image graphs, while knowledge graph entity linking technology is applied in text graphs to build a unified graph structure. Additionally, we introduce the Graph Information Bottleneck (GIB) technique combined with Graph Convolutional Networks (GCN) to optimize, and perform feature fusion by integrating semantic and structural multi-perspective information to predict relations. Experimental results demonstrate that this approach provides a new path for multimodal data processing, effectively addressing issues of knowledge graph incompleteness and ambiguous relation generation by large models in cross-modal scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

KITE-MRE: Knowledge-Infused and Transformer-Enhanced Multimodal Relation Extraction

  • Junjie Li,
  • Wenti Huang,
  • Yu He,
  • Xinjie Mo

摘要

Multimodal Relation Extraction (MRE) is a task that identifies semantic relations between two entities from image-text pairs. Existing methods often suffer from insufficient extraction accuracy and low cross-modal fusion efficiency due to inadequate auxiliary information. In this paper, we propose a multimodal relation extraction framework KITE-MRE,which integrating large model generation and knowledge graph (KG) to provide abundant additional auxiliary information. Specifically, prompts guide large language models to construct image semantic entity triplets and generate image graphs, while knowledge graph entity linking technology is applied in text graphs to build a unified graph structure. Additionally, we introduce the Graph Information Bottleneck (GIB) technique combined with Graph Convolutional Networks (GCN) to optimize, and perform feature fusion by integrating semantic and structural multi-perspective information to predict relations. Experimental results demonstrate that this approach provides a new path for multimodal data processing, effectively addressing issues of knowledge graph incompleteness and ambiguous relation generation by large models in cross-modal scenarios.