Multimodal event relation extraction aims to identify logical relationships (e.g., causality, enables, reaction) between video-text event pairs, which is critical for video understanding applications. Existing methods suffer from shallow feature fusion and inadequate cross-modal interaction, limiting their performance in complex scenarios. To address these challenges, we propose M3FERE(Multi-level Multimodal Fusion for Event Relation Extraction), a novel framework featuring three key innovations: (1) Hierarchical video representation integrating global spatio-temporal features (via SlowFast) with local object-centric features through Graph Convolutional Network (GCN)-based fusion; (2) Text-guided cross-modal screening that filters irrelevant visual noise using attention mechanisms; (3) Temporal-aware event-pair modeling to capture chronological dependencies. Comprehensive evaluations on the VidSitu benchmark demonstrate that M3FERE achieves 36.07% Macro-Acc@1, outperforming established baselines such as OME (+0.48%) and SlowFast (+1.92%). Ablation studies validate the consistent improvements from global-local fusion (+0.22% accuracy) and cross-modal screening (+0.28% accuracy). This work provides a principled approach for video-text understanding, with practical applications in news analysis and social media mining.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

M3FERE: Multi-Level Multimodal Fusion for Event Relation Extraction

  • Bo Qing,
  • Qing He,
  • Wei Liu

摘要

Multimodal event relation extraction aims to identify logical relationships (e.g., causality, enables, reaction) between video-text event pairs, which is critical for video understanding applications. Existing methods suffer from shallow feature fusion and inadequate cross-modal interaction, limiting their performance in complex scenarios. To address these challenges, we propose M3FERE(Multi-level Multimodal Fusion for Event Relation Extraction), a novel framework featuring three key innovations: (1) Hierarchical video representation integrating global spatio-temporal features (via SlowFast) with local object-centric features through Graph Convolutional Network (GCN)-based fusion; (2) Text-guided cross-modal screening that filters irrelevant visual noise using attention mechanisms; (3) Temporal-aware event-pair modeling to capture chronological dependencies. Comprehensive evaluations on the VidSitu benchmark demonstrate that M3FERE achieves 36.07% Macro-Acc@1, outperforming established baselines such as OME (+0.48%) and SlowFast (+1.92%). Ablation studies validate the consistent improvements from global-local fusion (+0.22% accuracy) and cross-modal screening (+0.28% accuracy). This work provides a principled approach for video-text understanding, with practical applications in news analysis and social media mining.