<p>Multimedia content understanding requires effectively integrating information from various modalities such as image, text, and audio. Traditional multimodal learning models often struggle with semantic misalignment and inadequate cross-modal interactions. To address these challenges, we propose TE-MFM (Transformer-Enhanced Multimodal Fusion Model), a novel framework that leverages a multi-level transformer architecture with cross-modal attention and semantic-guided alignment. Unlike existing methods, TE-MFM innovatively incorporates bidirectional cross-modal attention to capture mutual interactions more effectively and introduces a semantic-guided alignment mechanism that explicitly enforces consistent feature representation across modalities. TE-MFM dynamically captures fine-grained dependencies across modalities and aligns them within a unified semantic space, significantly improving performance in multimedia tasks such as emotion recognition, cross-modal retrieval, and event classification. Extensive experiments on benchmark datasets demonstrate that TE-MFM achieves state-of-the-art results, showing both superior accuracy and robustness.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Transformer-enhanced multimodal learning model for multimedia content understanding

  • Bing Fu,
  • Jichang Tian,
  • Ying Wei,
  • Lina Bai

摘要

Multimedia content understanding requires effectively integrating information from various modalities such as image, text, and audio. Traditional multimodal learning models often struggle with semantic misalignment and inadequate cross-modal interactions. To address these challenges, we propose TE-MFM (Transformer-Enhanced Multimodal Fusion Model), a novel framework that leverages a multi-level transformer architecture with cross-modal attention and semantic-guided alignment. Unlike existing methods, TE-MFM innovatively incorporates bidirectional cross-modal attention to capture mutual interactions more effectively and introduces a semantic-guided alignment mechanism that explicitly enforces consistent feature representation across modalities. TE-MFM dynamically captures fine-grained dependencies across modalities and aligns them within a unified semantic space, significantly improving performance in multimedia tasks such as emotion recognition, cross-modal retrieval, and event classification. Extensive experiments on benchmark datasets demonstrate that TE-MFM achieves state-of-the-art results, showing both superior accuracy and robustness.