Transformer-enhanced multimodal learning model for multimedia content understanding
摘要
Multimedia content understanding requires effectively integrating information from various modalities such as image, text, and audio. Traditional multimodal learning models often struggle with semantic misalignment and inadequate cross-modal interactions. To address these challenges, we propose TE-MFM (Transformer-Enhanced Multimodal Fusion Model), a novel framework that leverages a multi-level transformer architecture with cross-modal attention and semantic-guided alignment. Unlike existing methods, TE-MFM innovatively incorporates bidirectional cross-modal attention to capture mutual interactions more effectively and introduces a semantic-guided alignment mechanism that explicitly enforces consistent feature representation across modalities. TE-MFM dynamically captures fine-grained dependencies across modalities and aligns them within a unified semantic space, significantly improving performance in multimedia tasks such as emotion recognition, cross-modal retrieval, and event classification. Extensive experiments on benchmark datasets demonstrate that TE-MFM achieves state-of-the-art results, showing both superior accuracy and robustness.