The online platforms have made it increasingly convenient for users to express their thoughts and emotions through both text and images. This widespread sharing results in large volumes of user-generated content rich with emotional signals, which presents valuable opportunities for multi-modal sentiment analysis. Unlike traditional sentiment analysis that relies on a single modality, leveraging multiple modalities, such as text and images, enables a more holistic understanding of user sentiment by capturing complementary cues across different data types. As platforms like Twitter increasingly host diverse content formats, analyzing sentiment using multi-modal data has become a central focus in artificial intelligence research. However, many existing models fail to fully capture the interaction between text and image, often treating them separately, which limits prediction accuracy. This study addresses that gap by introducing a novel multi-modal sentiment analysis model that integrates contextual modeling with an attention mechanism for textual modality and a transformer for visual modality. An attention mechanism strengthens semantic understanding, while transformer encoder layers uncover long-range dependencies within images. The model processes text and images using specialized encoders and performs early feature-based fusion. Experimental evaluation on the MVSA dataset, using accuracy and F1-score metrics, shows that the proposed model significantly outperforms previous methods in effectively interpreting complex multi-modal sentiment analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

RCM-MSA: Robust Contextual Modeling for Multi-modal Sentiment Analysis Using Transformer and Attention Mechanism

  • Hoai-Phuong Nguyen-Cao,
  • Phuoc-Hung Vo,
  • Nhut-Lam Nguyen

摘要

The online platforms have made it increasingly convenient for users to express their thoughts and emotions through both text and images. This widespread sharing results in large volumes of user-generated content rich with emotional signals, which presents valuable opportunities for multi-modal sentiment analysis. Unlike traditional sentiment analysis that relies on a single modality, leveraging multiple modalities, such as text and images, enables a more holistic understanding of user sentiment by capturing complementary cues across different data types. As platforms like Twitter increasingly host diverse content formats, analyzing sentiment using multi-modal data has become a central focus in artificial intelligence research. However, many existing models fail to fully capture the interaction between text and image, often treating them separately, which limits prediction accuracy. This study addresses that gap by introducing a novel multi-modal sentiment analysis model that integrates contextual modeling with an attention mechanism for textual modality and a transformer for visual modality. An attention mechanism strengthens semantic understanding, while transformer encoder layers uncover long-range dependencies within images. The model processes text and images using specialized encoders and performs early feature-based fusion. Experimental evaluation on the MVSA dataset, using accuracy and F1-score metrics, shows that the proposed model significantly outperforms previous methods in effectively interpreting complex multi-modal sentiment analysis.