With the increasing importance of multimodal sentiment analysis in sentiment computing, how to efficiently fuse information from different modalities to improve the accuracy of sentiment recognition has become a key issue in current research. To this end, this paper proposes a Transformer-driven multimodal sentiment feature fusion method. First, in view of the different characteristics of multimodal data such as text, speech and image, this paper designs a Transformer model to extract the sentiment features of each modality respectively. Then, the multi-head self-attention mechanism is used to effectively fuse the information of different modalities. Subsequently, an adaptive weight mechanism is used to dynamically adjust the contribution of each modality to the final sentiment prediction to achieve more accurate sentiment analysis. In the experimental conclusion, on the English multimodal dataset, the Transformer model achieved a weighted accuracy of 85%, a macro F1 value of 0.82, and an AUC (Area Under the Curve) value of 0.91, which is significantly better than traditional models such as TFN (Tensor Fusion Network). In addition, the dynamic weight mechanism can effectively improve the robustness of the model in a noisy environment. Through ablation experiments and adversarial decoupling experiments, the key role of the cross-modal cross-attention mechanism and the dynamic weight mechanism in improving sentiment analysis performance was verified.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Transformer-Driven Multimodal Sentiment Feature Fusion Method

  • He Ma,
  • Yanli Xing

摘要

With the increasing importance of multimodal sentiment analysis in sentiment computing, how to efficiently fuse information from different modalities to improve the accuracy of sentiment recognition has become a key issue in current research. To this end, this paper proposes a Transformer-driven multimodal sentiment feature fusion method. First, in view of the different characteristics of multimodal data such as text, speech and image, this paper designs a Transformer model to extract the sentiment features of each modality respectively. Then, the multi-head self-attention mechanism is used to effectively fuse the information of different modalities. Subsequently, an adaptive weight mechanism is used to dynamically adjust the contribution of each modality to the final sentiment prediction to achieve more accurate sentiment analysis. In the experimental conclusion, on the English multimodal dataset, the Transformer model achieved a weighted accuracy of 85%, a macro F1 value of 0.82, and an AUC (Area Under the Curve) value of 0.91, which is significantly better than traditional models such as TFN (Tensor Fusion Network). In addition, the dynamic weight mechanism can effectively improve the robustness of the model in a noisy environment. Through ablation experiments and adversarial decoupling experiments, the key role of the cross-modal cross-attention mechanism and the dynamic weight mechanism in improving sentiment analysis performance was verified.