Scholarlink

<p>Multimodal sentiment analysis (MSA) has gained much more attentions as people are increasingly expressing emotions and opinions through multimodal media. Previously, powerful methods for MSA involved independently extracting linguistic, visual, and acoustic features and then integrating them into language models to make decisions. However, these different modal features were not aligned, and only the linguistic modality could be applied to the powerful large language model. To tackle the above issues, a framework named MLLM-enhanced sentiment networks has been proposed for MSA. Specifically, we introduce a multimodal large language model&#xa0;(MLLM) to do MSA with designed sentiment analysis prompts based on the text, video, and audio. After inputting the three modal information, the output of the last hidden layer is captured as enhanced linguistic features, which can be decoded into output text. The output text contains the result of sentiment analysis and the enhanced linguistic features of the text are selected to represent the judgement of MLLM. Then, MLLM is used to extract aligned features of the text, video, and audio, and the MLLM-refinement&#xa0;(MR) module is further designed to improve the prediction of original MLLM. In the MR module, cross-attention mechanism is utilized to associate the enhanced linguistic features output by MLLM with the aligned visual and audio features for injecting more clues to the prediction of MLLM. Finally, a series of Transformer layers are applied to learn such features and make refining predictions. Experiments demonstrated that the proposed model can achieve the state-of-the-art performance in MOSI, and MOSEI dataset.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal large language model enhancement network for multimodal sentiment analysis

  • 期刊论文

  • 发表日期：

    2025年8月21日

查看全文
  • Huanyu Zhu,
  • Zhihao Shen,
  • Chengxiao Dai,
  • Zhitao Yu

摘要

Multimodal sentiment analysis (MSA) has gained much more attentions as people are increasingly expressing emotions and opinions through multimodal media. Previously, powerful methods for MSA involved independently extracting linguistic, visual, and acoustic features and then integrating them into language models to make decisions. However, these different modal features were not aligned, and only the linguistic modality could be applied to the powerful large language model. To tackle the above issues, a framework named MLLM-enhanced sentiment networks has been proposed for MSA. Specifically, we introduce a multimodal large language model (MLLM) to do MSA with designed sentiment analysis prompts based on the text, video, and audio. After inputting the three modal information, the output of the last hidden layer is captured as enhanced linguistic features, which can be decoded into output text. The output text contains the result of sentiment analysis and the enhanced linguistic features of the text are selected to represent the judgement of MLLM. Then, MLLM is used to extract aligned features of the text, video, and audio, and the MLLM-refinement (MR) module is further designed to improve the prediction of original MLLM. In the MR module, cross-attention mechanism is utilized to associate the enhanced linguistic features output by MLLM with the aligned visual and audio features for injecting more clues to the prediction of MLLM. Finally, a series of Transformer layers are applied to learn such features and make refining predictions. Experiments demonstrated that the proposed model can achieve the state-of-the-art performance in MOSI, and MOSEI dataset.