Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to jointly identify aspect terms and their corresponding sentiment polarity from text-image pairs. Recent studies have applied prompt-based learning to MABSA tasks, which leverage the existing knowledge of pre-trained language models to fine-tuning with only a small number of samples, thereby significantly reducing the demand for data. However, these existing studies still exhibit notable deficiencies. They fail to fully explore the features of each modality and have only established a coarse-grained alignment between the textual and visual modalities, lacking a fine-grained cross-modal interaction. To address the aforementioned limitations, this paper proposes a novel Enhanced Generative Multimodal Prompt (EGMP) model. Specifically, we first employ GPT-4 combined with Chain-of-Thought (CoT) reasoning to generate image descriptions for visual semantic enhancement. Furthermore, to tackle the challenges of imprecise cross-modal alignment between image regions and textual aspects, a Syntax-aware Aspect-guided Fine-grained Alignment Module is introduced, which simultaneously filters semantically relevant information for each specific aspect from both textual tokens and visual patches, ultimately achieving aspect-region-opinion alignment. Finally, based on the semantically aligned representations, we construct enhanced multimodal embeddings with prompt for different tasks. Extensive experiments conducted on two benchmark datasets demonstrate that the proposed method outperforms the strong baseline model in two MABSA-related tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EGMP: Enhanced Generative Multimodal Prompt for Few-Shot Multimodal Aspect-Based Sentiment Analysis

  • Shuangfeng Cai,
  • Shengfa Miao,
  • Tao Chen,
  • Houcheng Liu,
  • Dinan Ma,
  • Yongkang Mu,
  • Xin Jin,
  • Qian Jiang,
  • Hua Jiang,
  • Puming Wang,
  • Hualong Deng,
  • Ahmed Zahir

摘要

Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to jointly identify aspect terms and their corresponding sentiment polarity from text-image pairs. Recent studies have applied prompt-based learning to MABSA tasks, which leverage the existing knowledge of pre-trained language models to fine-tuning with only a small number of samples, thereby significantly reducing the demand for data. However, these existing studies still exhibit notable deficiencies. They fail to fully explore the features of each modality and have only established a coarse-grained alignment between the textual and visual modalities, lacking a fine-grained cross-modal interaction. To address the aforementioned limitations, this paper proposes a novel Enhanced Generative Multimodal Prompt (EGMP) model. Specifically, we first employ GPT-4 combined with Chain-of-Thought (CoT) reasoning to generate image descriptions for visual semantic enhancement. Furthermore, to tackle the challenges of imprecise cross-modal alignment between image regions and textual aspects, a Syntax-aware Aspect-guided Fine-grained Alignment Module is introduced, which simultaneously filters semantically relevant information for each specific aspect from both textual tokens and visual patches, ultimately achieving aspect-region-opinion alignment. Finally, based on the semantically aligned representations, we construct enhanced multimodal embeddings with prompt for different tasks. Extensive experiments conducted on two benchmark datasets demonstrate that the proposed method outperforms the strong baseline model in two MABSA-related tasks.