Multimodal Aspect-Based Sentiment Analysis (MABSA) is a fine-grained sentiment analysis task that aims to extract aspect terms and predict their sentiment polarity from text-image pairs. Existing works mainly use image information to improve the performance of MABSA tasks. However, most studies overestimate the importance of images because there are many noisy images in the dataset that are irrelevant to the text, which will have a negative impact on precision. Although some works try to filter low-quality noisy images by setting thresholds or introducing a crossmodal relation detection module. There hasn’t been much research on how to dynamically filter image information during training. Therefore, we present a novel framework named Knowledge Distillation Framework with Dual-Stage Visual Filtering for Robust Multimodal Sentiment Analysis (DSVF-KD). In this work, we first borrow the idea of Mixture-of-Experts (MoE) to design a two-stage visual information filtering module. This module can achieve two-level filtering including image semantic information and visual regions through Sparse MoE and learnable region routers. To supervise the training of this module, we design a softlabels method based on semantic similarity to promote human-like attention behavior. Secondly, we propose a joint knowledge distillation strategy, in which the teacher model is trained on an augmented dataset generated by a large language model prompt, with the goal of transferring the image-text alignment knowledge in the augmented data to the student model trained using the original data.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DSVF-KD: Knowledge Distillation Framework with Dual-Stage Visual Filtering for Robust Multimodal Sentiment Analysis

  • Xiaoya Cui,
  • Tao Wan,
  • Zengchang Qin

摘要

Multimodal Aspect-Based Sentiment Analysis (MABSA) is a fine-grained sentiment analysis task that aims to extract aspect terms and predict their sentiment polarity from text-image pairs. Existing works mainly use image information to improve the performance of MABSA tasks. However, most studies overestimate the importance of images because there are many noisy images in the dataset that are irrelevant to the text, which will have a negative impact on precision. Although some works try to filter low-quality noisy images by setting thresholds or introducing a crossmodal relation detection module. There hasn’t been much research on how to dynamically filter image information during training. Therefore, we present a novel framework named Knowledge Distillation Framework with Dual-Stage Visual Filtering for Robust Multimodal Sentiment Analysis (DSVF-KD). In this work, we first borrow the idea of Mixture-of-Experts (MoE) to design a two-stage visual information filtering module. This module can achieve two-level filtering including image semantic information and visual regions through Sparse MoE and learnable region routers. To supervise the training of this module, we design a softlabels method based on semantic similarity to promote human-like attention behavior. Secondly, we propose a joint knowledge distillation strategy, in which the teacher model is trained on an augmented dataset generated by a large language model prompt, with the goal of transferring the image-text alignment knowledge in the augmented data to the student model trained using the original data.