Overcoming Feature Missing: Joint Reconstruction and Prior Semantics Transmission for Robust Multimodal Sentiment Analysis
摘要
In the field of Multimodal Sentiment Analysis (MSA), semantic sparsity caused by sensor malfunctions and translation errors poses significant challenges to the robustness of models. This paper proposes an innovative framework of joint reconstruction and prior semantics transmission for robust MSA. First, to address the semantic loss inherent in single-modality reconstruction, we design a novel joint reconstruction method that leverages the complementarity of multiple modalities to enhance semantic recovery. Additionally, we propose a simple and effective training paradigm for Robust MSA which incorporates Confidence-based Dynamic Distillation and Teacher-guided Contrastive Learning. These two approaches facilitate a prior semantic transmission from the perspectives of intra-sample and inter-sample, respectively. Finally, to achieve a balance between robustness and accuracy under different noise scenarios, a gated hierarchical feature fusion is designed for adaptively fusing multi-granularity features. We conduct extensive experiments on two benchmark MSA datasets under different missing scenarios. The experimental results demonstrate that our proposed framework outperforms existing baselines and shows superior performance across multiple evaluation metrics. Our code will be publicly released on GitHub https://github.com/TouchYourYearn/PRCV2025-Robust-MSA .