The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. To address this, we proposed DGFN (Distributed Global Fusion Network), a brain-inspired framework designed to achieve robust and generalizable sentiment prediction under random missing conditions. DGFN integrated two major components: the Global-aware Cross-modal Representation Enhancement (GCRE) module and the Decentralized Fusion with Knowledge Guidance (DFKG) module. GCRE captured both temporal dynamics and text-centric cross-modal interactions while also constructing global semantic knowledge through contrastive learning. DFKG adopted a brain-inspired multi-center fusion strategy, enabling modality-aware integration that dynamically adapted to incomplete inputs. To improve robustness, DGFN introduced two components: a feature reconstructor and a modality integrity loss. The feature reconstructor restores corrupted features, while the modality integrity loss encourages adaptive compensation under random modality incompleteness. Together, they help maintain performance when input features are missing or degraded. We evaluated DGFN on three benchmark datasets—MOSI, MOSEI, and SIMS—under extensive random missing conditions using standardized evaluation protocols. Experimental results consistently demonstrated its superiority over existing state-of-the-art methods in noisy multimodal scenarios (The code is available at: https://github.com/USTC-KnowledgeComputingLab/MMA-DFGN.git ).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DGFN: Brain-Inspired Distributed Fusion for Robust Multimodal Sentiment Analysis

  • Ningping Li,
  • Ruibao Zhang,
  • Xiangyu Zhou,
  • Weichen Dai,
  • Ji Qi,
  • Yi Zhou

摘要

The field of Multimodal Sentiment Analysis (MSA) has recently witnessed an emerging direction seeking to tackle the issue of data incompleteness. To address this, we proposed DGFN (Distributed Global Fusion Network), a brain-inspired framework designed to achieve robust and generalizable sentiment prediction under random missing conditions. DGFN integrated two major components: the Global-aware Cross-modal Representation Enhancement (GCRE) module and the Decentralized Fusion with Knowledge Guidance (DFKG) module. GCRE captured both temporal dynamics and text-centric cross-modal interactions while also constructing global semantic knowledge through contrastive learning. DFKG adopted a brain-inspired multi-center fusion strategy, enabling modality-aware integration that dynamically adapted to incomplete inputs. To improve robustness, DGFN introduced two components: a feature reconstructor and a modality integrity loss. The feature reconstructor restores corrupted features, while the modality integrity loss encourages adaptive compensation under random modality incompleteness. Together, they help maintain performance when input features are missing or degraded. We evaluated DGFN on three benchmark datasets—MOSI, MOSEI, and SIMS—under extensive random missing conditions using standardized evaluation protocols. Experimental results consistently demonstrated its superiority over existing state-of-the-art methods in noisy multimodal scenarios (The code is available at: https://github.com/USTC-KnowledgeComputingLab/MMA-DFGN.git ).