The proliferation of multimodal fake news content has garnered increasing attention. Existing multimodal fake news detection methods have significantly contributed to capturing the fusion features of multimodal news content. However, effectively fusing modalities to learn discriminative cross-modal features remains challenging, partly due to “gradient starvation” and the difficulty of learning complex interactions. Consequently, models often rely heavily on easily learned unimodal cues, impeding the acquisition of advanced cross-modal understanding. To promote balanced learning across modalities, we propose SESAME, a self-distillation across modalities framework for multimodal fake news detection, which leverages internal self-distillation from multimodal representations to unimodal, facilitating the aggregation of aligned modal information. Specifically, we add auxiliary classification heads to the unimodal representations, and the fused deep cross-modal representation serves as a teacher, then perform self-distillation on the unimodal output probabilities using the fused multimodal probabilities, this encourages the model to prioritize features valuable for cross-modal fusion during training, leading to the learning of more effective cross-modal features and thus improving performance derived from modality interaction. The experimental results demonstrate that our proposed SESAME exhibits superior performance in multimodal fake news detection compare to other strong and competitive deep learning methods for multimodal fake news detection.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Self-Distillation Across Modalities: Enhancing Cross-Modal Correlation Perception for Multimodal Fake News Detection

  • Yidu Chen,
  • Bo Ma,
  • Yating Yang,
  • Dilxat Abdureyim,
  • Chirui Zhang,
  • Rui Dong,
  • Lei Wang,
  • Xi Zhou

摘要

The proliferation of multimodal fake news content has garnered increasing attention. Existing multimodal fake news detection methods have significantly contributed to capturing the fusion features of multimodal news content. However, effectively fusing modalities to learn discriminative cross-modal features remains challenging, partly due to “gradient starvation” and the difficulty of learning complex interactions. Consequently, models often rely heavily on easily learned unimodal cues, impeding the acquisition of advanced cross-modal understanding. To promote balanced learning across modalities, we propose SESAME, a self-distillation across modalities framework for multimodal fake news detection, which leverages internal self-distillation from multimodal representations to unimodal, facilitating the aggregation of aligned modal information. Specifically, we add auxiliary classification heads to the unimodal representations, and the fused deep cross-modal representation serves as a teacher, then perform self-distillation on the unimodal output probabilities using the fused multimodal probabilities, this encourages the model to prioritize features valuable for cross-modal fusion during training, leading to the learning of more effective cross-modal features and thus improving performance derived from modality interaction. The experimental results demonstrate that our proposed SESAME exhibits superior performance in multimodal fake news detection compare to other strong and competitive deep learning methods for multimodal fake news detection.