<p>This paper addresses the issue of modality imbalance in multimodal fake news detection, where models tend to over-rely on a single modality, leading to suboptimal performance. The aim is to introduce a method that mitigates this imbalance to improve the accuracy and generalization of fake news detection models. The paper proposes a method called Dynamic Contrastive Learning with Re-initialization (DCLR). DCLR monitors the learning status of each modality by evaluating clustering purity in the representation space to identify imbalance. To address this, it employs a network re-initialization strategy to lessen the model’s dependence on the dominant modality and provide more training opportunities for the suppressed one. Additionally, it uses a strategy named "dynamic activation contrastive learning" to mitigate the overfitting risk of the dominant modality by applying a supervised contrastive learning strategy when a certain threshold is exceeded. The proposed DCLR method was tested on the Weibo and Twitter datasets and compared against several unimodal and multimodal baseline methods. The experimental results show that DCLR outperforms previous representative methods, achieving superior overall performance on both datasets. Ablation studies confirmed the effectiveness of both the re-initialization and dynamic contrastive learning components. Visual analysis using t-SNE demonstrated that the DCLR method improved the discriminative ability of the weaker modality’s encoder after re-initialization. The study concludes that mitigating modality imbalance during the training process can significantly enhance the accuracy of multimodal fake news detection. The DCLR method, through its re-initialization and dynamic contrastive learning strategies, effectively addresses this imbalance, leading to better model performance and generalization. This highlights the importance of ensuring all modalities are fully trained for effective information fusion in fake news detection.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DCLR: mitigating modality imbalance in multimodal fake news detection via dynamic contrastive learning with re-initialization

  • Zhenyu Wang,
  • Chao Jiang,
  • Kun Wang

摘要

This paper addresses the issue of modality imbalance in multimodal fake news detection, where models tend to over-rely on a single modality, leading to suboptimal performance. The aim is to introduce a method that mitigates this imbalance to improve the accuracy and generalization of fake news detection models. The paper proposes a method called Dynamic Contrastive Learning with Re-initialization (DCLR). DCLR monitors the learning status of each modality by evaluating clustering purity in the representation space to identify imbalance. To address this, it employs a network re-initialization strategy to lessen the model’s dependence on the dominant modality and provide more training opportunities for the suppressed one. Additionally, it uses a strategy named "dynamic activation contrastive learning" to mitigate the overfitting risk of the dominant modality by applying a supervised contrastive learning strategy when a certain threshold is exceeded. The proposed DCLR method was tested on the Weibo and Twitter datasets and compared against several unimodal and multimodal baseline methods. The experimental results show that DCLR outperforms previous representative methods, achieving superior overall performance on both datasets. Ablation studies confirmed the effectiveness of both the re-initialization and dynamic contrastive learning components. Visual analysis using t-SNE demonstrated that the DCLR method improved the discriminative ability of the weaker modality’s encoder after re-initialization. The study concludes that mitigating modality imbalance during the training process can significantly enhance the accuracy of multimodal fake news detection. The DCLR method, through its re-initialization and dynamic contrastive learning strategies, effectively addresses this imbalance, leading to better model performance and generalization. This highlights the importance of ensuring all modalities are fully trained for effective information fusion in fake news detection.