Multimodal Fake News Detection Based on Cross-Modal Attention and Contrastive Learning
摘要
To address the limitations of effectively integrating textual and visual information while maintaining their unique characteristics, this paper proposes a Cross-modal Attention and Contrastive Learning Neural Network (CACLN) model for multimodal fake news detection. This model employs a cross-modal attention mechanism to capture semantic associations between images and text, while incorporating contrastive learning to enhance the consistency of representation across modalities. Specifically, the CACLN model leverages Vision Transformer (ViT) to capture semantic features from images, and employs a GPT decoder to generate an image caption. The BERT is utilized for embedding of news articles and image captions, and ResNet50 is employed to capture visual features of images. The CACLN model achieves effective multimodal information fusion via cross-modal attention and employs a softmax classifier for fake news prediction. Extensive experiments on three datasets demonstrate that the CACLN model significantly gains better results than baseline models. Ablation studies reveal that caption embeddings and contrastive learning provide critical semantic information, while the cross-modal attention effectively promotes semantic alignment and feature interaction across modalities.