A Comprehensive Evaluation of Multimodal Fake News Detection: Assessing Robustness and Efficacy
摘要
The escalating threat of multimodal misinformation, propagated through manipulated text and imagery, necessitates robust and resilient detection frameworks. As generative AI advances, adversarial attacks exploit both modalities to bypass existing defenses, posing a critical challenge. While multimodal fake news detection models offer promising solutions by integrating textual and visual features, their inherent vulnerabilities under adversarial conditions and their behavior in incomplete modality scenarios remain insufficiently explored. This study provides a systematic evaluation of three widely-used multimodal architectures−BERT-VGG, CNN-VGG, and LSTM-VGG−on two benchmark datasets, D1 and RECOVERY. We significantly extend the scope of robustness assessment by subjecting these models to a diverse range of adversarial attacks, including ten image-based and five text-based perturbations, alongside novel large language model (LLM)-driven manipulations using LLAMA3. Furthermore, our research rigorously investigates the impact of modality removal (unimodal settings) on detection performance, offering crucial insights into the intrinsic reliance on specific modalities. Our findings reveal significant vulnerabilities within current multimodal fake news detection models, particularly their susceptibility to advanced text-based adversarial attacks and evidence of “Unimodal Bias." Crucially, the image modality is demonstrated to play an indispensable role in maintaining detection efficacy and resilience. This comprehensive analysis highlights the urgent need for adaptive strategies and advanced architectures that can withstand sophisticated adversarial tactics and manage modality inconsistencies, paving the way for the development of more resilient and effective next-generation misinformation detection systems.