The digital age has witnessed an unprecedented surge in fake news, a challenge further exacerbated by advanced Large Language Models (LLMs) capable of subtle content manipulation. This study systematically evaluates the robustness of fake news detection methods—spanning traditional machine learning and deep learning-based models—against LLM-generated texts using the WELFake dataset. Transformer-based models (e.g., DistilBERT) show near-perfect accuracy on original data but suffer up to a 33% drop in F1-score on LLM-manipulated content, whereas traditional methods exhibit greater resilience. We further demonstrate that incorporating both original and manipulated texts during training improves detection robustness, albeit with increased computational costs. Our findings underscore the urgent need for adaptive detection strategies that balance high accuracy with robustness against sophisticated content manipulation. To ensure full reproducibility and support future research, our codebase, datasets, and trained models are openly available at https://github.com/inflaton/fake-news .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating the Impact of LLM-Manipulated Content on Fake News Detection

  • Donghao Huang,
  • Darius Ng,
  • Zhaoxia Wang,
  • Haibo Pen,
  • Erik Cambria

摘要

The digital age has witnessed an unprecedented surge in fake news, a challenge further exacerbated by advanced Large Language Models (LLMs) capable of subtle content manipulation. This study systematically evaluates the robustness of fake news detection methods—spanning traditional machine learning and deep learning-based models—against LLM-generated texts using the WELFake dataset. Transformer-based models (e.g., DistilBERT) show near-perfect accuracy on original data but suffer up to a 33% drop in F1-score on LLM-manipulated content, whereas traditional methods exhibit greater resilience. We further demonstrate that incorporating both original and manipulated texts during training improves detection robustness, albeit with increased computational costs. Our findings underscore the urgent need for adaptive detection strategies that balance high accuracy with robustness against sophisticated content manipulation. To ensure full reproducibility and support future research, our codebase, datasets, and trained models are openly available at https://github.com/inflaton/fake-news .