Large language models (LLMs) are increasingly used in scientific research, education, and healthcare. Their roles in data searching, screening, extraction, and quality assessment hold promise for improving the systematic review process. However, concerns remain about their accuracy in literature research. This study aimed to compare the accuracy of LLMs and human reviewers in full-text screening for a systematic review. We searched for relevant studies from databases and registers. Full-text screening was performed by human reviewers, ChatGPT, and DeepSeek based on predefined inclusion and exclusion criteria. We observed that ChatGPT had 73.6% agreement with human reviewers (κ = 0.43), while DeepSeek had 70.3% (κ = 0.35). Moreover, ChatGPT showed 73.6% accuracy, high sensitivity (0.923), but low specificity (0.487) compared to the human consensus. Similarly, DeepSeek had 70.3% accuracy, higher sensitivity (0.962), but lower specificity (0.359). Both ChatGPT and DeepSeek show promise for assisting full-text screening in systematic reviews but require further evaluation with well-defined prompt engineering.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Comparative Assessment of ChatGPT, DeepSeek, and Human Reviewers for Full-Text Screening in Systematic Reviews on the Impact of Air Pollution in Respiratory Diseases

  • Amja Manullang,
  • Xiaohong Gao,
  • Christophe Viavattene,
  • Annisa Ristya Rahmanti

摘要

Large language models (LLMs) are increasingly used in scientific research, education, and healthcare. Their roles in data searching, screening, extraction, and quality assessment hold promise for improving the systematic review process. However, concerns remain about their accuracy in literature research. This study aimed to compare the accuracy of LLMs and human reviewers in full-text screening for a systematic review. We searched for relevant studies from databases and registers. Full-text screening was performed by human reviewers, ChatGPT, and DeepSeek based on predefined inclusion and exclusion criteria. We observed that ChatGPT had 73.6% agreement with human reviewers (κ = 0.43), while DeepSeek had 70.3% (κ = 0.35). Moreover, ChatGPT showed 73.6% accuracy, high sensitivity (0.923), but low specificity (0.487) compared to the human consensus. Similarly, DeepSeek had 70.3% accuracy, higher sensitivity (0.962), but lower specificity (0.359). Both ChatGPT and DeepSeek show promise for assisting full-text screening in systematic reviews but require further evaluation with well-defined prompt engineering.