Scholarlink

Background <p>Large language models (LLMs) are proposed as decision-support tools in medicine, but their ability to reason through neurological cases—requiring anatomoclinical correlation—remains uncertain.</p> Methods <p>Five LLMs (Copilot GPT-5, DeepSeek-V3.1, ChatGPT 3.5, ChatGPT 5 Auto, Gemini 2.5) were tested on 86 neurological vignettes. Outputs were graded as <i>Correct</i>, <i>Partial</i>, or <i>Incorrect</i> for diagnosis and lesion site. Accuracy was analyzed using binary (correct vs. not correct), weighted (partial = 0.5), and ordinal (0–1–2 scale) methods. Global comparisons employed Cochran’s Q or Friedman tests; pairwise differences used McNemar or Wilcoxon tests with Holm correction. Interrater reliability was assessed with Fleiss’ κ, Cohen’s κ, and raw percent agreement.</p> Results <p>For diagnosis, ChatGPT 5 Auto, Gemini 2.5, Copilot GPT-5, and DeepSeek-V3.1 each achieved &gt; 90% accuracy, all significantly outperforming ChatGPT 3.5. For lesion localization, ChatGPT 5 Auto achieved the highest diagnostic accuracy, followed closely by Copilot GPT-5 and Gemini, while ChatGPT-3.5 lagged behind and had the highest rate of incorrect responses (13.9%). Weighted and ordinal analyses confirmed the same rank order, clustering the four advanced models together. Agreement was limited for diagnosis (Fleiss’ κ = 0.22) and for localization (Fleiss’ κ = 0.20).</p> Conclusion <p>Advanced LLMs demonstrate high and comparable performance in neurology, while ChatGPT 3.5 is significantly less accurate. Errors were heterogeneous across models, suggesting potential ensemble benefits but reinforcing the need for human oversight, particularly in the irreplaceable human tasks of history‑taking, examination, and contextual clinical judgment.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

AI for clinical neurology: Comparing five large language models in neurological diagnosis and lesion localization

  • 期刊论文

  • 发表日期：

    2026年2月7日

查看全文
  • Francesco Brigo,
  • Arian Zaboli,
  • Elisabetta Maida,
  • Luigi Lavorgna

摘要

Background

Large language models (LLMs) are proposed as decision-support tools in medicine, but their ability to reason through neurological cases—requiring anatomoclinical correlation—remains uncertain.

Methods

Five LLMs (Copilot GPT-5, DeepSeek-V3.1, ChatGPT 3.5, ChatGPT 5 Auto, Gemini 2.5) were tested on 86 neurological vignettes. Outputs were graded as Correct, Partial, or Incorrect for diagnosis and lesion site. Accuracy was analyzed using binary (correct vs. not correct), weighted (partial = 0.5), and ordinal (0–1–2 scale) methods. Global comparisons employed Cochran’s Q or Friedman tests; pairwise differences used McNemar or Wilcoxon tests with Holm correction. Interrater reliability was assessed with Fleiss’ κ, Cohen’s κ, and raw percent agreement.

Results

For diagnosis, ChatGPT 5 Auto, Gemini 2.5, Copilot GPT-5, and DeepSeek-V3.1 each achieved > 90% accuracy, all significantly outperforming ChatGPT 3.5. For lesion localization, ChatGPT 5 Auto achieved the highest diagnostic accuracy, followed closely by Copilot GPT-5 and Gemini, while ChatGPT-3.5 lagged behind and had the highest rate of incorrect responses (13.9%). Weighted and ordinal analyses confirmed the same rank order, clustering the four advanced models together. Agreement was limited for diagnosis (Fleiss’ κ = 0.22) and for localization (Fleiss’ κ = 0.20).

Conclusion

Advanced LLMs demonstrate high and comparable performance in neurology, while ChatGPT 3.5 is significantly less accurate. Errors were heterogeneous across models, suggesting potential ensemble benefits but reinforcing the need for human oversight, particularly in the irreplaceable human tasks of history‑taking, examination, and contextual clinical judgment.