Scholarlink

Objectives <p>To assess the concordance and diagnosis acceptability of differential diagnoses (DDs) generated by artificial intelligence (AI) chatbots compared to expert neonatology physicians.</p> Methods <p>Thirteen clinical cases from neonatology were developed, each with five expert-defined differential diagnoses. Seven AI chatbots (ChatGPT, Claude, Copilot, Deepseek, Gemini, Grok, Perplexity) were queried to provide their most probable diagnosis along with five DDs per case. Four neonatologists evaluated the chatbot responses. Concordance scores were assigned based on the ranking of chatbot diagnoses against expert-defined diagnoses. Diagnosis acceptability scores were based on the number of DDs deemed clinically acceptable.</p> Results <p>There were no statistically significant differences in the concordance (<i>P</i> = 0.205) or diagnosis acceptability (<i>P</i> = 0.297) scores across the seven chatbots. Chatbots had statistically significant discrepancies when compared to hypothetical values of 5, 4.5, and 4, with none of the chatbots achieving the desired concordance score of 80%. However, in terms of diagnosis acceptability, ChatGPT, Claude, and Grok had scores similar to 90% diagnosis acceptability, while Copilot, Deepseek, Gemini, and Perplexity had 80% diagnosis acceptability.</p> Conclusions <p>There were no significant differences in concordance and diagnosis acceptability scores across the seven chatbots. While none achieved the desired concordance of 80%, several models showed near 80–90% diagnosis acceptability. Hence, integration of chatbots should be done with caution.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Assessment of Generative AI Chatbots for Neonatal Diagnostic Reasoning: A Comparative Concordance and Diagnosis Acceptability Analysis

  • 期刊论文

  • 发表日期：

    2026年1月13日

查看全文
  • Gaurav Gupta,
  • Somashekhar Nimbalkar,
  • Rajesh Kumar,
  • Poonam Singh,
  • Himel Mondal

摘要

Objectives

To assess the concordance and diagnosis acceptability of differential diagnoses (DDs) generated by artificial intelligence (AI) chatbots compared to expert neonatology physicians.

Methods

Thirteen clinical cases from neonatology were developed, each with five expert-defined differential diagnoses. Seven AI chatbots (ChatGPT, Claude, Copilot, Deepseek, Gemini, Grok, Perplexity) were queried to provide their most probable diagnosis along with five DDs per case. Four neonatologists evaluated the chatbot responses. Concordance scores were assigned based on the ranking of chatbot diagnoses against expert-defined diagnoses. Diagnosis acceptability scores were based on the number of DDs deemed clinically acceptable.

Results

There were no statistically significant differences in the concordance (P = 0.205) or diagnosis acceptability (P = 0.297) scores across the seven chatbots. Chatbots had statistically significant discrepancies when compared to hypothetical values of 5, 4.5, and 4, with none of the chatbots achieving the desired concordance score of 80%. However, in terms of diagnosis acceptability, ChatGPT, Claude, and Grok had scores similar to 90% diagnosis acceptability, while Copilot, Deepseek, Gemini, and Perplexity had 80% diagnosis acceptability.

Conclusions

There were no significant differences in concordance and diagnosis acceptability scores across the seven chatbots. While none achieved the desired concordance of 80%, several models showed near 80–90% diagnosis acceptability. Hence, integration of chatbots should be done with caution.