Scholarlink

Background <p>Direct clinical uses of large language models (LLMs) remain controversial, partly because of the lack of methodological rigor in assessing their risks and benefits in medicine.</p> Methods <p>We developed Medieval, a multidisciplinary, randomized, and blinded expert evaluation framework. A ten-point Dreyfus-based scoring scale linked to career stages of human physicians was designed to reflect response qualities. Seven advanced LLMs or their distilled versions that were released within a short time-frame ( ≤ 45 days) in early 2025 were tested. Incidence of fabricated medical facts were documented. Linear mixed-effects models and variance-stabilizing Bayesian generalized linear mixed models were employed to perform statistical analyses.</p> Results <p>We first develop a high-quality question bank comprising 685 real and simulated clinical cases across 13 specialties. An expert panel of 27 clinicians (average years of services: 25.9) evaluated the 4795 model responses. We show that these LLM ratings (n = 9856) have excellent reliability (intraclass correlation coefficients <InlineEquation ID="IEq1"><EquationSource Format="TEX">\( &gt; \)</EquationSource><EquationSource Format="MATHML"><math><mo>&gt;</mo></math></EquationSource></InlineEquation>0.9). Among the seven LLMs tested, Gemini 2.0 Flash achieved the highest raw scores. However, after adjusting for confounders, DeepSeek-R1 was the top-performing model with a mean score of 6.36 (95% confidence interval 6.03 − 6.69), a performance level equivalent to an early-career physician. Despite these strengths, 3–19% LLM responses were rated as incompetent and 40 instances of LLM hallucination were also identified.</p> Conclusions <p>Our study shows that in spite of LLMs’ substantial potentials in medicine, their unguarded clinical application could present serious risks, which must be continuously monitored by human expert panels. The evaluation framework developed and validated in this study will facilitate such efforts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multidisciplinary blinded randomized expert evaluation of large language models for clinical diagnosis and management

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月15日

查看全文
  • Peikai Chen,
  • Jifu Cai,
  • Jiaying Zhou,
  • Shaoxi Chen,
  • Chenguang Xu,
  • Lihua Yuan,
  • Xiaoying Dai,
  • Xiaowei Chen,
  • Yanzhe Wei,
  • Xia Li,
  • Shaofeng Gong,
  • Xiaolong Liang,
  • Jiancheng Yang,
  • Jun Jin,
  • Kanglin Dai,
  • Yuzhen Cui,
  • Guan-Ming Kuang,
  • Jiansheng Xie,
  • Libing Luo,
  • Haibing Xiao,
  • Shijie Yin,
  • Jun Yang,
  • Yulan Yan,
  • Jianliang Chen,
  • Yihua Chen,
  • Qianshen Zhang,
  • Qingshan Zhou,
  • Lina Zhao,
  • Min Wu,
  • Xin Tang,
  • Lei Rong,
  • Zanxin Wang,
  • Weifu Qiu,
  • Yanli Wang,
  • Liwen Cui,
  • Xiangyang Li,
  • Yong Hu,
  • Huiren Tao,
  • Nan Wu,
  • David J. H. Shih,
  • Pearl Pai,
  • Minxin Wei,
  • Michael Kai-tsun To,
  • Kenneth M. C. Cheung

摘要

Background

Direct clinical uses of large language models (LLMs) remain controversial, partly because of the lack of methodological rigor in assessing their risks and benefits in medicine.

Methods

We developed Medieval, a multidisciplinary, randomized, and blinded expert evaluation framework. A ten-point Dreyfus-based scoring scale linked to career stages of human physicians was designed to reflect response qualities. Seven advanced LLMs or their distilled versions that were released within a short time-frame ( ≤ 45 days) in early 2025 were tested. Incidence of fabricated medical facts were documented. Linear mixed-effects models and variance-stabilizing Bayesian generalized linear mixed models were employed to perform statistical analyses.

Results

We first develop a high-quality question bank comprising 685 real and simulated clinical cases across 13 specialties. An expert panel of 27 clinicians (average years of services: 25.9) evaluated the 4795 model responses. We show that these LLM ratings (n = 9856) have excellent reliability (intraclass correlation coefficients \( > \)>0.9). Among the seven LLMs tested, Gemini 2.0 Flash achieved the highest raw scores. However, after adjusting for confounders, DeepSeek-R1 was the top-performing model with a mean score of 6.36 (95% confidence interval 6.03 − 6.69), a performance level equivalent to an early-career physician. Despite these strengths, 3–19% LLM responses were rated as incompetent and 40 instances of LLM hallucination were also identified.

Conclusions

Our study shows that in spite of LLMs’ substantial potentials in medicine, their unguarded clinical application could present serious risks, which must be continuously monitored by human expert panels. The evaluation framework developed and validated in this study will facilitate such efforts.