Scholarlink

<p>While multimodal large language models (LLMs) demonstrate significant potential in healthcare applications, their clinical utility is difficult to appraise. Current evaluations of medical-assisting LLMs are often limited by sparse human expertise, narrow specialty scope, and reliance on multiple-choice benchmarks or synthetic vignettes, which can inflate performance and obscure clinical utility. We conducted a multicenter, multidisciplinary study in which more than 400 physicians—spanning seven specialties, varied experience levels, and multiple geographic settings—evaluated LLM-generated free-text responses to real, de-identified clinical cases. In a matched-control design, we also deployed an equivalent number of AI agents configured to mirror physician characteristics to examine whether automated evaluators can supplement or replace human assessment. Our results demonstrated that physician assessments exhibited substantial heterogeneity by clinical seniority and practice environment, leading to notable shifts in relative model rankings across cohorts. While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation. Instead, they promise assistive tools that can triage or pre-screen outputs to reduce human burden.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月2日

查看全文
  • Peilun Shi,
  • Jian Li,
  • Ziqi Yang,
  • Yuntai Cao,
  • Lie Ju,
  • Haihua Bao,
  • Hao Xu,
  • Yichen Fan,
  • Tianmin Ren,
  • Yiyang Xiao,
  • Jinjing Zhu,
  • Junhong Chen,
  • Yong Dong,
  • Xun Hu,
  • Yaliang Tong,
  • Zhan Wang,
  • Yuchen Xiang,
  • Jinshi Zhao,
  • Junlin Zhou,
  • Burak Temelkuran,
  • Yukun Zhou,
  • Long Jin,
  • Chenguang Li,
  • Meiyun Wang,
  • Hui Xiong,
  • Pearse A. Keane,
  • Junbo Ge,
  • Ningli Wang

摘要

While multimodal large language models (LLMs) demonstrate significant potential in healthcare applications, their clinical utility is difficult to appraise. Current evaluations of medical-assisting LLMs are often limited by sparse human expertise, narrow specialty scope, and reliance on multiple-choice benchmarks or synthetic vignettes, which can inflate performance and obscure clinical utility. We conducted a multicenter, multidisciplinary study in which more than 400 physicians—spanning seven specialties, varied experience levels, and multiple geographic settings—evaluated LLM-generated free-text responses to real, de-identified clinical cases. In a matched-control design, we also deployed an equivalent number of AI agents configured to mirror physician characteristics to examine whether automated evaluators can supplement or replace human assessment. Our results demonstrated that physician assessments exhibited substantial heterogeneity by clinical seniority and practice environment, leading to notable shifts in relative model rankings across cohorts. While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation. Instead, they promise assistive tools that can triage or pre-screen outputs to reduce human burden.