Scholarlink

<p>To preliminarily evaluate agreement between large language models (LLMs) and ophthalmic clinicians regarding thyroid eye disease (TED) clinical decision-making, we developed a structured questionnaire for TED-related scenarios. A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians’ approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro).&#xa0;Agreement between LLMs and clinicians was assessed using similarity metrics, including Jaccard index and cosine similarity. Inter-physician agreement was additionally calculated as a reference baseline. The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician’s experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (<i>p</i> &lt; 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (<i>p</i> &lt; 0.001). LLMs exhibit partial alignment with clinician response patterns in a structured, questionnaire-based setting. However, these findings reflect performance under simplified and decontextualized conditions, further studies incorporating real patient data and open-ended clinical tasks are needed.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Comparative analysis of large language models and clinicians in thyroid eye disease using structured questionnaires

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月9日

查看全文
  • Shiqi Hui,
  • Lihua Luo,
  • Zhijia Hou,
  • Dongmei Li

摘要

To preliminarily evaluate agreement between large language models (LLMs) and ophthalmic clinicians regarding thyroid eye disease (TED) clinical decision-making, we developed a structured questionnaire for TED-related scenarios. A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians’ approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro). Agreement between LLMs and clinicians was assessed using similarity metrics, including Jaccard index and cosine similarity. Inter-physician agreement was additionally calculated as a reference baseline. The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician’s experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (p < 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (p < 0.001). LLMs exhibit partial alignment with clinician response patterns in a structured, questionnaire-based setting. However, these findings reflect performance under simplified and decontextualized conditions, further studies incorporating real patient data and open-ended clinical tasks are needed.