Scholarlink

Purpose <p>Large Language Models (LLMs), such as ChatGPT, are increasingly used in medical education, particularly for training clinical reasoning. Script Concordance Tests (SCTs) are valuable tools for assessing reasoning under uncertainty. Our objective was to evaluate the performance of four ChatGPT versions (3.5, 4, 4o, and 5) on SCTs compared to Geriatric Medicine experts.</p> Methods <p>Twenty validated SCT questions covering diverse geriatric topics were administered to ChatGPT-3.5, 4, 4o, and 5, as well as a panel of 43 experts. SCTs assess responses on a five-point Likert scale, and scores were calculated using aggregate expert answers as the reference. Performance was analyzed based on alignment with expert majority responses.</p> Results <p>ChatGPT-3.5 achieved a score of 8/20, aligning with the majority of experts on 15% of responses. ChatGPT-4 and -5 demonstrated significantly improved performance, scoring 15.6/20 and 15.2/20, respectively, and aligning with the majority on 45% of responses. ChatGPT 4o scored 13.6/20, slightly underperforming GPT-4, potentially due to differences in its architecture or training data. While GPT-4 and 5 excelled in reasoning and coherence, GPT-3.5 struggled with nuanced clinical scenarios.</p> Conclusion <p>ChatGPT-4 and -5 performed comparably to geriatric experts in SCT-based reasoning, highlighting its potential for medical education. It can simulate case discussions, enhance autonomous learning, and support formative assessments. However, limitations such as variability across versions and question-specific challenges emphasize the need for further evaluation of LLMs across broader clinical domains. These findings support the integration of advanced LLMs into medical education to complement traditional teaching methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating how different versions of ChatGPT align with expert opinions on geriatric script concordance tests

  • 期刊论文

  • 发表日期：

    2025年10月20日

查看全文
  • Matthieu Lilamand,
  • Théodore Decaix,
  • Pierre-Antoine Gourraud,
  • Richard Dufour,
  • Gaëtan Gavazzi,
  • Jacques Boaddaert,
  • Claire Roubaud-Baudron,
  • Hélène Vallet,
  • Cédric Annweiler,
  • Guillaume Chapelet

摘要

Purpose

Large Language Models (LLMs), such as ChatGPT, are increasingly used in medical education, particularly for training clinical reasoning. Script Concordance Tests (SCTs) are valuable tools for assessing reasoning under uncertainty. Our objective was to evaluate the performance of four ChatGPT versions (3.5, 4, 4o, and 5) on SCTs compared to Geriatric Medicine experts.

Methods

Twenty validated SCT questions covering diverse geriatric topics were administered to ChatGPT-3.5, 4, 4o, and 5, as well as a panel of 43 experts. SCTs assess responses on a five-point Likert scale, and scores were calculated using aggregate expert answers as the reference. Performance was analyzed based on alignment with expert majority responses.

Results

ChatGPT-3.5 achieved a score of 8/20, aligning with the majority of experts on 15% of responses. ChatGPT-4 and -5 demonstrated significantly improved performance, scoring 15.6/20 and 15.2/20, respectively, and aligning with the majority on 45% of responses. ChatGPT 4o scored 13.6/20, slightly underperforming GPT-4, potentially due to differences in its architecture or training data. While GPT-4 and 5 excelled in reasoning and coherence, GPT-3.5 struggled with nuanced clinical scenarios.

Conclusion

ChatGPT-4 and -5 performed comparably to geriatric experts in SCT-based reasoning, highlighting its potential for medical education. It can simulate case discussions, enhance autonomous learning, and support formative assessments. However, limitations such as variability across versions and question-specific challenges emphasize the need for further evaluation of LLMs across broader clinical domains. These findings support the integration of advanced LLMs into medical education to complement traditional teaching methods.