Scholarlink

<p>The aim of this study is to compare the accuracy levels of different LLMs on multiple-choice questions (MCQs) related to the field of restorative dentistry in the dental specialisation exam (DUS) administered in Turkey and to evaluate response consistency (test–retest reliability) between two sessions. In this study, 127 text-based MCQs related to restorative dentistry, without visual material, were used in the DUS. The responses from the ChatGPT-5.1, Gemini 2.5 Pro, Microsoft Copilot, and DeepSeek-v3.2 models were evaluated at two different time points (T1 and T2) and coded as correct/incorrect according to the official answer key. The accuracy of the models was analysed using Cochran’s Q test, and the inter-session variation was analysed using the McNemar test. Test–retest response reliability was assessed using Cohen’s Kappa coefficient and percentage agreement rates. ChatGPT-5.1 achieved the highest accuracy rate in both sessions. However, no statistically significant difference was found between the models’ T1 and T2 accuracy rates. No significant performance difference was identified between the models in subcategory-based analyses either. Test-retest analyses revealed that, despite the accuracy rates, response consistency may vary depending on the model, and Cohen’s Kappa values ranged from low to moderate levels. It is thought that LLMs can answer questions about theoretical knowledge in the field of restorative dentistry with high accuracy, but may show limitations in terms of time-dependent response consistency. These findings suggest that while LLMs have potential as supportive tools in dental education, their use without human oversight requires careful consideration.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Temporal consistency of large language model responses to restorative dentistry questions from the Turkish dental specialty examination

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月7日

查看全文
  • Kemal Furkan Güdül,
  • Baturalp Arslan

摘要

The aim of this study is to compare the accuracy levels of different LLMs on multiple-choice questions (MCQs) related to the field of restorative dentistry in the dental specialisation exam (DUS) administered in Turkey and to evaluate response consistency (test–retest reliability) between two sessions. In this study, 127 text-based MCQs related to restorative dentistry, without visual material, were used in the DUS. The responses from the ChatGPT-5.1, Gemini 2.5 Pro, Microsoft Copilot, and DeepSeek-v3.2 models were evaluated at two different time points (T1 and T2) and coded as correct/incorrect according to the official answer key. The accuracy of the models was analysed using Cochran’s Q test, and the inter-session variation was analysed using the McNemar test. Test–retest response reliability was assessed using Cohen’s Kappa coefficient and percentage agreement rates. ChatGPT-5.1 achieved the highest accuracy rate in both sessions. However, no statistically significant difference was found between the models’ T1 and T2 accuracy rates. No significant performance difference was identified between the models in subcategory-based analyses either. Test-retest analyses revealed that, despite the accuracy rates, response consistency may vary depending on the model, and Cohen’s Kappa values ranged from low to moderate levels. It is thought that LLMs can answer questions about theoretical knowledge in the field of restorative dentistry with high accuracy, but may show limitations in terms of time-dependent response consistency. These findings suggest that while LLMs have potential as supportive tools in dental education, their use without human oversight requires careful consideration.