Scholarlink

Background <p>The integration of artificial intelligence (AI) in healthcare and medical education has advanced rapidly, with conversational AI systems gaining attention for their potential in academic assessment and clinical reasoning. This study aimed to evaluate AI chatbots’ performance on restorative dentistry questions from the Turkish Dental Specialty Examination (DUS), a high-stakes national exam assessing theoretical and clinical knowledge.</p> Methods <p>An in silico, cross-sectional, comparative design was employed. A total of 190 multiple-choice questions (MCQs) from 19 DUS sessions between 2012 and 2025 were obtained from the Assessment, Selection, and Placement Center (ÖSYM) website. After excluding annulled items, 188 questions were analyzed. Eight AI chatbots (ChatGPT-3.5, ChatGPT-4o Free, ChatGPT-4o Plus, Claude Sonnet 4, Microsoft Copilot, DeepSeek, Gemini 1.5, and Gemini Advanced) were tested using a standardized single-attempt protocol in Turkish. Performance measures included accuracy, response length, and response time. Questions were categorized by year, content domain, and length for subgroup analyses. Statistical analyses were conducted in Python using standard libraries. Descriptive statistics and Pearson’s correlation were calculated, while comparisons involved the Shapiro-Wilk test, Levene’s test, Kruskal-Wallis test, and Dunn’s post hoc test, with significance set at <i>p</i> &lt; 0.05.</p> Results <p>No significant difference was found in overall accuracy (<i>p</i> = 0.18). However, response time and word count differed significantly (<i>p</i> &lt; 0.001). Gemini Advanced showed the highest accuracy (96.28%), followed by ChatGPT-4o Plus (93.62%). Gemini 1.5 produced the longest yet fastest responses, while DeepSeek had the lowest accuracy and slowest responses. Accuracy remained stable across years but varied by topic, with lower performance in complex areas such as cavity preparation. In case-based questions, Gemini Advanced, Gemini 1.5, and ChatGPT-4o Plus achieved 100% accuracy. Performance in image-based questions was inconsistent, underscoring limitations in visual reasoning.</p> Conclusions <p>AI chatbots demonstrated high accuracy in answering restorative dentistry exam questions, with Gemini Advanced, ChatGPT-4o Plus, and Gemini 1.5 showing superior performance. Despite differences in response time and content length, their potential as supplementary tools in dental education is evident, warranting further validation across specialties and contexts.</p> Trial registration <p>Not applicable.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Can artificial intelligence chatbots think like dentists? A comparative analysis based on dental specialty examination questions in restorative dentistry

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月6日

查看全文
  • Merve Haberal,
  • Dilek Hançerlioğulları

摘要

Background

The integration of artificial intelligence (AI) in healthcare and medical education has advanced rapidly, with conversational AI systems gaining attention for their potential in academic assessment and clinical reasoning. This study aimed to evaluate AI chatbots’ performance on restorative dentistry questions from the Turkish Dental Specialty Examination (DUS), a high-stakes national exam assessing theoretical and clinical knowledge.

Methods

An in silico, cross-sectional, comparative design was employed. A total of 190 multiple-choice questions (MCQs) from 19 DUS sessions between 2012 and 2025 were obtained from the Assessment, Selection, and Placement Center (ÖSYM) website. After excluding annulled items, 188 questions were analyzed. Eight AI chatbots (ChatGPT-3.5, ChatGPT-4o Free, ChatGPT-4o Plus, Claude Sonnet 4, Microsoft Copilot, DeepSeek, Gemini 1.5, and Gemini Advanced) were tested using a standardized single-attempt protocol in Turkish. Performance measures included accuracy, response length, and response time. Questions were categorized by year, content domain, and length for subgroup analyses. Statistical analyses were conducted in Python using standard libraries. Descriptive statistics and Pearson’s correlation were calculated, while comparisons involved the Shapiro-Wilk test, Levene’s test, Kruskal-Wallis test, and Dunn’s post hoc test, with significance set at p < 0.05.

Results

No significant difference was found in overall accuracy (p = 0.18). However, response time and word count differed significantly (p < 0.001). Gemini Advanced showed the highest accuracy (96.28%), followed by ChatGPT-4o Plus (93.62%). Gemini 1.5 produced the longest yet fastest responses, while DeepSeek had the lowest accuracy and slowest responses. Accuracy remained stable across years but varied by topic, with lower performance in complex areas such as cavity preparation. In case-based questions, Gemini Advanced, Gemini 1.5, and ChatGPT-4o Plus achieved 100% accuracy. Performance in image-based questions was inconsistent, underscoring limitations in visual reasoning.

Conclusions

AI chatbots demonstrated high accuracy in answering restorative dentistry exam questions, with Gemini Advanced, ChatGPT-4o Plus, and Gemini 1.5 showing superior performance. Despite differences in response time and content length, their potential as supplementary tools in dental education is evident, warranting further validation across specialties and contexts.

Trial registration

Not applicable.