Scholarlink

Background <p>Large language models (LLMs) have shown promise in medical education and assessment, but their performance on highly specialized, context-specific exams remains under-explored. This study aimed to evaluate the accuracy of five advanced LLMs – Qwen3, DeepSeek-R1-0528, ChatGPT o3-pro, Gemini 2.5 pro, and Claude 4.0 – in answering oral and maxillofacial surgery multiple-choice questions, and compare their performance.</p> Methods <p>A total of 110 oral and maxillofacial surgery questions were collected from the 2013–2024 Chinese National Medical Licensing Examination (CNMLE), including 80 specialized knowledge questions and 30 case analysis questions. All questions were presented in Chinese to each LLM, and the correctness of each model’s first answer was recorded. We also attempted to verify the authenticity of any references provided in the models’ answers. Performance differences among models were analyzed with a significance level of <i>P</i> &lt; 0.05.</p> Results <p>All models achieved higher accuracy on knowledge-based questions than on case analysis questions. Qwen3 and DeepSeek showed no significant within-model accuracy difference between question types, whereas ChatGPT, Gemini, and Claude had significantly lower accuracy on case-based questions (<i>P</i> &lt; 0.05). Across questions on tooth extraction, maxillofacial trauma/infection, and maxillofacial tumors, the models performed well (overall accuracy 87.3%–95.5%) with no significant inter-model differences. Qwen3 achieved the highest overall accuracy (95.5%), followed by DeepSeek (94.5%), ChatGPT (89.1%), and Gemini and Claude (87.3% each). However, 37.7%–43.9% of the references provided by the models were not verifiable, with no significant difference among the models.</p> Conclusion <p>LLMs demonstrated high accuracy on oral and maxillofacial surgery exam questions, indicating great potential as assistive tools in medical education and examinations. Nevertheless, their performance in complex case reasoning is notably lower and a substantial portion of their cited references are spurious. These findings highlight the need for further improvements in LLMs’ clinical reasoning capabilities and for reducing reference hallucinations before they can be reliably used in unsupervised clinical decision-making.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Performance of five large language models in oral and maxillofacial surgery exam questions: a comparative study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月6日

查看全文
  • Lisi Liu,
  • Ke Ma,
  • Yansong Wang

摘要

Background

Large language models (LLMs) have shown promise in medical education and assessment, but their performance on highly specialized, context-specific exams remains under-explored. This study aimed to evaluate the accuracy of five advanced LLMs – Qwen3, DeepSeek-R1-0528, ChatGPT o3-pro, Gemini 2.5 pro, and Claude 4.0 – in answering oral and maxillofacial surgery multiple-choice questions, and compare their performance.

Methods

A total of 110 oral and maxillofacial surgery questions were collected from the 2013–2024 Chinese National Medical Licensing Examination (CNMLE), including 80 specialized knowledge questions and 30 case analysis questions. All questions were presented in Chinese to each LLM, and the correctness of each model’s first answer was recorded. We also attempted to verify the authenticity of any references provided in the models’ answers. Performance differences among models were analyzed with a significance level of P < 0.05.

Results

All models achieved higher accuracy on knowledge-based questions than on case analysis questions. Qwen3 and DeepSeek showed no significant within-model accuracy difference between question types, whereas ChatGPT, Gemini, and Claude had significantly lower accuracy on case-based questions (P < 0.05). Across questions on tooth extraction, maxillofacial trauma/infection, and maxillofacial tumors, the models performed well (overall accuracy 87.3%–95.5%) with no significant inter-model differences. Qwen3 achieved the highest overall accuracy (95.5%), followed by DeepSeek (94.5%), ChatGPT (89.1%), and Gemini and Claude (87.3% each). However, 37.7%–43.9% of the references provided by the models were not verifiable, with no significant difference among the models.

Conclusion

LLMs demonstrated high accuracy on oral and maxillofacial surgery exam questions, indicating great potential as assistive tools in medical education and examinations. Nevertheless, their performance in complex case reasoning is notably lower and a substantial portion of their cited references are spurious. These findings highlight the need for further improvements in LLMs’ clinical reasoning capabilities and for reducing reference hallucinations before they can be reliably used in unsupervised clinical decision-making.