Background/purpose Large Language Models (LLMs) are used in clinical decision support across medical specialties, yet systematic benchmarking in Otorhinolaryngology–Head and Neck Surgery (OHNS) remains limited and heterogeneous. Emerging reporting standards such as TRIPOD-LLM & STARD-AI underscore the need for transparent, reproducible evaluations. This study provides a comprehensive, multi-model, subspecialty-level assessment of accuracy, guideline adherence, and safety using real-world cases in OHNS. Methods This cross-sectional evaluation analyzed five state-of-the-art LLMs—ChatGPT-5.1, Gemini 3 Pro, Grok 4, LLaMA 4, and DeepSeek V4-R1—using 250 anonymized OHNS patient cases across otology, rhinology, laryngology, head and neck oncology, and miscellaneous presentations. Each case included six clinical document types and was processed under a standardized prompting protocol. Two board-certified otorhinolaryngologists independently scored model outputs across six domains using a 6-point Likert scale. Results Composite decision-quality scores (mean of five 6-point Likert domains; 1 = very poor, 6 = excellent) differed among models (p < 0.001), with ChatGPT achieving the highest overall performance (5.72 ± 0.21), outperforming Gemini (p < 0.001), DeepSeek (p < 0.001), LLaMA (p < 0.001), and Grok (p < 0.001). Subspecialty-level analyses demonstrated consistent ranking across clinical domains, with firm performance in otology and head and neck oncology. Diagnostic accuracy (ChatGPT: 5.81 ± 0.17) and guideline adherence (5.77 ± 0.19) were strongly correlated (r = 0.62, p < 0.001). Safety-related concerns were rare; ChatGPT generated the fewest unsafe recommendations (0.4%), whereas Grok produced the most (2.4%), primarily due to omissions or radiologic misinterpretations. Inter-rater reliability was excellent (κ = 0.82), and internal consistency was high (α = 0.91). Conclusion ChatGPT-5.1 consistently outperformed Gemini, DeepSeek, LLaMA, and Grok across all subspecialties and clinical domains. These findings support the responsible adoption of LLMs as adjunct tools in otolaryngologic practice, while highlighting ongoing needs for multimodal integration and prospective validation.

Guideline-based benchmarking of large language models in otorhinolaryngology using 250 real-world cases

期刊论文
发表日期：

2026年3月14日

查看全文

Koray Tümüklü,
Behçet Günsoy

摘要

Background/purpose

Large Language Models (LLMs) are used in clinical decision support across medical specialties, yet systematic benchmarking in Otorhinolaryngology–Head and Neck Surgery (OHNS) remains limited and heterogeneous. Emerging reporting standards such as TRIPOD-LLM & STARD-AI underscore the need for transparent, reproducible evaluations. This study provides a comprehensive, multi-model, subspecialty-level assessment of accuracy, guideline adherence, and safety using real-world cases in OHNS.

Methods

This cross-sectional evaluation analyzed five state-of-the-art LLMs—ChatGPT-5.1, Gemini 3 Pro, Grok 4, LLaMA 4, and DeepSeek V₄-R₁—using 250 anonymized OHNS patient cases across otology, rhinology, laryngology, head and neck oncology, and miscellaneous presentations. Each case included six clinical document types and was processed under a standardized prompting protocol. Two board-certified otorhinolaryngologists independently scored model outputs across six domains using a 6-point Likert scale.

Results

Composite decision-quality scores (mean of five 6-point Likert domains; 1 = very poor, 6 = excellent) differed among models (p < 0.001), with ChatGPT achieving the highest overall performance (5.72 ± 0.21), outperforming Gemini (p < 0.001), DeepSeek (p < 0.001), LLaMA (p < 0.001), and Grok (p < 0.001). Subspecialty-level analyses demonstrated consistent ranking across clinical domains, with firm performance in otology and head and neck oncology. Diagnostic accuracy (ChatGPT: 5.81 ± 0.17) and guideline adherence (5.77 ± 0.19) were strongly correlated (r = 0.62, p < 0.001). Safety-related concerns were rare; ChatGPT generated the fewest unsafe recommendations (0.4%), whereas Grok produced the most (2.4%), primarily due to omissions or radiologic misinterpretations. Inter-rater reliability was excellent (κ = 0.82), and internal consistency was high (α = 0.91).

Conclusion

ChatGPT-5.1 consistently outperformed Gemini, DeepSeek, LLaMA, and Grok across all subspecialties and clinical domains. These findings support the responsible adoption of LLMs as adjunct tools in otolaryngologic practice, while highlighting ongoing needs for multimodal integration and prospective validation.