Scholarlink

Objective <p>Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown.</p> Methods <p>We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021–2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar’s test.</p> Results <p>Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold.</p> Conclusions <p>Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications

  • 期刊论文

  • 发表日期：

    2026年4月27日

查看全文
  • Kentaro Kasa,
  • Daisuke Yoshimaru,
  • Hiroki Ohta,
  • Takao Ohki,
  • Hirotaka James Okano

摘要

Objective

Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown.

Methods

We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021–2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar’s test.

Results

Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold.

Conclusions

Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning.