Scholarlink

<p>Large language models (LLMs) are increasingly utilized in medical education and practice, yet their application in niche fields such as radiation oncology remains underexplored. This study evaluates and compares the performance of OpenAI’s GPT-4o and Baidu’s ERNIE Bot in a Chinese-language radiation oncology examination. We employed the Chinese National Health Professional Technical Qualification Examination (Intermediate Level) for Radiation Oncology, using a question bank of 1128 items across four sections: Basic Knowledge, Relevant Knowledge, Specialized Knowledge, and Practice Competence. A passing score required an accuracy rate of 60% or higher in all sections. The models’ responses were assessed for accuracy against standard answers, with key metrics including overall accuracy, section-specific performance, case analysis performance, and accuracy consensus between the models. The overall accuracy rates were 79.3% for GPT-4o and 76.9% for ERNIE Bot (<Emphasis Type="BoldItalic">p</Emphasis> = 0.154). Across the four sections, GPT-4o achieved accuracy rates of 82.1%, 84.6%, 78.6%, and 60.9%, respectively, while ERNIE Bot achieved 81.6%, 73.9%, 77.9%, and 69.0%. In the Relevant Knowledge section, GPT-4o achieved significantly higher accuracy (<Emphasis Type="BoldItalic">p</Emphasis> = 0.002), while no significant differences were found in the other three sections. Across various question types—including single-choice, multiple-answer, case analysis, non-case analysis, and different content areas of case analysis—both models exhibited satisfied accuracy, and ERNIE Bot achieved accuracy rates that were comparable to GPT-4o. The accuracy consensus between the two models was 84.5%, significantly exceeding the individual accuracy rates of GPT-4o (<Emphasis Type="BoldItalic">p</Emphasis> = 0.003) and ERNIE Bot (<Emphasis Type="BoldItalic">p</Emphasis> &lt; 0.001). Both GPT-4o and ERNIE Bot successfully passed the highly specialized Chinese-language medical examination in radiation oncology and demonstrated comparable performance. This study provides valuable insights into the application of LLMs in Chinese medical education. These findings support the integration of LLMs in medical education and training within specialized, non-English-speaking contexts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Comparative Analysis of GPT-4o and ERNIE Bot in a Chinese Radiation Oncology Exam

  • 期刊论文

  • 发表日期：

    2025年5月26日

查看全文
  • Weiping Wang,
  • Jingxuan Fu,
  • Yiming Zhang,
  • Ke Hu

摘要

Large language models (LLMs) are increasingly utilized in medical education and practice, yet their application in niche fields such as radiation oncology remains underexplored. This study evaluates and compares the performance of OpenAI’s GPT-4o and Baidu’s ERNIE Bot in a Chinese-language radiation oncology examination. We employed the Chinese National Health Professional Technical Qualification Examination (Intermediate Level) for Radiation Oncology, using a question bank of 1128 items across four sections: Basic Knowledge, Relevant Knowledge, Specialized Knowledge, and Practice Competence. A passing score required an accuracy rate of 60% or higher in all sections. The models’ responses were assessed for accuracy against standard answers, with key metrics including overall accuracy, section-specific performance, case analysis performance, and accuracy consensus between the models. The overall accuracy rates were 79.3% for GPT-4o and 76.9% for ERNIE Bot (p = 0.154). Across the four sections, GPT-4o achieved accuracy rates of 82.1%, 84.6%, 78.6%, and 60.9%, respectively, while ERNIE Bot achieved 81.6%, 73.9%, 77.9%, and 69.0%. In the Relevant Knowledge section, GPT-4o achieved significantly higher accuracy (p = 0.002), while no significant differences were found in the other three sections. Across various question types—including single-choice, multiple-answer, case analysis, non-case analysis, and different content areas of case analysis—both models exhibited satisfied accuracy, and ERNIE Bot achieved accuracy rates that were comparable to GPT-4o. The accuracy consensus between the two models was 84.5%, significantly exceeding the individual accuracy rates of GPT-4o (p = 0.003) and ERNIE Bot (p < 0.001). Both GPT-4o and ERNIE Bot successfully passed the highly specialized Chinese-language medical examination in radiation oncology and demonstrated comparable performance. This study provides valuable insights into the application of LLMs in Chinese medical education. These findings support the integration of LLMs in medical education and training within specialized, non-English-speaking contexts.