Scholarlink

Background <p>Medical large language models (LLMs) have demonstrated remarkable performance in answering medical examinations. However, the extent to which this high performance is transferable to medical questions in Spanish and from a Latin American country remains unexplored. This knowledge is crucial as LLM-based medical applications gain traction in Latin America.</p> Aims <p>To build a dataset of questions from medical examinations taken by Peruvian physicians pursuing specialty training; to fine-tune a LLM on this dataset; to evaluate and compare the performance in terms of accuracy between vanilla LLMs and the fine-tuned LLM.</p> Methods <p>We curated PeruMedQA, a multiple-choice question-answering (MCQA) dataset containing 8,380 questions spanning 12 specialties (2018–2025). We selected ten medical LLMs, including medgemma-4b-it and medgemma-27b-text-it, and developed zero-shot task-specific prompts to answer the questions. We employed parameter-efficient fine tuning (PEFT) and low-rank adaptation (LoRA) to fine-tune medgemma-4b-it utilizing all questions except those from 2025 (test set).</p> Results <p>Medgemma-27b showed the highest accuracy across all specialties, achieving the highest score of 89.29% in Psychiatry; yet, in two specialties, OctoMed-7B exhibited slight superiority: Neurosurgery with 77.25% and 77.38%, respectively; and Radiology with 76.13% and 77.30%, respectively. Across specialties, most LLMs with &lt; 10 billion parameters exhibited &lt; 50% of correct answers. The fine-tuned version of medgemma-4b-it emerged victorious against all LLMs with &lt; 10 billion parameters and rivaled a LLM with 70 billion parameters across various examinations.</p> Conclusions <p>For medical AI applications and research that require knowledge bases from Spanish-speaking countries and those exhibiting similar epidemiological profiles to Peru’s, interested parties should utilize medgemma-27b-text-it.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams—Dataset Construction and Evaluation

  • 期刊论文

  • 发表日期：

    2026年3月18日

查看全文
  • Rodrigo M. Carrillo-Larco,
  • Jesús Lovón-Melgarejo,
  • Manuel Castillo-Cara,
  • Gusseppe Bravo-Rocca

摘要

Background

Medical large language models (LLMs) have demonstrated remarkable performance in answering medical examinations. However, the extent to which this high performance is transferable to medical questions in Spanish and from a Latin American country remains unexplored. This knowledge is crucial as LLM-based medical applications gain traction in Latin America.

Aims

To build a dataset of questions from medical examinations taken by Peruvian physicians pursuing specialty training; to fine-tune a LLM on this dataset; to evaluate and compare the performance in terms of accuracy between vanilla LLMs and the fine-tuned LLM.

Methods

We curated PeruMedQA, a multiple-choice question-answering (MCQA) dataset containing 8,380 questions spanning 12 specialties (2018–2025). We selected ten medical LLMs, including medgemma-4b-it and medgemma-27b-text-it, and developed zero-shot task-specific prompts to answer the questions. We employed parameter-efficient fine tuning (PEFT) and low-rank adaptation (LoRA) to fine-tune medgemma-4b-it utilizing all questions except those from 2025 (test set).

Results

Medgemma-27b showed the highest accuracy across all specialties, achieving the highest score of 89.29% in Psychiatry; yet, in two specialties, OctoMed-7B exhibited slight superiority: Neurosurgery with 77.25% and 77.38%, respectively; and Radiology with 76.13% and 77.30%, respectively. Across specialties, most LLMs with < 10 billion parameters exhibited < 50% of correct answers. The fine-tuned version of medgemma-4b-it emerged victorious against all LLMs with < 10 billion parameters and rivaled a LLM with 70 billion parameters across various examinations.

Conclusions

For medical AI applications and research that require knowledge bases from Spanish-speaking countries and those exhibiting similar epidemiological profiles to Peru’s, interested parties should utilize medgemma-27b-text-it.