Scholarlink

<p>Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a ‘passing’ score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (<i>P</i> &lt; 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Toward expert-level medical question answering with large language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年1月8日

查看全文
  • Karan Singhal,
  • Tao Tu,
  • Juraj Gottweis,
  • Rory Sayres,
  • Ellery Wulczyn,
  • Mohamed Amin,
  • Le Hou,
  • Kevin Clark,
  • Stephen R. Pfohl,
  • Heather Cole-Lewis,
  • Darlene Neal,
  • Qazi Mamunur Rashid,
  • Mike Schaekermann,
  • Amy Wang,
  • Dev Dash,
  • Jonathan H. Chen,
  • Nigam H. Shah,
  • Sami Lachgar,
  • Philip Andrew Mansfield,
  • Sushant Prakash,
  • Bradley Green,
  • Ewa Dominowska,
  • Blaise Agüera y Arcas,
  • Nenad Tomašev,
  • Yun Liu,
  • Renee Wong,
  • Christopher Semturs,
  • S. Sara Mahdavi,
  • Joelle K. Barral,
  • Dale R. Webster,
  • Greg S. Corrado,
  • Yossi Matias,
  • Shekoofeh Azizi,
  • Alan Karthikesalingam,
  • Vivek Natarajan

摘要

Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a ‘passing’ score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.