Scholarlink

Purpose <p>This study aims to evaluate two advanced reasoning LLMs in generating treatment recommendations for real-world gastrointestinal stromal tumor (GIST) cases and assess their concordance with multidisciplinary team (MDT) decisions at a certified tertiary sarcoma center.</p> Methods <p>Sixty-five real-world GIST cases from a tertiary sarcoma center were used to compare two advanced reasoning models—OpenAI o1 and DeepSeek-R1. Recommendations were generated using a multi-expert prompting strategy with current clinical guidelines as context. Five sarcoma specialists and an independent LLM (Mistral AI) evaluated alignment with MDT decisions and guideline concordance.</p> Results <p>OpenAI o1 achieved higher concordance with MDT decisions than DeepSeek-R1 (76.9% vs. 53.8%, <i>p</i> &lt; 0.001) and more recommendations aligned with either MDT or guidelines (80.0% vs. 63.1%, <i>p</i> = 0.005). Inter-rater reliability among human evaluators was excellent (ICC = 0.929). The LLM judge’s evaluations showed moderate agreement with human assessments (κ = 0.647). OpenAI o1 responses were significantly longer than those of DeepSeek-R1 and MDT records.</p> Conclusions <p>OpenAI o1 outperformed DeepSeek-R1 in generating clinically relevant GIST therapy recommendations. The study highlights the feasibility of using LLMs both as decision support tools and as evaluators (“LLM-as-a-judge”) in oncology, while emphasizing the need for expert oversight in clinical deployment.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating reasoning models for therapy recommendations in gastrointestinal stromal tumors: expert and LLM-based evaluations of OpenAI o1 and DeepSeek-R1

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月18日

查看全文
  • Melissa Harbrücker,
  • Franka Menge,
  • Arshia Taebi,
  • Dominik Nörenberg,
  • Tobias Speer,
  • Christoph Reißfelder,
  • Peter Hohenberger,
  • Jens Jakob,
  • Chengpeng Li,
  • Cui Yang

摘要

Purpose

This study aims to evaluate two advanced reasoning LLMs in generating treatment recommendations for real-world gastrointestinal stromal tumor (GIST) cases and assess their concordance with multidisciplinary team (MDT) decisions at a certified tertiary sarcoma center.

Methods

Sixty-five real-world GIST cases from a tertiary sarcoma center were used to compare two advanced reasoning models—OpenAI o1 and DeepSeek-R1. Recommendations were generated using a multi-expert prompting strategy with current clinical guidelines as context. Five sarcoma specialists and an independent LLM (Mistral AI) evaluated alignment with MDT decisions and guideline concordance.

Results

OpenAI o1 achieved higher concordance with MDT decisions than DeepSeek-R1 (76.9% vs. 53.8%, p < 0.001) and more recommendations aligned with either MDT or guidelines (80.0% vs. 63.1%, p = 0.005). Inter-rater reliability among human evaluators was excellent (ICC = 0.929). The LLM judge’s evaluations showed moderate agreement with human assessments (κ = 0.647). OpenAI o1 responses were significantly longer than those of DeepSeek-R1 and MDT records.

Conclusions

OpenAI o1 outperformed DeepSeek-R1 in generating clinically relevant GIST therapy recommendations. The study highlights the feasibility of using LLMs both as decision support tools and as evaluators (“LLM-as-a-judge”) in oncology, while emphasizing the need for expert oversight in clinical deployment.