Accurate prescribing is vital for patient safety, yet errors can cause severe harm. Large language model (LLM)-based clinical decision support systems (CDSS) show promise in detecting such errors, but their performance in complex settings like ICU polypharmacy remains underexplored. Current evaluation datasets are often oversimplified, with limited error types and poor public availability. Moreover, class imbalance and rarity of key medication errors complicate reliable model evaluation. To address these challenges, we propose a hybrid LLM-human error injection framework and construct MIMIC-RxBench, the first benchmark for prescription error classification based on real ICU discharge records. It covers seven error types plus a “No Error” category, with strict controls on plausibility, context complexity, and sample diversity, substantially enhancing both difficulty and granularity. We benchmark various LLMs and prompt strategies, finding that while performance improves with model size, the best model reaches only a 0.462 micro-F1 score, underscoring the challenge posed by this benchmark. Models with stronger reasoning or medical specialization perform better, and prompt effectiveness varies across model types.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MIMIC-RxBench: Benchmarking Large Language Models for Prescription Error Classification

  • Shuanglin Zu,
  • Yanhong Li,
  • Yibing Zhan,
  • Dapeng Tao

摘要

Accurate prescribing is vital for patient safety, yet errors can cause severe harm. Large language model (LLM)-based clinical decision support systems (CDSS) show promise in detecting such errors, but their performance in complex settings like ICU polypharmacy remains underexplored. Current evaluation datasets are often oversimplified, with limited error types and poor public availability. Moreover, class imbalance and rarity of key medication errors complicate reliable model evaluation. To address these challenges, we propose a hybrid LLM-human error injection framework and construct MIMIC-RxBench, the first benchmark for prescription error classification based on real ICU discharge records. It covers seven error types plus a “No Error” category, with strict controls on plausibility, context complexity, and sample diversity, substantially enhancing both difficulty and granularity. We benchmark various LLMs and prompt strategies, finding that while performance improves with model size, the best model reaches only a 0.462 micro-F1 score, underscoring the challenge posed by this benchmark. Models with stronger reasoning or medical specialization perform better, and prompt effectiveness varies across model types.