This study presents a comprehensive evaluation of large language model based medical NLP code generation for clinical trial eligibility screening. Using 51 criteria across 19 categories and 51 case reports, we assess systems that transform natural-language rules into FHIR-compliant structured representations and executable patient-retrieval logic. The benchmark requires participants to generate FHIR Bundles rather than only end-to-end code to promote transparency, standards alignment, and interoperability. Top-performing teams adopted strategies such as intermediate-schema decoupling, dynamic few-shot prompting, iterative refinement, structured prompt engineering, and agent-based synthesis. Results demonstrate that LLMs can generate clinically meaningful, standards-compliant code, while also highlighting the need for explicit semantic layers to ensure safety and interpretability. This task provides the first large-scale evaluation of medical NLP code generation grounded in FHIR Profiles and offers a foundation for future development of verifiable, trustworthy clinical AI systems. Additional details, datasets, and evaluation materials are available at the CHIP 2025 website: http://cips-chip.org.cn/2025/eval3 .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Overview of Medical NLP Code Generation with FHIR for Clinical Trial Screening

  • Liang Tao,
  • Lan Mi,
  • Chunxiao Wu,
  • Dong Wen,
  • Buzhou Tang,
  • Xiaoyan Zhang,
  • Hui Zong,
  • Zuofeng Li

摘要

This study presents a comprehensive evaluation of large language model based medical NLP code generation for clinical trial eligibility screening. Using 51 criteria across 19 categories and 51 case reports, we assess systems that transform natural-language rules into FHIR-compliant structured representations and executable patient-retrieval logic. The benchmark requires participants to generate FHIR Bundles rather than only end-to-end code to promote transparency, standards alignment, and interoperability. Top-performing teams adopted strategies such as intermediate-schema decoupling, dynamic few-shot prompting, iterative refinement, structured prompt engineering, and agent-based synthesis. Results demonstrate that LLMs can generate clinically meaningful, standards-compliant code, while also highlighting the need for explicit semantic layers to ensure safety and interpretability. This task provides the first large-scale evaluation of medical NLP code generation grounded in FHIR Profiles and offers a foundation for future development of verifiable, trustworthy clinical AI systems. Additional details, datasets, and evaluation materials are available at the CHIP 2025 website: http://cips-chip.org.cn/2025/eval3 .