Overview of Medical NLP Code Generation with FHIR for Clinical Trial Screening
摘要
This study presents a comprehensive evaluation of large language model based medical NLP code generation for clinical trial eligibility screening. Using 51 criteria across 19 categories and 51 case reports, we assess systems that transform natural-language rules into FHIR-compliant structured representations and executable patient-retrieval logic. The benchmark requires participants to generate FHIR Bundles rather than only end-to-end code to promote transparency, standards alignment, and interoperability. Top-performing teams adopted strategies such as intermediate-schema decoupling, dynamic few-shot prompting, iterative refinement, structured prompt engineering, and agent-based synthesis. Results demonstrate that LLMs can generate clinically meaningful, standards-compliant code, while also highlighting the need for explicit semantic layers to ensure safety and interpretability. This task provides the first large-scale evaluation of medical NLP code generation grounded in FHIR Profiles and offers a foundation for future development of verifiable, trustworthy clinical AI systems. Additional details, datasets, and evaluation materials are available at the CHIP 2025 website: http://cips-chip.org.cn/2025/eval3 .