<p>The application of large language models (LLMs) in biomedical natural language processing (NLP) shows great promise, yet the complexity and specificity of biomedical texts often challenge general-purpose models. Fine-tuning, which adapts LLMs to domain-specific tasks, is essential to address these challenges and optimize performance. In this work, we present a systematic benchmark of full fine-tuning (FFT) and several parameter-efficient fine-tuning (PEFT) techniques-LoRA, QLoRA, and P-tuning-for adapting LLMs to biomedical NLP. We evaluate these methods across 12 benchmark datasets from the Biomedical Language Understanding and Reasoning Benchmark (BLURB) using modern 3–9B-parameter LLMs, including LLaMA-3, FLAN-T5, ChatGLM-4, and UL2, and we compare them against domain-specific models such as PubMedBERT and BioClinicalBERT. All experiments are conducted on a GPU-based high-performance computing (HPC) cluster; running four LLMs <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\times$</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> four adaptation strategies <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\times$</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> 12 tasks with multiple random seeds entails millions of gradient steps and substantial GPU memory and compute, and would be impractical on a single workstation. Our results show that fine-tuning consistently improves over zero-shot and few-shot prompting, with LoRA and QLoRA emerging as the most effective and computationally efficient approaches: on UL2-3B and FLAN-T5-3B, they improve the macro-averaged BLURB score over full fine-tuning by +9.3/+9.0 and +2.6/+2.0 points, respectively, while using only 30–32 M trainable parameters instead of 3B and reducing peak GPU memory by up to  60%. On BC5-chem NER, for example, UL2-3B with LoRA attains 95.3 F1 versus 94.3 for full fine-tuning (+1.0), and QLoRA reaches 94.9 F1, with similar gains on BC5-disease. Fine-tuned LLMs outperform domain-specific BERT-based models on the majority of BLURB tasks, underscoring the potential of general-purpose LLMs when appropriately adapted; in contrast, in-context learning yields lower and less stable performance in this setting. We also analyze computational cost and robustness, providing practical guidance on how to allocate supercomputing resources and choose PEFT strategies for large-scale biomedical LLM adaptation. Limitations include the use of English-only BLURB tasks, the inclusion of the partially closed-source ChatGLM-4 backbone, and potential ethical risks if such models are applied to clinical text without appropriate de-identification and governance. The code for this study is publicly available on <a href="https://github.com/jjzhu0579/fine-tuning-and-fewshot">https://github.com/jjzhu0579/fine-tuning-and-fewshot</a>.</p>

Adapting LLMs for biomedical natural language processing: a comprehensive benchmark study on fine-tuning methods

期刊论文
发表日期：

2026年1月27日

查看全文

Junjie Zhu,
Jin Li,
Shen Zhao,
Yiyan Deng,
Yongming Miao,
Jun Xu

摘要

The application of large language models (LLMs) in biomedical natural language processing (NLP) shows great promise, yet the complexity and specificity of biomedical texts often challenge general-purpose models. Fine-tuning, which adapts LLMs to domain-specific tasks, is essential to address these challenges and optimize performance. In this work, we present a systematic benchmark of full fine-tuning (FFT) and several parameter-efficient fine-tuning (PEFT) techniques-LoRA, QLoRA, and P-tuning-for adapting LLMs to biomedical NLP. We evaluate these methods across 12 benchmark datasets from the Biomedical Language Understanding and Reasoning Benchmark (BLURB) using modern 3–9B-parameter LLMs, including LLaMA-3, FLAN-T5, ChatGLM-4, and UL2, and we compare them against domain-specific models such as PubMedBERT and BioClinicalBERT. All experiments are conducted on a GPU-based high-performance computing (HPC) cluster; running four LLMs $\times$ $\times$ four adaptation strategies $\times$ $\times$ 12 tasks with multiple random seeds entails millions of gradient steps and substantial GPU memory and compute, and would be impractical on a single workstation. Our results show that fine-tuning consistently improves over zero-shot and few-shot prompting, with LoRA and QLoRA emerging as the most effective and computationally efficient approaches: on UL2-3B and FLAN-T5-3B, they improve the macro-averaged BLURB score over full fine-tuning by +9.3/+9.0 and +2.6/+2.0 points, respectively, while using only 30–32 M trainable parameters instead of 3B and reducing peak GPU memory by up to 60%. On BC5-chem NER, for example, UL2-3B with LoRA attains 95.3 F1 versus 94.3 for full fine-tuning (+1.0), and QLoRA reaches 94.9 F1, with similar gains on BC5-disease. Fine-tuned LLMs outperform domain-specific BERT-based models on the majority of BLURB tasks, underscoring the potential of general-purpose LLMs when appropriately adapted; in contrast, in-context learning yields lower and less stable performance in this setting. We also analyze computational cost and robustness, providing practical guidance on how to allocate supercomputing resources and choose PEFT strategies for large-scale biomedical LLM adaptation. Limitations include the use of English-only BLURB tasks, the inclusion of the partially closed-source ChatGLM-4 backbone, and potential ethical risks if such models are applied to clinical text without appropriate de-identification and governance. The code for this study is publicly available on https://github.com/jjzhu0579/fine-tuning-and-fewshot.