Scholarlink

<p>The increasing size of language models raises great research interests in parameter-efficient fine-tuning (PEFT) such as LoRA that freezes the main body of a pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note, two public medical dialogue datasets, and is further validated on News summarization dataset XSum and end-to-end text generation dataset E2E. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation quality over ROUGE scores on sequence-to-sequence (Seq2Seq) tasks. The study demonstrated the potential of an integrated PEFT framework, which proves effective in both medical text generation and diverse scenarios.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Parameter-efficient fine-tuning with layer pruning on medical sequence-to-sequence modeling

  • 期刊论文

  • 发表日期：

    2026年4月10日

查看全文
  • Yunqi Zhu,
  • Yuanyuan Wu,
  • Wensheng Zhang,
  • Xuebing Yang

摘要

The increasing size of language models raises great research interests in parameter-efficient fine-tuning (PEFT) such as LoRA that freezes the main body of a pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note, two public medical dialogue datasets, and is further validated on News summarization dataset XSum and end-to-end text generation dataset E2E. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation quality over ROUGE scores on sequence-to-sequence (Seq2Seq) tasks. The study demonstrated the potential of an integrated PEFT framework, which proves effective in both medical text generation and diverse scenarios.