Scholarlink

With the development of large language models, parameter-efficient fine-tuning (PEFT) methods have gained increasing attention. However, fine-tuning methods often struggle to capture diverse input features and fully enhance model performance, especially in complex contexts and multi-task scenarios. To address this issue, we propose a new fine-tuning framework called Expertise-Centric Parameter-Efficient Fine-Tuning (EC-PEFT). We introduce Mixture of Experts, treating existing parameter-efficient tuning modules as experts. By leveraging expert centers for contrastive learning of different features, we amplify the inherent advantages of expert models. An expert balancing mechanism is also employed to optimize the feature learning differences among experts and the balance of expert selection. Additionally, we propose two specific implementations of the new framework combined with classic parameter-efficient tuning modules: EC-LoRA and EC-Adapter. We conducted experiments on eight benchmark datasets, and the results demonstrate that EC-PEFT performs exceptionally well in both single-task and multi-task learning, significantly outperforming existing fine tuning methods.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

EC-PEFT: An Expertise-Centric Parameter-Efficient Fine-Tuning Framework for Large Language Models

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Yimeng Zhang,
  • Xuelin Cheng,
  • Yan Jiang,
  • Yijun Bei

摘要

With the development of large language models, parameter-efficient fine-tuning (PEFT) methods have gained increasing attention. However, fine-tuning methods often struggle to capture diverse input features and fully enhance model performance, especially in complex contexts and multi-task scenarios. To address this issue, we propose a new fine-tuning framework called Expertise-Centric Parameter-Efficient Fine-Tuning (EC-PEFT). We introduce Mixture of Experts, treating existing parameter-efficient tuning modules as experts. By leveraging expert centers for contrastive learning of different features, we amplify the inherent advantages of expert models. An expert balancing mechanism is also employed to optimize the feature learning differences among experts and the balance of expert selection. Additionally, we propose two specific implementations of the new framework combined with classic parameter-efficient tuning modules: EC-LoRA and EC-Adapter. We conducted experiments on eight benchmark datasets, and the results demonstrate that EC-PEFT performs exceptionally well in both single-task and multi-task learning, significantly outperforming existing fine tuning methods.