Scholarlink

<p>In practical use, language models (LM) must efficiently adapt to new tasks and knowledge while avoiding catastrophic forgetting, a requirement that sparked research on continual learning (CL). Despite advances, current CL methods still lack a unified solution that delivers strong knowledge transfer and parameter-efficient capacity management while preventing catastrophic forgetting, which limits effective use of task synergies under tight training and memory budgets. We bridge this gap by introducing Gated Expandable Parameter-Efficient Fine-Tuning (GE-PEFT), a novel approach that shares knowledge of previous tasks through leveraging a single, dynamically expanding PEFT module within LMs while selectively gating irrelevant previous tasks. Our experiments across multiple task-incremental CL benchmarks show that GE-PEFT outperforms existing state-of-the-art CL approaches in both full CL and few-shot settings. Our ablation and parameter sensitivity studies highlight the benefit of each proposed component, demonstrating that GE-PEFT offers a more efficient and adaptive solution for CL in LMs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GE-PEFT: Gated Expandable Parameter-Efficient Fine-Tuning for Continual Learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月6日

查看全文
  • Janna Omeliyanenko,
  • Andreas Hotho,
  • Daniel Schlör

摘要

In practical use, language models (LM) must efficiently adapt to new tasks and knowledge while avoiding catastrophic forgetting, a requirement that sparked research on continual learning (CL). Despite advances, current CL methods still lack a unified solution that delivers strong knowledge transfer and parameter-efficient capacity management while preventing catastrophic forgetting, which limits effective use of task synergies under tight training and memory budgets. We bridge this gap by introducing Gated Expandable Parameter-Efficient Fine-Tuning (GE-PEFT), a novel approach that shares knowledge of previous tasks through leveraging a single, dynamically expanding PEFT module within LMs while selectively gating irrelevant previous tasks. Our experiments across multiple task-incremental CL benchmarks show that GE-PEFT outperforms existing state-of-the-art CL approaches in both full CL and few-shot settings. Our ablation and parameter sensitivity studies highlight the benefit of each proposed component, demonstrating that GE-PEFT offers a more efficient and adaptive solution for CL in LMs.