Scholarlink

The backpropagation method is the predominant method for pre-training and fine-tuning of Large Language models. At the same time, it is considerably demanding in terms of memory and hardware. Therefore, it makes fine-tuning and pre-training very expensive, harmful for the environment due to the large carbon footprint, and raises the blocks for the development of frontline models by new companies. This paper presents a novel method of a fine-tuning strategy – Hebb-inspired Low Rank Adapters (HiLoRA) based on partial elimination of the backpropagation with a localized learning rule. Theoretically, the new strategy can bring up to 1.5x acceleration and 1.9x memory reduction to any Large Language model fine-tuning. The proposed method demonstrates acceleration of the fine-tuning of LLaMA-2-7B by up to 77% and considerable reduction of memory requirements of DeBERTa-V2-XL by up to 73% while keeping an accuracy drop of 3.04% on average.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Hebb-Inspired Low Rank Adapters for Large Language Models Fine-Tuning

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Alexander Demidovskij,
  • Artyom Tugaryov,
  • Igor Salnikov,
  • Olga Frolova,
  • Aleksei Trutnev,
  • Pengcheng Xie,
  • Irina Novikova,
  • Egor Zharikov,
  • Vasilisa Blyudova,
  • Yuri Ignatiev

摘要

The backpropagation method is the predominant method for pre-training and fine-tuning of Large Language models. At the same time, it is considerably demanding in terms of memory and hardware. Therefore, it makes fine-tuning and pre-training very expensive, harmful for the environment due to the large carbon footprint, and raises the blocks for the development of frontline models by new companies. This paper presents a novel method of a fine-tuning strategy – Hebb-inspired Low Rank Adapters (HiLoRA) based on partial elimination of the backpropagation with a localized learning rule. Theoretically, the new strategy can bring up to 1.5x acceleration and 1.9x memory reduction to any Large Language model fine-tuning. The proposed method demonstrates acceleration of the fine-tuning of LLaMA-2-7B by up to 77% and considerable reduction of memory requirements of DeBERTa-V2-XL by up to 73% while keeping an accuracy drop of 3.04% on average.