Scholarlink

Abstract <p>The problem of optimizing neural networks for large language models (LLMs) such as ChatGPT is discussed. One of the directions being developed for optimizing LLMs is knowledge distillation—the transfer of knowledge from a large teacher model to a smaller student model without significant loss of accuracy of the result. The existing methods of knowledge distillation have certain disadvantages: inaccurate knowledge transfer, long learning process, and error accumulation in long sequences. A combination of methods that contribute to improving the quality of knowledge distillation is considered: selective teacher intervention in the student’s learning process and low-rank adaptation. The proposed combination of knowledge distillation methods can be applied to problems with limited computational resources.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Combination of Methods for Selective Teacher Intervention in the Student’s Learning Process and Low-Rank Adaptation in the Knowledge Distillation Models

  • 期刊论文

  • 发表日期：

    2025年12月1日

查看全文
  • I. A. Sikarev,
  • T. M. Tatarnikova,
  • V. M. Abramov

摘要

Abstract

The problem of optimizing neural networks for large language models (LLMs) such as ChatGPT is discussed. One of the directions being developed for optimizing LLMs is knowledge distillation—the transfer of knowledge from a large teacher model to a smaller student model without significant loss of accuracy of the result. The existing methods of knowledge distillation have certain disadvantages: inaccurate knowledge transfer, long learning process, and error accumulation in long sequences. A combination of methods that contribute to improving the quality of knowledge distillation is considered: selective teacher intervention in the student’s learning process and low-rank adaptation. The proposed combination of knowledge distillation methods can be applied to problems with limited computational resources.