Knowledge distillation aims to transfer knowledge from a complex teacher model to a simpler student model by optimizing both task and distillation objectives, which focus on learning from the ground truth and the teacher model, respectively. However, most existing works overlook the inherent difficulty disparity between these two objectives and rigidly treat them equally important during the training process. Through theoretical deduction and experimental observation, we have uncovered a significant difference in the difficulty levels between the task and distillation objectives. Inspired by the curriculum learning paradigm, we hypothesize that better convergence of the student model can be achieved when the objectives are optimized progressively from simpler to more challenging ones during training. To address this issue, we propose a simple yet effective method called Objective-oriented Adaptive Knowledge Distillation (OAKD), leveraging curriculum-guided dynamic weighting. This approach dynamically adjusts the weights of the task and distillation objectives to control the difficulty of knowledge during the student’s learning process. Designed as a flexible and easily integrable technology, OAKD can seamlessly integrate into existing knowledge distillation methods, offering universal improvements with minimal computational overhead. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate the effectiveness of our approach.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

OAKD: Objective-Oriented Adaptive Knowledge Distillation via Curriculum-Guided Dynamic Weighting

  • Chuan Li,
  • Lijun Tan,
  • Xiao Teng,
  • Guijian Tang,
  • Yan Ding,
  • Long Lan

摘要

Knowledge distillation aims to transfer knowledge from a complex teacher model to a simpler student model by optimizing both task and distillation objectives, which focus on learning from the ground truth and the teacher model, respectively. However, most existing works overlook the inherent difficulty disparity between these two objectives and rigidly treat them equally important during the training process. Through theoretical deduction and experimental observation, we have uncovered a significant difference in the difficulty levels between the task and distillation objectives. Inspired by the curriculum learning paradigm, we hypothesize that better convergence of the student model can be achieved when the objectives are optimized progressively from simpler to more challenging ones during training. To address this issue, we propose a simple yet effective method called Objective-oriented Adaptive Knowledge Distillation (OAKD), leveraging curriculum-guided dynamic weighting. This approach dynamically adjusts the weights of the task and distillation objectives to control the difficulty of knowledge during the student’s learning process. Designed as a flexible and easily integrable technology, OAKD can seamlessly integrate into existing knowledge distillation methods, offering universal improvements with minimal computational overhead. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate the effectiveness of our approach.