OAKD: Objective-Oriented Adaptive Knowledge Distillation via Curriculum-Guided Dynamic Weighting
摘要
Knowledge distillation aims to transfer knowledge from a complex teacher model to a simpler student model by optimizing both task and distillation objectives, which focus on learning from the ground truth and the teacher model, respectively. However, most existing works overlook the inherent difficulty disparity between these two objectives and rigidly treat them equally important during the training process. Through theoretical deduction and experimental observation, we have uncovered a significant difference in the difficulty levels between the task and distillation objectives. Inspired by the curriculum learning paradigm, we hypothesize that better convergence of the student model can be achieved when the objectives are optimized progressively from simpler to more challenging ones during training. To address this issue, we propose a simple yet effective method called Objective-oriented Adaptive Knowledge Distillation (OAKD), leveraging curriculum-guided dynamic weighting. This approach dynamically adjusts the weights of the task and distillation objectives to control the difficulty of knowledge during the student’s learning process. Designed as a flexible and easily integrable technology, OAKD can seamlessly integrate into existing knowledge distillation methods, offering universal improvements with minimal computational overhead. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate the effectiveness of our approach.