Self-distillation with Mutual Assistance Mechanism: Enhancing Model Performance Through Collaborative Learning
摘要
Self-distillation addresses the challenge of selecting an appropriate teacher model in traditional knowledge distillation by enabling the model to serve as both the teacher and the student. Traditional self-distillation methods often use shallow classifiers as the student model at various depths of the network, with the deepest classifier serving as the teacher model to enhance the overall performance of the model. However, these methods primarily focus on how the teacher model aids the student model in learning, while overlooking the mutual assistance between models at different layers. In order to make up the deficiency, we propose a self-distillation method with a mutual assistance mechanism designed to explore and uncover the effectiveness of the mutual assistance relationship within the self-distillation models. We have conducted experiments on the CIFAR-100 and Tiny ImageNet datasets, achieving improvements of 5.06 \(\%\) and 6.03 \(\%\) respectively, compared to the baseline. This demonstrates the effectiveness of the mutual assistance relationship within the self-distillation models in our proposed method.