Self-distillation addresses the challenge of selecting an appropriate teacher model in traditional knowledge distillation by enabling the model to serve as both the teacher and the student. Traditional self-distillation methods often use shallow classifiers as the student model at various depths of the network, with the deepest classifier serving as the teacher model to enhance the overall performance of the model. However, these methods primarily focus on how the teacher model aids the student model in learning, while overlooking the mutual assistance between models at different layers. In order to make up the deficiency, we propose a self-distillation method with a mutual assistance mechanism designed to explore and uncover the effectiveness of the mutual assistance relationship within the self-distillation models. We have conducted experiments on the CIFAR-100 and Tiny ImageNet datasets, achieving improvements of 5.06 \(\%\) and 6.03 \(\%\) respectively, compared to the baseline. This demonstrates the effectiveness of the mutual assistance relationship within the self-distillation models in our proposed method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Self-distillation with Mutual Assistance Mechanism: Enhancing Model Performance Through Collaborative Learning

  • Lin Liu,
  • Zhe Li,
  • Yuqing Tian

摘要

Self-distillation addresses the challenge of selecting an appropriate teacher model in traditional knowledge distillation by enabling the model to serve as both the teacher and the student. Traditional self-distillation methods often use shallow classifiers as the student model at various depths of the network, with the deepest classifier serving as the teacher model to enhance the overall performance of the model. However, these methods primarily focus on how the teacher model aids the student model in learning, while overlooking the mutual assistance between models at different layers. In order to make up the deficiency, we propose a self-distillation method with a mutual assistance mechanism designed to explore and uncover the effectiveness of the mutual assistance relationship within the self-distillation models. We have conducted experiments on the CIFAR-100 and Tiny ImageNet datasets, achieving improvements of 5.06 \(\%\) and 6.03 \(\%\) respectively, compared to the baseline. This demonstrates the effectiveness of the mutual assistance relationship within the self-distillation models in our proposed method.