Knowledge distillation enables compact student models to learn from larger teachers, achieving competitive accuracy with fewer parameters. Despite its efficiency, challenges remain in effectively transferring knowledge when architectural gaps exist between teacher and student. To mitigate this, Teacher-Assistant Knowledge Distillation (TAKD) introduces an intermediate assistant model to bridge the gap. While later researches have improved assistant architectures and training objectives, existing methods still suffer from suboptimal classification guidance, as the student learns from an assistant classifier that approximates the original teacher’s classification capability and lacks of a progressive learning scheme that adapts to the student’s evolving training needs. In this paper, we propose Collaborative and Progressive Teacher-Assistant Knowledge Distillation (CPTAKD), which guides student learning by jointly aligning feature representations from both assistant and original teacher. Meanwhile, a fused classifier is directly reused which is constructed by integrating the pretrained classifiers of two teachers. Furthermore, we design a guidance schedule that gradually shifts supervision from the assistant to the original teacher, enabling a smooth transition from simple to complex knowledge. Extensive experiments prove the effectiveness of our proposed method, showing superior performance over related distillation works.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Collaborative and Progressive Teacher-Assistant Knowledge Distillation

  • Rui Gao,
  • Ping Wang,
  • Jianping Gou,
  • Lan Du,
  • Weihua Ou

摘要

Knowledge distillation enables compact student models to learn from larger teachers, achieving competitive accuracy with fewer parameters. Despite its efficiency, challenges remain in effectively transferring knowledge when architectural gaps exist between teacher and student. To mitigate this, Teacher-Assistant Knowledge Distillation (TAKD) introduces an intermediate assistant model to bridge the gap. While later researches have improved assistant architectures and training objectives, existing methods still suffer from suboptimal classification guidance, as the student learns from an assistant classifier that approximates the original teacher’s classification capability and lacks of a progressive learning scheme that adapts to the student’s evolving training needs. In this paper, we propose Collaborative and Progressive Teacher-Assistant Knowledge Distillation (CPTAKD), which guides student learning by jointly aligning feature representations from both assistant and original teacher. Meanwhile, a fused classifier is directly reused which is constructed by integrating the pretrained classifiers of two teachers. Furthermore, we design a guidance schedule that gradually shifts supervision from the assistant to the original teacher, enabling a smooth transition from simple to complex knowledge. Extensive experiments prove the effectiveness of our proposed method, showing superior performance over related distillation works.