<p>Knowledge distillation (KD) is an efficient model compression technique for transferring knowledge from large teacher models to lightweight student models. However, traditional methods rely on single-view soft label transfer, with two key limitations: First, the semantic gap between teacher and student impairs the student’s inheritance of the teacher’s essential feature encoding, thus limiting generalization; Second, single views fail to cover real-world data perturbations, making the student sensitive to input changes and less robust. To address these issues, we propose Dual-view Collaborative Distillation (DvCD), which establishes two complementary views: a weakly augmented view serves as a stable anchor for knowledge transfer, while a strongly augmented view introduces severe perturbations to force the model to focus on anti-interference core features. Inspired by curriculum learning, we design a dynamic knowledge fusion strategy that fuses the student’s predictions with the teacher’s knowledge to generate a smooth transitional supervision signal, where the weight of the teacher’s knowledge is adaptively increased to guide the student through a progressive learning curriculum. A unified consistency loss further ensures cross-view predictive consistency. Experiments on CIFAR-100 and ImageNet show that DvCD consistently outperforms mainstream distillation methods across network architectures, verifying its effectiveness and generalization. The code is open-sourced at: <a href="https://github.com/wsh081/DvCD.git">https://github.com/wsh081/DvCD.git</a></p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-view collaborative distillation

  • Shihui Wang,
  • Xinwei Li,
  • Bingfeng Li,
  • Yi Yang

摘要

Knowledge distillation (KD) is an efficient model compression technique for transferring knowledge from large teacher models to lightweight student models. However, traditional methods rely on single-view soft label transfer, with two key limitations: First, the semantic gap between teacher and student impairs the student’s inheritance of the teacher’s essential feature encoding, thus limiting generalization; Second, single views fail to cover real-world data perturbations, making the student sensitive to input changes and less robust. To address these issues, we propose Dual-view Collaborative Distillation (DvCD), which establishes two complementary views: a weakly augmented view serves as a stable anchor for knowledge transfer, while a strongly augmented view introduces severe perturbations to force the model to focus on anti-interference core features. Inspired by curriculum learning, we design a dynamic knowledge fusion strategy that fuses the student’s predictions with the teacher’s knowledge to generate a smooth transitional supervision signal, where the weight of the teacher’s knowledge is adaptively increased to guide the student through a progressive learning curriculum. A unified consistency loss further ensures cross-view predictive consistency. Experiments on CIFAR-100 and ImageNet show that DvCD consistently outperforms mainstream distillation methods across network architectures, verifying its effectiveness and generalization. The code is open-sourced at: https://github.com/wsh081/DvCD.git