Online knowledge distillation has emerged as a powerful approach for training student networks in real-time, exhibiting promising results in image classification tasks. However, current online knowledge distillation methods primarily focus on enhancing performance by transferring prediction and feature knowledge, neglecting the potential benefits of leveraging feature correlation. To address this gap and effectively distill valuable relational knowledge, we introduce a novel online knowledge distillation method named Inter-class Correlation-based Online Knowledge Distillation (ICOKD). Our approach establishes feature correlations among inter-class samples within each network and facilitates the mutual transfer of relational knowledge across different online networks. Additionally, to extract more informative feature relation information, we incorporate a feature enhancement module to enrich features. Moreover, we design an adaptive distillation module to guide each student network to learn at the logits level, thereby further distilling effective logits-based knowledge. Experimental results on CIFAR-100 and Tiny-ImageNet datasets validate the effectiveness of our proposed ICOKD method, demonstrating its superiority over state-of-the-art online methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Inter-Class Correlation-Based Online Knowledge Distillation

  • Hongfang Zhu,
  • Jianping Gou,
  • Lan Du,
  • Weihua Ou

摘要

Online knowledge distillation has emerged as a powerful approach for training student networks in real-time, exhibiting promising results in image classification tasks. However, current online knowledge distillation methods primarily focus on enhancing performance by transferring prediction and feature knowledge, neglecting the potential benefits of leveraging feature correlation. To address this gap and effectively distill valuable relational knowledge, we introduce a novel online knowledge distillation method named Inter-class Correlation-based Online Knowledge Distillation (ICOKD). Our approach establishes feature correlations among inter-class samples within each network and facilitates the mutual transfer of relational knowledge across different online networks. Additionally, to extract more informative feature relation information, we incorporate a feature enhancement module to enrich features. Moreover, we design an adaptive distillation module to guide each student network to learn at the logits level, thereby further distilling effective logits-based knowledge. Experimental results on CIFAR-100 and Tiny-ImageNet datasets validate the effectiveness of our proposed ICOKD method, demonstrating its superiority over state-of-the-art online methods.