Scholarlink

<p>Knowledge distillation aims to transfer insights from a complex teacher network to a streamlined student network for model compression. While self-knowledge distillation further enhances efficiency by distilling through the student network itself, existing methods often introduce additional computational costs. To address this, we propose a novel self-knowledge distillation approach named Self-Knowledge Distillation through Ensemble Model Averaging (EMA-SKD). This method employs exponentially weighted averaging to merge predictions from multiple past training phases, enabling the model to assimilate new knowledge while revisiting historical data. To enhance generalization, we perform interpolation-based intermediate sampling of same-category samples during self-knowledge distillation. A new loss function is devised to filter soft labels, optimizing the balance between teacher-derived information and model-extracted knowledge. Experimental results on CIFAR-100, Tiny-ImageNet, and ImageNet datasets demonstrate that EMA-SKD outperforms advanced state-of-the-art self-knowledge distillation methods across different model architectures, achieving significant improvements without introducing additional training parameters. Our code is available at <a href="https://github.com/bigbigyellow/EMA-SKD">https://github.com/bigbigyellow/EMA-SKD</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Self-knowledge distillation through ensemble model averaging: a novel approach for image classification

  • 期刊论文

  • 发表日期：

    2025年7月18日

查看全文
  • Wei Zhu,
  • Hengyi Huang,
  • Longxi Zhu,
  • ChunYang Shao,
  • Ningzhong Liu,
  • Yu Wang

摘要

Knowledge distillation aims to transfer insights from a complex teacher network to a streamlined student network for model compression. While self-knowledge distillation further enhances efficiency by distilling through the student network itself, existing methods often introduce additional computational costs. To address this, we propose a novel self-knowledge distillation approach named Self-Knowledge Distillation through Ensemble Model Averaging (EMA-SKD). This method employs exponentially weighted averaging to merge predictions from multiple past training phases, enabling the model to assimilate new knowledge while revisiting historical data. To enhance generalization, we perform interpolation-based intermediate sampling of same-category samples during self-knowledge distillation. A new loss function is devised to filter soft labels, optimizing the balance between teacher-derived information and model-extracted knowledge. Experimental results on CIFAR-100, Tiny-ImageNet, and ImageNet datasets demonstrate that EMA-SKD outperforms advanced state-of-the-art self-knowledge distillation methods across different model architectures, achieving significant improvements without introducing additional training parameters. Our code is available at https://github.com/bigbigyellow/EMA-SKD.