Scholarlink

<p>Traditional knowledge distillation mostly adopts a two-stage training strategy, and its performance heavily relies on pre-trained teacher models. Online knowledge distillation alleviates the limitations of traditional knowledge distillation to some extent through an end-to-end one-stage training process. However, existing methods generally focus on the extraction of deep semantic features, often neglecting the potential value of shallow features in knowledge transfer. To address this, we propose a cross-branch knowledge distillation framework via shallow layer guidance (SLG). This framework innovatively integrates the shallow feature-guided cross-branch collaboration mechanism with a deep feature integration strategy. It activates fine-grained information interaction between branches through shallow features while strengthening the transfer of semantic knowledge by means of deep features. Experimental results demonstrate that the proposed SLG framework consistently outperforms the baseline models across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, achieving average Top-1 accuracy improvements of 1.01%, 3.10%, and 2.66%, respectively.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Cross-branch knowledge distillation via shallow layer guidance

  • 期刊论文

  • 发表日期：

    2026年3月11日

查看全文
  • Mingfu Zhu,
  • Qiaoqiao Wang,
  • Chuanqi Ma

摘要

Traditional knowledge distillation mostly adopts a two-stage training strategy, and its performance heavily relies on pre-trained teacher models. Online knowledge distillation alleviates the limitations of traditional knowledge distillation to some extent through an end-to-end one-stage training process. However, existing methods generally focus on the extraction of deep semantic features, often neglecting the potential value of shallow features in knowledge transfer. To address this, we propose a cross-branch knowledge distillation framework via shallow layer guidance (SLG). This framework innovatively integrates the shallow feature-guided cross-branch collaboration mechanism with a deep feature integration strategy. It activates fine-grained information interaction between branches through shallow features while strengthening the transfer of semantic knowledge by means of deep features. Experimental results demonstrate that the proposed SLG framework consistently outperforms the baseline models across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, achieving average Top-1 accuracy improvements of 1.01%, 3.10%, and 2.66%, respectively.