Scholarlink

<p>Existing self-knowledge distillation (Self-KD) solutions usually focus on transferring historical predictions of individual instances to the current network. However, this approach tends to create overconfidence for easy instances and underconfidence for hard instances. The widely used temperature-based strategies to smooth or sharpen the predicted distributions can lead to inconsistencies across instances, causing sensitivity issues. To address this, our approach views a queue of instances as an ensemble rather than treating each instance independently. We propose a novel method that distills historical knowledge from a dimensional perspective, utilizing intra class characteristics and interclass relationships within each ensemble. First, we align each dimension distribution from the current network to the historical output. Second, we ensure each dimension is closer to similar dimensions than dissimilar ones, maintaining consistent attitudes from present and historical perspectives. Our insights reveal that distilling historical knowledge from a dimensional perspective is more effective than the traditional instance-based approach, with potential applications in related tasks. Empirical results on three famous datasets and various network architectures demonstrate the superiority of our proposed method. Our code is available at <a href="https://github.com/WenkeHuang/DimSelfKD">https://github.com/WenkeHuang/DimSelfKD</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Self-knowledge distillation with dimensional history knowledge

  • 期刊论文

  • 发表日期：

    2025年3月28日

查看全文
  • Wenke Huang,
  • Mang Ye,
  • Zekun Shi,
  • He Li,
  • Bo Du

摘要

Existing self-knowledge distillation (Self-KD) solutions usually focus on transferring historical predictions of individual instances to the current network. However, this approach tends to create overconfidence for easy instances and underconfidence for hard instances. The widely used temperature-based strategies to smooth or sharpen the predicted distributions can lead to inconsistencies across instances, causing sensitivity issues. To address this, our approach views a queue of instances as an ensemble rather than treating each instance independently. We propose a novel method that distills historical knowledge from a dimensional perspective, utilizing intra class characteristics and interclass relationships within each ensemble. First, we align each dimension distribution from the current network to the historical output. Second, we ensure each dimension is closer to similar dimensions than dissimilar ones, maintaining consistent attitudes from present and historical perspectives. Our insights reveal that distilling historical knowledge from a dimensional perspective is more effective than the traditional instance-based approach, with potential applications in related tasks. Empirical results on three famous datasets and various network architectures demonstrate the superiority of our proposed method. Our code is available at https://github.com/WenkeHuang/DimSelfKD.