Scholarlink

<p>Knowledge distillation (KD) has become a pivotal technique in deep learning, facilitating model compression and regularization by transferring knowledge from one neural network to another, enhancing its capabilities for downstream tasks such as classification. However, real-world datasets often suffer from noisy label problems, significantly hindering neural network learning in supervised tasks. Recent advancements in KD aim to improve noise-robustness and regularization in deep neural networks through different learning paradigms. Yet, prevalent approaches often exhibit noise-prone behaviors as the student network heavily relies on the teacher’s learning. To address this challenge, we propose a robust knowledge transfer method, <b>NoRD</b>: a <b>N</b>oise-<b>R</b>esilient Self-<b>D</b>istillation framework. This approach leverages relative self-supervision combined with decision matching to minimize noise susceptibility during the knowledge transfer process. Our study evaluates this technique on CIFAR-10, CIFAR-100, and MNIST datasets with synthetic label noise. Results showcase that our method achieves 8-10% higher test accuracy compared to state-of-the-art noise-robust loss functions at noise rates exceeding 50%, surpassing well-known KD methods by 4-5% in top-1 test accuracy. The code is available at <a href="https://github.com/philsaurabh/NoRD_Applied-Intelligence">https://github.com/philsaurabh/NoRD_Applied-Intelligence</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

NoRD: A framework for noise-resilient self-distillation through relative supervision

  • 期刊论文

  • 发表日期：

    2025年2月15日

查看全文
  • Saurabh Sharma,
  • Shikhar Singh Lodhi,
  • Vanshika Srivastava,
  • Joydeep Chandra

摘要

Knowledge distillation (KD) has become a pivotal technique in deep learning, facilitating model compression and regularization by transferring knowledge from one neural network to another, enhancing its capabilities for downstream tasks such as classification. However, real-world datasets often suffer from noisy label problems, significantly hindering neural network learning in supervised tasks. Recent advancements in KD aim to improve noise-robustness and regularization in deep neural networks through different learning paradigms. Yet, prevalent approaches often exhibit noise-prone behaviors as the student network heavily relies on the teacher’s learning. To address this challenge, we propose a robust knowledge transfer method, NoRD: a Noise-Resilient Self-Distillation framework. This approach leverages relative self-supervision combined with decision matching to minimize noise susceptibility during the knowledge transfer process. Our study evaluates this technique on CIFAR-10, CIFAR-100, and MNIST datasets with synthetic label noise. Results showcase that our method achieves 8-10% higher test accuracy compared to state-of-the-art noise-robust loss functions at noise rates exceeding 50%, surpassing well-known KD methods by 4-5% in top-1 test accuracy. The code is available at https://github.com/philsaurabh/NoRD_Applied-Intelligence.