Deploying computer animation and social agent models on devices with limited computing resources presents a significant challenge. Knowledge distillation (KD) emerges as an effective model compression technique, harnessing the extensive knowledge of a large teacher model to facilitate the training of a smaller student model. However, existing KD methodologies predominantly concentrate on transferring task-specific knowledge from supervised tasks, such as logit and feature, overlooking the valuable insights into cross-sample discrepancy inherent in teacher and student models. In response, we propose a novel KD approach, termed augmented knowledge distillation via contrastive learning (CAKD). Initially, in the supervision task, we enhance vanilla KD by integrating logit and feature outputs derived from both the original and the augmented data. Subsequently, in the self-supervision task, we identify pivotal sample pairs and delineate the inter-sample multi-discrepancy relationships using the intrinsic data structure, thus obviating the need for external labels or supervision. This enables knowledge transfer through contrastive learning. The fusion of knowledge from both tasks synergistically enhances student performance. Experimental assessments conducted on two publicly available datasets demonstrate that CAKD surpasses state-of-the-art knowledge distillation methodologies.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Augmented Knowledge Distillation via Contrastive Learning

  • Jianhua Xu,
  • Lin Li,
  • Jianping Gou,
  • Lan Du,
  • Yibing Zhan

摘要

Deploying computer animation and social agent models on devices with limited computing resources presents a significant challenge. Knowledge distillation (KD) emerges as an effective model compression technique, harnessing the extensive knowledge of a large teacher model to facilitate the training of a smaller student model. However, existing KD methodologies predominantly concentrate on transferring task-specific knowledge from supervised tasks, such as logit and feature, overlooking the valuable insights into cross-sample discrepancy inherent in teacher and student models. In response, we propose a novel KD approach, termed augmented knowledge distillation via contrastive learning (CAKD). Initially, in the supervision task, we enhance vanilla KD by integrating logit and feature outputs derived from both the original and the augmented data. Subsequently, in the self-supervision task, we identify pivotal sample pairs and delineate the inter-sample multi-discrepancy relationships using the intrinsic data structure, thus obviating the need for external labels or supervision. This enables knowledge transfer through contrastive learning. The fusion of knowledge from both tasks synergistically enhances student performance. Experimental assessments conducted on two publicly available datasets demonstrate that CAKD surpasses state-of-the-art knowledge distillation methodologies.