<p>Knowledge distillation involves transferring the “dark knowledge” acquired by a teacher model to a student model to enhance the student’s generalization ability, and it represents a crucial technique for model compression. The teacher’s feature representations effectively serve as knowledge sources that help the student leverage robust representation capabilities. However, current research on representation distillation predominantly focuses on the design of representation tasks while overlooking the significant impact of representation diversity. In this work, we introduce representation diversity into the distillation process for the first time and argue that it is equally important as representation tasks. We propose a novel representation distillation method, QKD, designed to improve student performance by creating diverse teacher representations and introducing new representation learning tasks. Specifically, we first construct diverse teacher representations by perturbing features in the teacher’s penultimate layer within a Q-neighborhood to increase representation diversity. Then, we develop an efficient batch-based contrastive learning approach to facilitate the student model in learning from these diverse teacher representations. Finally, we integrate the teacher’s Softmax Regression (SR) classifier with the diversified teacher representations to further guide the student in acquiring distinct semantic representations. Extensive experiments demonstrate that our method outperforms other state-of-the-art distillation techniques across various knowledge distillation scenarios, including classification, multi-teacher distillation, and binary network distillation.</p>