Deep learning models, particularly Convolutional Neural Networks (CNNs), have achieved remarkable success in various computer vision tasks. However, high computational and memory requirements often constrain their deployment on resource-limited devices. To address these limitations, knowledge distillation (KD) was proposed to transfer knowledge from a large teacher model to a smaller student model using softened probability distributions that reveal inter-class relationships. Hint-based KD (HKD) improves this by incorporating intermediate feature representations, allowing the student to mimic the teacher’s internal features. While enhancing knowledge transfer, HKD requires a regressor layer to align feature dimensions, adding a computational overhead. In this work, a mutual information-based feature distillation (MIFD) method is proposed that selects only the most informative teacher channels based on their mutual information with the target class. These selected channels are then used to guide an intermediate layer of the student model within the knowledge distillation framework. This targeted selection ensures that the student model receives the most relevant feature representations, improving learning efficiency while maintaining a lightweight architecture. A joint-conditional mutual information for selecting informative feature (JCIF) based feature distillation method is also employed to balance feature relevance and redundancy, enabling the selection of a highly informative and non-redundant channel subset. The proposed methods were evaluated across different architectures, including a simple CNN and WideResNet for the CIFAR-10 and CIFAR-100 image classification tasks, respectively, and a U-Net model for hyperspectral image segmentation using the Indian Pines dataset. Experimental results show that MI-based distillation improved student performance and decreased parameter count.