MPKD-DCFI: multi-path knowledge distillation via dynamic contextual feature interaction
摘要
The most essential problems faced by multi-point distillation are ensuring the accuracy of feature alignment and the flexibility of dynamically adjusting distillation points. However, these problems must be addressed while handling complex features and model fitting capabilities. Therefore, in this article, to address these problems, we propose a novel method called Multi-Path Knowledge Distillation with Dynamic Contextual Feature Interaction (MPKD-DCFI). Firstly, we utilize channel-attention and spatial-attention mechanisms, which perform multi-layer feature extraction on the raw input data to dynamically adjust the weight assignments. This process generates a weight matrix that incorporates the attention mechanisms in the policy network, enhancing the flexibility of feature extraction while increasing the model’s attention to important features. Secondly, we address the limitation of fixed distillation points in existing multipoint distillation by innovatively introducing the Gram matrix, which optimizes the feature alignment process between teacher and student networks. The Gram matrix captures the correlation between feature maps, thus ensuring the comprehensive extraction of key information from both teacher and student networks. Finally, we propose a flexible multi-point distillation method, which not only extracts the current key information between teachers and students but also synthesizes the contextual feature information of the two network models. This method dynamically adjusts the distillation points to ensure that the models needing distillation are judged in real time during the training process. The results show that the approach is feasible, and has satisfactory accuracy and time efficiency.