Federated Learning (FL) is susceptible to backdoor attacks due to its distributed architecture and the non-independent and identically distributed (Non-IID) characteristics of client data. Existing defense strategies face two significant limitations: first, in Non-IID scenarios, benign clients are at risk of being misclassified as malicious nodes, which disrupts model convergence; second, while directly eliminating suspicious models can mitigate backdoor attacks, it may inevitably lead to the loss of potentially beneficial knowledge. To address these challenges, this paper proposes a hierarchical knowledge distillation defense framework that balances backdoor defense and knowledge retention within models. Specifically, we first design a clustering method based on Singular Value Decomposition (SVD) of high-level parameters. By analyzing the characteristics of high-level model parameters, we significantly enhance the accuracy of malicious node detection. Additionally, we construct a three-layer knowledge distillation mechanism (output layer, feature layer, and parameter layer) that progressively decouples the association between backdoors and model representations, enabling the removal of backdoor trigger patterns while extracting and integrating general classification knowledge. Theoretical analysis and extensive experiments demonstrate that, under Non-IID data distributions, ours defense method exhibits significant advantages compared to mainstream methodologies, while achieving an average increase of over 7.32% in classification accuracy on primary tasks. This research provides a novel approach for balancing security and efficiency in federated learning.