In recent years, Pre-trained Models (PTMs) have garnered widespread attention in Computer Vision (CV), owing to their exceptional feature extraction capabilities. Enterprises and research institutes with limited data annotation capabilities can leverage PTMs and transfer learning to markedly enhance model performance in downstream tasks, thereby expediting the deployment of real-world applications. However, the utilization of PTMs from unverified or untrusted sources poses significant security risks. A major threat comes from the so-called backdoor attack, in which attackers inject backdoor triggers during the pre-training phase, and consequently compromise the predictive behavior of downstream tasks. Although existing defense strategies can mitigate the backdoor attacks to some extent, they often struggle to eradicate their adverse effects completely, and frequently incur the drawback of degraded model performance. This paper proposes a novel defense scheme against backdoor attacks by integrating Group \(L_{1/2}\) regularization with knowledge distillation. Our scheme effectively reduces the weights of neurons sensitive to backdoor triggers. It removes redundant neurons, simultaneously enhances the classification accuracy while defending against various backdoor attacks, and thus achieving a dual-purpose outcome. We deploy the proposed defense scheme in seven distinct backdoor attack scenarios to evaluate its performance, and compare it with four other popular defense methods. The results demonstrate that our method exhibits superior defensive performance under multiple evaluation metrics, while concurrently improving the model performance.