In recent years, the privacy security of deep learning models has received widespread attention, and many articles have pointed out that deep learning models are vulnerable to various privacy attacks. As a typical application of data privacy protection technology differential privacy in deep learning models, the Differentially Private Stochastic Gradient Descent (DPSGD) algorithm has gradually become one of the main technologies for privacy security protection of deep learning models and has been widely used in academic and industrial research. However, DPSGD also has some unavoidable shortcomings, that is, when DPSGD provides privacy protection for deep learning models, it will damage the classification accuracy of the model. In this paper, we are inspired by model compression technology and propose a method of using pre-trained models as feature extractor, combined with knowledge distillation technology. Our method can reduce the input dimension of the model, achieve the effect of reducing the model gradient norm. In this way, we can achieve a significant improvement in model classification accuracy under the same privacy budget, especially in complex data sets (such as the ImageNet data set). Extensive experiments demonstrate that our method achieves State-of-the-Art on multiple datasets.