<p>Recently, pre-trained code models (PCMs) have advanced software development by automating tasks and improving productivity. However, their large size hinders seamless adoption in developers’ daily workflows, as local execution is often infeasible and reliance on cloud-based services raises concerns about data privacy. These challenges highlight the need for effective compression techniques to enable secure and efficient on-device deployment. To address this problem, this paper investigates knowledge distillation (KD) as a means to compress large PCMs. We systematically evaluate the effectiveness of KD on PCMs by comparing different distillation paradigms across various code generation and understanding tasks. Our results show that feature-based distillation generally outperforms response-based approaches, though the effectiveness varies depending on the specific PCM and downstream task. Furthermore, we identify key factors that influence the effectiveness of the feature-based knowledge distillation, including loss functions, mapping configurations, and the number of intermediate layer distillation iterations. Building on these insights, we propose <span>BOKD</span>, a Bayesian optimization–based method that adaptively selects optimal distillation configurations according to the student model’s capacity and task complexity. Empirical results demonstrate the practicality of <span>BOKD</span>. A compact 2-layer model (just 5% of teacher parameters) retains 96–97% (relative performance score) of teacher performance on classification tasks, while achieving a 30% perplexity reduction over baseline distillation methods. Our contributions include a comprehensive empirical studies on distillation paradigms, and a novel parameter optimization technique to enhance KD performance.</p>