With increasing concerns about potential data privacy issues, Machine Unlearning(MU) has attracted growing attention, aiming to remove specific data from a trained model without retraining. Most current methods suffer from performance degradation on either remembering and forgetting categories, named as overly forgetting and incomplete forgetting respectively. To alleviate these issues, we propose Cross-level Distillation based Machine Unlearning(CD-MU) with contrastive enhanced knowledge. To mitigate overly unlearning, a cross-level distillation structure is first proposed to transfer both abstracted and discriminative knowledge from teacher to student. Built on student network, we further design contrastive feature enhancing module to strengthen the ambiguous knowledge about the forgetting categories, thus mitigating incomplete unlearning. Experiments on public dataset show the superior performance of CD-MU. Tests in privacy-sensitive scenario show its robustness against membership inference attack.