Currently, highly realistic face-swapped videos/images pose significant potential security risks to society, which forces researchers to develop an effective defense mechanism against deepfakes, especially in a proactive manner. This involves adding imperceptible adversarial perturbations to facial images, which render deepfake models ineffective. Although existing proactive defense methods based on adversarial perturbations show promising performance in white-box scenarios, most of them suffer from significant performance degradation in black-box scenarios. Moreover, we observe that adversarial perturbations appearing in low-frequency areas (e.g., cheeks) are more easily noticed by the human eye. To this end, we propose a novel Frequency-Aware Multi-Teacher Distillation (FMTD) framework, which aims to generate a universal and imperceptible perturbation with remarkable generalization property against various facial manipulation methods. To extract universal property, we propose a multi-teacher distillation block for learning valuable shared knowledge from different perturbation generators. Moreover, we design a Frequency-aware Mean Squared Error (FMSE) loss based on wavelet transform to prefer to generate imperceptible perturbations in high-frequency regions. Experimental results demonstrate the superior performance of our proposed proactive defense method to state-of-the-art methods.