Face recognition technology has gradually come into wide application in real life. However, in real-life unconstrained scenarios, low-quality images could be matched to unpredictable individuals, thereby affecting the stability of recognition and even causing security problems. Conducting quality assessment on face images before facial recognition can help alleviate these problems by rejecting low-quality samples. In this paper, we propose a Fast Diffusion-model-based Face Image Quality Assessment method (FDif-FIQA). It uses a diffusion model to perturb face images and measures image quality based on the similarity of features before and after the perturbation. The denoising process is transferred from the pixel space to the latent space, thereby reducing the computational cost. We evaluated the performance of this method with multiple mainstream face recognition models on various public datasets. Compared to state-of-the-art methods, our approach demonstrates highly competitive performance and significant improvements in computational efficiency.