Face recognition system is vulnerable to the threat of adversarial attacks. By adding small perturbations to the original image, attackers generate adversarial samples to mislead the target model, highlighting the importance of research on face recognition against aggression. However, existing face recognition attack methods are not satisfactory due to the low transferability and the lack of imperceptibility. In this paper, we propose a new attack framework: Saliency Semantic Ranking with Quality Restoration Synergistic Adversarial Attack on Face Recognition (SRRA). The framework divides the face into semantic regions, sorts them according to the saliency map, and finds the most critical region. In addition, the interpolation-based quality restoration module is designed to dynamically interpolate the original and generated images to improve the imperceptibility of adversarial samples. Extensive experiments on public datasets CelebA-HQ and FFHQ prove that the adversarial samples generated by SRRA have better transferability and imperceptibility compared with the existing methods. This study constructs a new attack architecture for face recognition system, and provides a new technical reference for system security evaluation and design.