With the advancement of adversarial attack techniques, face forgery detection models have received increasing attention in the research community. Most existing adversarial methods rely on global perturbations; however, the human visual system is susceptible to such manipulations when applied to facial features. To overcome this limitation, we propose the Adversarial Semantic Mask Attack (ASMA) framework, which leverages facial segmentation to confine perturbations to small, less perceptible regions of the face, while preserving both attack effectiveness and transferability. Specifically, ASMA generates a semantic mask over facial input and iteratively applies perturbations guided by the gradient information of the target detection model, ensuring both imperceptibility and efficacy. Furthermore, we extend this framework by integrating a differential evolution algorithm to perturb a limited set of pixels within the selected regions, achieving effective attacks with minimal distortion. Extensive experiments conducted on multiple face forgery detection models and public datasets demonstrate that our method outperforms existing approaches in terms of attack success rate and perturbation magnitude. The source code and models are available at https://github.com/yhwang-xdu/ASMA.