Deep neural networks are successfully used in various applications but show their vulnerability to adversarial examples. With the development of adversarial patches, the feasibility of attacks in physical scenes increases, and the defenses against patch attacks are urgently needed. However, the technology for defending against such adversarial patch attacks still requires to be improved. In this paper, we analyze the characteristics of adversarial patches and find that adversarial patches will lead to the appearance or contextual inconsistency in the target objects. The patch region will show abnormal changes on the high-level feature maps of the objects extracted by a backbone network. Consequently, we propose a novel defense method based on a “localizing and inpainting” mechanism to pre-process the input examples. Specifically, we design a unified framework, where the “localizing” sub-network utilizes a two-branch structure corresponding to two characteristics of patches to accurately detect the adversarial patch region in the image. The “inpainting” subnetwork utilizes the surrounding contextual cues to recover the original content covered by the adversarial patch. The quality of inpainted images is also evaluated by measuring the appearance consistency and the effects of adversarial attacks. These two sub-networks are jointly trained via an iterative optimization approach, allowing the ‘localizing’ and ‘inpainting’ modules to closely interact and learn a better solution. Extensive experiments on traffic sign classification and detection tasks demonstrate that our method outperforms the state-of-the-art method, increasing accuracy by 37%, which verifies the effectiveness and superiority of the proposed method.