Significant improvements in image inpainting techniques have been achieved due to advancements in deep learning. At present, the mainstream network structure follows the encoder-decoder architecture, and the model sometimes has an attention module to optimize the inpainting effect. However, shadows tend to occur for different missing areas in image restoration, especially at the edges of missing areas and areas where texture is repeated. To address this problem, we propose a new two-stage, end-to-end generative model that extracts textures through a Gabor filter. Among them, the first stage network adopts the U-Net network architecture to predict and inpainting the texture information of the unknown area, and the second stage takes the output from the first stage as prior information to guide the image inpainting network for better filling the missing area. In the network architecture, we also introduce the contextual attention mechanism to optimize the effect. Experiments demonstrate that our method outperforms the current state-of-the-art methods on two widely used datasets, CelebA-HQ and Paris Street View. Our model produces better inpainting results both quantitatively and qualitatively.