Old photos are precious for preserving the happy moments of the past. Typically, most old photos contain various defects such as scratches and dirty dots, presenting a challenging task known as image restoration. However, this task is extremely difficult, and existing methods cannot achieve satisfactory performance. The reason is that localized defects in old photos are typically irregularly shaped and randomly distributed, making these defects difficult to detect completely and also hard to repair. In this paper, we propose a two-stage image restoration model. Firstly, Super-pixel Segmentation is employed to divide an image into super-pixel blocks. Rather than pinpointing actual defects within an image, the first stage of our model is designed to identify the super-pixel blocks that are suspected to harbor defects. The other stage focuses on inpainting these detected blocks using a Fast Fourier Convolution model. The proposed method has been implemented and its efficacy has been evaluated on the DIV2K and CelebA datasets. We conducted a comparative analysis with five previous image restoration methods. Experimental outcomes demonstrate that our method exhibits superior performance.