As a typical low-level vision task, single image defocus deblurring aims to produce a sharp clear image from a defocused blurry image. However, image degradation inevitably results in a significant loss of information and non-uniqueness of the solution, making it challenging to learn the mapping from low-quality inputs to high-quality outputs. To address this issue, we propose a three-stage learning framework for image defocus deblurring, namely 3SDeblur, which consists of stages for accurate feature extraction, residual prior estimation, and image restoration. First, the solution space is constrained by applying identity mapping to all-in-focus images, to extract an accurate feature representation (AFR) of ground truth images. Second, a dynamic fractional derivative network is designed to estimate residual maps as a prior to compensate for the lost details caused by the defocus blurring. Finally, we propose a residual-guided attention module to aggregate the defocused image and residual map. Under the guidance of AFR and residual priors, a sharp image is restored. Extensive experiments demonstrate that our 3SDeblur achieves superior performance on defocus deblurring, especially a significant improvement of 0.81 dB in PSNR on the LFDOF dataset, compared to state-of-the-art methods. The code is publicly available at https://github.com/lizhangray/3SDeblur .