<p>Traditional diffusion-based speech enhancement models generally require numerous sampling steps, and optimizing these potent diffusion models incurs high computational costs. Existing accelerated sampling techniques often sacrifice speech quality under constrained computational budgets, resulting in suboptimal restoration results. To address this issue, we propose an efficient residual denoising diffusion speech enhancement model. This method decouples the conventional single forward diffusion process into residual diffusion and noise diffusion, which reduces the number of reverse iterations required for accurate reconstruction of clean speech from noisy and better handling of speech with non-Gaussian noise. To further improve the model performance, we proposed a two-stage training strategy and improved objective for the pre-training stage to reduce error accumulation in the sampling process. Experiments on the VoiceBank-DEMAND dataset demonstrate that RDSE outperforms the mainstream SGMSE+ model in metrics such as PESQ and SI-SDR with only 3 sampling steps, while requiring less than 1/5 of its model capacity. Notably, RDSE exhibits robust noise reduction and dereverberation capabilities even under mismatched noise types and reverberant conditions. Source code and audio examples are available online.</p>