Healthy tissue inpainting has many applications, for instance, generating pseudo-healthy baselines for tumor growth models or simplifying image registration. In prior editions of the BraTS Local Synthesis of Healthy Brain Tissue via Inpainting Challenge, denoising diffusion probabilistic models (DDPMs) demonstrated qualitatively convincing results but suffered from low sampling speed. To mitigate this limitation, we present a modified 3D wavelet diffusion model (WDM3D), denoted as fastWDM3D. Our proposed model employs a variance-preserving noise schedule and reconstruction losses over the full image as well as over the masked area only. Using fastWDM3D with only two time steps we achieved a SSIM of 0.8571, a MSE of 0.0079, and a PSNR of 22.26 on the BraTS inpainting test set. The 3D inpainting process took only 1.81 s per image. Compared to other DDPMs used for healthy brain tissue inpainting, our model is up to \(\sim \) 800 \(\times \) faster but still achieves superior performance metrics. Our proposed method, fastWDM3D, represents a promising approach for fast and accurate healthy tissue inpainting. Our code is available at https://github.com/AliciaDurrer/fastWDM3D .