Recent advances in diffusion models have demonstrated impressive capabilities in image generation. However, these models rely on multiple denoising steps, each requiring a neural network to predict noise, significantly affecting generation speed. As a result, accelerating diffusion models have become a key area of research. In this study, we aim to reduce the number of noise predictions while retaining the same number of denoising steps. To achieve this, we propose CacheNoise to accelerate diffusion models by reusing noise: for timesteps where the noise is highly similar to that of adjacent steps, we skip noise prediction and directly reuse the noise from the previous step. Additionally, we introduce the concept of relative change to assess the similarity between noises and design a caching mechanism based on this similarity. The process of noise reuse can be seen as introducing new error noise while denoising, but due to the similarity of noise between adjacent steps, this new noise is minimal and can be corrected in subsequent noise prediction steps. Experimental results validate the effectiveness of our method: on U-Net architectures, our approach achieves nearly a 2 \(\times \) speedup on DDPM with only a minor drop in FID, while on Stable Diffusion, we achieve up to a 3 \(\times \) speedup with even better Clip Score compared to the original model. On LDM, our method outperforms others at high acceleration ratios. Furthermore, our method performs better on transformer architectures than fast sampling methods.