In this work, we propose an early-fusion of infrared images and visible images to improve performance of object detectors under complex environments. We frame the fusion of infrared and visible images as a generation task, and propose a simple yet effective scheme built on diffusion model with guided image to generate the fused images, called DiffIVF. How to quickly generate fused images with high texture and intensity fidelity is an important issue for object detection. We address these issues in the whole process of the diffusion model. In the forward process, a cosine noise schedule is proposed to improve the smoothness of the noise level, which stabilizes the diffusion process for the fast sampling. In the reverse process, a large sampling interval is utilized to accelerate the denoising process. To compensate for the information loss during fast sampling, the visible image signal is selected to guide the fusion at each denoising timestep, aiming at enriching textural details and structural information. The guided image weight averaged with an intermediate fused image inputs into a pre-trained U-net for the next time of reversing. Notice that the proposed method needs no training or fine-tuning, which enables us exclusively to make use of off-the-shelf models for fusion. Experimental results show that the proposed method obtains better image quality scores of textures and intensity fidelity than previous methods, thus triggering better detection performance for the infrared-visible fused images. Code for https://anonymous.4open.science/r/DiffIVF .