Image fusion aims to integrate complementary information from different images to provide richer scene details. However, in real-world scenarios, low-light illumination conditions not only affect the brightness, contrast, and color of visible images but also impact the quality of fusion results. Therefore, this paper proposes an illumination-free infrared and visible image fusion method, named IFFusion. Specifically, the whole task is considered as a multitasking problem: fusion and luminance adjustment. Firstly, based on the Retinex theory, the visible image can be decoupled into reflectance and illumination components, with reflectance should be consistent under different natural lighting conditions. Therefore, we encourage the network to learn the prior of reflectance consistency from pairs of visible images. Secondly, to obtain a fused image with appropriate brightness, a luminance perception network (LP-Net) is designed to perceive the scene brightness of the input visible image. Here, we use the visible image with normal brightness to guide LP-Net for adaptively adjusting the illumination component. Extensive experiments on public datasets show that the proposed method has better performance than the state-of-the-art fusion methods. Last but not least, we performed downstream task experiments on object detection and the improved performance verified the effectiveness of IFFusion.