The objective of infrared and visible image fusion is to integrate multi-sensor data into a comprehensive image that combines complementary information, including salient features from infrared images and rich texture details from visible images. Infrared and visible image fusion methods based on diffusion models have demonstrated superior performance. However, the absence of ground truth during the training process still indirectly leads to insufficient extraction of complementary information. Moreover, conventional cross-attention mechanisms only concentrate on the correlation between source images but overlook the more crucial fine-grained differences between source images. To address the aforementioned limitations, this paper proposes an infrared and visible image fusion method based on reverse cross-attention and diffusion model (RDIF). Specifically, we extract diffusion features that capture both visible and infrared information within a latent space via forward and reverse diffusion processes. And we design a reverse cross-attention (RCA) mechanism to extract differential information between source images. Furthermore, the proposed intensity loss function weighted by cross-modal difference helps further enhance the capability of RCA, precisely balancing texture details and salient targets on a pixel-wise basis. Finally, we conducted comparative experiments against other state-of-the-art image fusion methods on three public datasets (MSRS, \(\text {M}^3\text {FD}\) and RoadScene), and the results demonstrate that our method is more robust and effective.