Traditional imaging has a very limited capacity to capture information in poor visual conditions, such as low illumination or haze. By combining the complementary data from both modalities, infrared and visible image fusion has become a successful solution to this issue. Convolutional Neural Networks (CNNs) are the main feature extractors used in traditional fusion techniques. Even though CNN-based methods have shown some degree of success, they are unable to accurately model deep cross-modal interactions and long-range dependencies due to their intrinsic limitations in local receptive fields. As a result, fused images often suffer from detail loss and blurred object contours. This study proposes a two-stage training framework based on the Vision Transformer (ViT) to overcome these difficulties and capture enough feature information during fusion. ViT demonstrates a robust ability to model globally by utilizing the self-attention mechanism, which allows the network to directly capture dependencies between any pair of regions in the image. ViT is better at extracting globally consistent features that are essential for high-quality image fusion than CNNs, which need to stack multiple layers to indirectly expand the receptive field. The feature modeling capabilities of ViT in the context of fusion are further enhanced by the proposed two-stage training approach. Comparing the suggested approach to current baseline methods, experimental results show that it not only produces better visual performance but also reaches state-of-the-art outcomes across a number of quantitative evaluation metrics.