Multi-modal fundus image registration aligns retinal fundus images from different modalities to enable more effective disease analysis and diagnosis. However, the majority of state-of-the-art multi-modal fundus image registration methods are trained by leveraging segmentation labels to reframe the task of registering fundus images from different modalities as a single-modality segmentation labels registration task. The intricate vascular structures inherent in fundus images hinder the accuracy of segmentation models, which in turn obstructs the registration process. Addressing this, we pioneer a segmentation label free registration framework for deformable multi-modal fundus image registration. Specifically, the proposed framework consists of three key steps: image to image translation, coarse registration, and fine deformation registration. In the image-to-image translation, we convert color fundus images to fluorescein angiography images and design a struct loss to enforce structural similarity. Due to factors such as the shooting position and other conditions, there can be a spatial deviation between the moving images and the fixed images, so we use coarse registration for a rough warp. For fine registration, we introduce the Cross Fusion Transformer (CFT) and Cross Attention for Feature Fusion (CAFF) at the skip connection. This enables multi-scale, multi-channel feature fusion, effectively addressing semantic discrepancies and enhancing fine registration accuracy. Extensive experiments on available medical datasets prove the efficacy of the proposed CFT and CAFF modules and highlight the superiority of the proposed approach over the other methods.