Multi-focus image fusion is a highly challenging task in robotic vision. Although significant progress has been made in deep learning-based Multi-focus image fusion, existing methods still have some limitations in generating satisfactory results, especially for continuous outdoor scenes. There are two main problems: i) for outdoor scenes, the defocus diffusion effect can lead to blurred areas in near focus and far focus images; ii) for continuous depth scenes, there is no boundary between near and far focus images. To deal with the above problems, this paper proposes an end-to-end multi-focus image fusion method - XFusion, which takes an X-shaped multi-scale convolutional module as the basic network and the transformer module as the fusion unit. The fusion unit utilizes cross-focus feature statistics to effectively de-blur the defocused area to a certain extent. Furthermore, we constructed a new dataset for outdoor scene multi-focus image fusion. Experiments show that the proposed XFusion achieves excellent performance compared to other competitors on our dataset and several other benchmark datasets. The source code and dataset can be found on https://github.com/Shouxi-Zhao/XFusion .