Infrared and visible image fusion technology plays a crucial role in various fields. However, in practical applications, the majority of available fusion methods struggle to achieve satisfactory fusion results due to a lack of consideration for reducing the modality gap between visible and infrared images during the feature fusion process. To remedy this issue, this study proposes MPNet, an improved model for infrared and visible image fusion that employs a shared feature distribution alignment (SFDA) strategy to mitigate the modality discrepancy and enable fine-grained fusing of shared features. Additionally, to suppress noise while enhancing the high-frequency details of the fused image, a simple but effective phase detail loss (PDL) is introduced to encode the high-frequency information from the source images into the fused image via Fourier phase spectrum, significantly enhancing the visual quality of the fused results. A series of experiments on several public benchmarks demonstrate the superiority of our method.