Rectal cancer is a significant contributor to cancer-related mortality, with the pathologic complete response (pCR) being crucial for organ preservation. Accurate prediction of pCR through advanced imaging techniques, particularly multiparametric magnetic resonance imaging (mp-MRI), is essential for optimizing treatment strategies in locally advanced rectal cancer (LARC). This study addresses the limitations of existing deep learning models that often suffer from overfitting due to the high similarity among mp-MRI sequences. We propose a novel multimodal MRI feature fusion module designed to reduce inter-modal correlation while enhancing feature complementarity. By implementing regularization constraints for orthogonality and incorporating a cross-attention mechanism, we effectively fuse features from different MRI modalities. Our method utilizes diffusion-weighted imaging (DWI) as the query, the apparent diffusion coefficient (ADC) as the key, and T2-weighted imaging (T2W) as the value. We retrospectively collected mp-MRI data from locally advanced rectal cancer (LARC) patients who received only neoadjuvant chemotherapy. Experimental results indicate that our method significantly outperforms existing competitive models.