Current multimodal MRI segmentation methods often use simple fusion strategies, such as direct concatenation or independent feature extraction for each modality. These approaches fail to leverage modality similarities and specificities, neglecting inter-modal relationships, which leads to suboptimal tumor segmentation. To address this, we designed a refined multimodal fusion strategy. Specifically, MRI modalities are categorated into two groups based on their ability to identify different tumor regions. Modalities within the same group focus on the same region, while those in different groups target distinct regions. A Hybrid Attention Feature Enhancement (HAFE) module is designed to extract and enhance consistent features within each group, highlighting commonly attended regions while suppressing irrelevant regions to improve boundary clarity. The Global Feature Difference Supplementation (GFDS) module, based on Swin-Transformer, extracts differential features between modality groups and performs feature supplementation through a cross-attention mechanism to better distinguish tumor regions. In the bottleneck layer, we emphasize critical views in high-level semantic features and perform global fusion of regional and directional information to enhance the model’s understanding of complex tumor structures. We compared the proposed model with other models on the BraTS2020 and BraTS2021 datasets, and the proposed model achieved superior segmentation performance.