<p>The locality of convolutional operations limits their ability to capture contextual information, especially in brain tumor segmentation tasks where rich inter-modal interactions are essential. To address this challenge, this study proposes a novel segmentation framework based on the optimized UNet, integrating multi-modal and unified-scale multi-window fusion attention mechanisms for brain tumor segmentation. The proposed multi-modal fusion Transformer (MF-Trans) employs a convolution operation to extract local information from four modalities and utilizes the proposed modality-exchange transformer to effectively combine inter-modal correlation and global information. Moreover, the unified-scale multi-window Transformer (UM-Trans) considers multiple window sizes for global attention to mitigate the limitations of traditional convolutional operations’ locality and the Transformer’s restricted global semantic information acquisition. Experimental results on glioma segmentation tasks demonstrate the effectiveness of the proposed method. Compared to state-of-the-art segmentation networks, our approach achieves superior performance with an average Dice coefficient of 0.8475, average precision of 0.8809, and an average Hausdorff distance of 1.2969. Our contributions can be summarized as follows: (1) The MF-Trans transmits information between modalities and closely connects the various modalities of information in brain tumors. (2) The UM-Trans addresses the limitations of a fixed-size global attention window, facilitating further interaction and semantic fusion in brain tumor images. (3) The overall network outperforms existing methods in glioma segmentation, showing improved delineation of tumor boundaries and sub-regions.</p>