Many publicly available medical image datasets are characterized by complex structures, blurred lesion edges, and low contrast between background and segmented regions. Although many models perform well in image segmentation tasks, they still face challenges in dealing with regions with blurred edges and localized lesion regions that are not obvious. In this paper, we propose an innovative Swin Transformer and CNN dual encoder model, MT-Net, by adapting the CNN backbone encoder to achieve better performance on medical image segmentation tasks. MT-Net introduces a multi-scale fusion module, MFM, which learns the multi-scale information through four different branches, which helps the model to better capture the localized lesion information from complex data. In addition, we designed a coding network MFB, which realizes the interaction of information between high and low dimensions on the basis of the original, and adds residual connections to make the model fuse different scales of information, which improves the model’s expression and learning ability. We also add a cross-attention module TCD to the model, which gradually refines and fuses the information of different hierarchical features through two cascading cross-automatic attention computation steps, improving the model’s ability to capture localized foci and fuzzy edges. We perform segmentation experiments on the ISIC2016, ISIC2017, and ISIC2018 datasets as well as the GLaS dataset. The results show that our model significantly outperforms the traditional model in all the metrics, which verifies the validity of the model MT-Net structure and the rationality of the improvement. Therefore, our improved Swin Transformer and CNN hybrid dual encoder model shows excellent performance in medical image segmentation task, which provides a valuable reference for research and application in related fields.