The success of Transformer in NLP and its multimodal fusion potential demonstrate cross-domain generalization capabilities, making it a key focus in current AI. Considering medical data characteristics, such as sample scarcity and multimodality, this survey systematically reviews medical image segmentation trends and identifies two critical gaps: (1) excessive focus on accuracy over generalization, and (2) sample scarcity necessitating novel data utilization and annotation generation paradigms. We suggest developing full Transformer model with enhanced generalization based on ViT meta-architecture, and propose Mixture of Layers Attention (MoLA) method with dynamic patching to balance multi-scale feature extraction and computational efficiency through learnable inter/intra-layer attention sparsification. To address current limitations, we summarize model optimization techniques and training strategies, while outlining promising research directions.