Text-driven fashion image editing modifies apparel elements in human images based on natural language descriptions to generate diverse outfit combinations. This technology holds significant potential for applications such as virtual try-on and human-computer interaction. However, current diffusion model-based editing methods still face several technical challenges: limited model capacity may lead to insufficient textual comprehension, causing geometric and texture discrepancies between generated garments and their textual descriptions. Additionally, unintended alterations to non-target regions during editing often compromise image coherence. To address these issues, we propose TMFit, a novel editing framework that leverages semantically robust masks for guided generation. Our approach consists of two key components: (1) The TMNet module, which integrates textual semantics with human body modality information to produce precise garment masks; and (2) A dual-guidance mechanism that synthesizes semantically aligned apparel content within target body regions, driven by both TMNet-extracted spatial body features and textual descriptions. Experimental results demonstrate that TMFit significantly outperforms existing methods in terms of generation stability, editing precision, and text-image alignment, achieving superior performance in both qualitative visual assessment and quantitative metrics.