Computer vision tasks widely use convolutional neural networks (CNNs) due to their effective feature extraction capabilities. CNNs have numerous positive points, but they face challenges in capturing long-range dependencies due to convolutional operations. Transformers in natural language processing inspire the development of Vision Transformers (ViTs). ViTs demonstrate proficiency in capturing long-range dependencies; however, they face challenges in effectively modeling low-level features. Successful image segmentation necessitates an examination of both local and global dependencies. This paper introduces an enhanced HiFormer architecture to address this issue. This approach combines convolutional neural networks and transformers to effectively capture both low-level features and long-range dependencies. The suggested architecture integrates ResNeXt-50 CNN and Swin-L transformer in a structured hierarchy. The system employs a dual-layer fusion mechanism that successfully combines local and global features. The proposed model exhibited strong performance during evaluation on the Synapse dataset. The results indicate an average Dice score of 81.91% and an HD95 score of 17.15%. The results highlight the improved effectiveness of the proposed model in segmentation tasks.