In medical imaging, accurate and efficient segmentation is crucial for diagnostics, treatment planning, and monitoring disease progression. Traditional methods, while capable of providing reliable results, often require substantial computational resources, which may not be feasible on devices with limited capabilities such as standard CPUs and limited RAM. To address this challenge, we present an optimized universal segmentation framework that leverages a lightweight image encoder RepViT-M0.6, distilled from Swin-T. Our comprehensive analysis of the online validation set shows that our method surpasses the baseline LiteMedSAM model. We achieve a Dice Similarity Coefficient (DSC) of 84.68% and a Normalized Surface Dice (NSD) of 85.28%. Furthermore, the method achieves a more than threefold increase in inference speed, making it viable for real-time applications on devices with limited computational power. This demonstrates that our adaptation significantly enhances processing speed and resource efficiency without sacrificing accuracy.