Medical image segmentation is a crucial step for accurate diagnosis and treatment planning, as it provides quantitative information about anatomical structures and pathological lesions in various clinical scenarios. However, the existing methodologies have limitations in terms of their generalizability and computational efficiency. In this study, we propose SwiftMedSAM, an ultra-lightweight prompt-based general model, to enable efficient medical image segmentation even in resource-constrained environments. Based on LiteMedSAM, we significantly reduced the model size and computational complexity through the hyperparameter optimization of the image encoder and mask decoder components. SwiftMedSAM showed remarkable performance across various imaging modalities enabling real-time inference in resource-limited computing environments. It achieved a validation score of 0.75 demonstrating that SwiftMedSAM outperformed the existing methodologies in terms of the trade-off between accuracy and efficiency. Owing to its unprecedented generalizability and low computational cost, SwiftMedSAM is expected to enable high-quality medical image analysis in resource-constrained settings, thereby contributing to advancements in precision medicine and telemedicine.