Segmentation foundation models (SFMs) hold promise for medical image analysis, but their direct clinical application is limited by computational cost, potentially suboptimal accuracy, and fairness concerns. In this paper, we propose a novel framework to address these challenges by distilling knowledge from a heterogeneous ensemble of pre-trained SFMs, generating specialized, high-performance models for domain-specific medical image segmentation. Unlike existing single-SFM approaches, our methodology leverages the collective intelligence of diverse SFMs to enhance accuracy, fairness, and efficiency. A key contribution is a ground-truth-free knowledge distillation strategy using the ensemble’s aggregate predictions on unlabeled data to minimize reliance on manual annotation. Evaluated on a large, diverse dataset of CT and MRI scans from 702 individuals, our distilled model significantly outperforms individual SFMs and their ensemble average, achieving state-of-the-art segmentation accuracy, improved fairness across demographics (sex, age, BMI), and substantially reduced computational cost. These results offer a practical paradigm for leveraging foundation models in real-world clinical settings, mitigating key SFM limitations.