Although current deep learning-based models have achieved tremendous successes in medical segmentation tasks, the deployment of such models on CPU-only devices is still challenging due to the substantial computational resources required for segmentation inference, especially for 3D medical images. Small-sized models capable of efficient inference have been proposed to mitigate the computational overheads. However, these small models usually largely sacrifice the segmentation accuracy. In order to tackle the challenge in compliance with the requirements of MICCAI FLARE 2024 Challenge Task 2, i.e., deploying advanced 3D abdominal CT segmentation models in non-GPU environments while maintaining high accuracy, we introduce a multi-scale knowledge distillation method to train a student model that maximally retains the segmentation performance of the teacher model. In order to improve the segmentation performance of tiny organs and overcome the quality issues of pseudo-labels themselves, we also design a weighted composite loss function to train the model. Furthermore, for efficient segmentation inference on CPU-only devices, we introduce a liver-based Z-axis Region-of-Interest (RoI) localization strategy, which effectively improves the segmentation efficiency. Experiments on the MICCAI FLARE 2024 datasets have shown significant improvements in both segmentation accuracy and efficiency. The proposed method achieves an average organ Dice Similarity Coefficient (DSC) of 88.70% and a Normalized Surface Dice (NSD) of 94.29% on the public validation set. In the FLARE 2024 Task2 online validation, the method achieved an average organ Dice Similarity Coefficient (DSC) of 88.47% and a Normalized Surface Dice (NSD) of 94.71%, with an impressive average inference time of 12.33 s. The code is available at https://github.com/lay-john/FLARE24-Task2