In recent years, the development of medical imaging technology has transformed imaging solutions from laboratory-based to point-of-care imaging with real-time capabilities. However, these point-of-care devices are often constrained by environmental factors such as ambient light and noise, leading to poor image quality and consequently affecting the diagnostic accuracy of point-of-care devices. Furthermore, due to the need for lightweight models in point-of-care devices, traditional models fail to meet requirements in terms of computational resources, model parameters, and inference time. Therefore, to address the aforementioned issues, this paper proposes an optimized lightweight student model that focuses on residual information. A lightweight structure based on Shift MLP is designed on the residual branch of the model to enhance the model’s capability to acquire spatial feature information at multiple scales. Simultaneously, we propose a multi-teacher distillation strategy to improve the accuracy and noise-resistance of the student model. Firstly, we introduce an adaptive learning approach based on auxiliary teachers, leveraging unlabeled and noisy data for adaptive learning to enhance the model’s robustness. Then, we design a global teacher model to enhance the accuracy of the student model and indirectly improve the teaching ability of auxiliary teacher model, thereby achieving knowledge transfer at a global level. We evaluate our approach on two public medical image classification datasets, and the results demonstrate that while almost maintaining accuracy, we reduce the number of parameters by 38 times, decrease computational complexity by 11 times, and achieve an inference time of only 18.94ms on CPU.