Food, especially fruits, is essential for living, but eating spoiled food can lead to illness or even death, especially in developing countries where sanitation is less practiced. None of the literature done so far addresses the issue from the standpoint of a low-income end user who is to consume the fruit, whose spoilage is to be identified using edge devices, like cellphones, with poor computational specs. The difficulty lies in developing a model that satisfies three conflicting needs: it must be precise to prevent infection, tiny enough to operate on edge devices like a consumer’s smartphone, and responsive enough to allow the user to consume immediately. To address these competing goals, this paper suggests an architecture for detecting fruit spoilage that contains pre-processing (data augmentation, edge, and shape detection), fine-tuning of small pre-trained models (MobileNets, DenseNet, and Xception), evaluation, size, and response time reduction (model quantization), and deployment modules implemented using Python programming. The proposed approach demonstrated competitive performance compared to existing methods (maximum accuracy of 92.39% for MobileNets which is slightly less than two baselines because it uses small models and separate test sets). In the best case, the size of the Xception model is 21.7% of the original, while MobileNets is 35 \(\times \) faster with negligible effect on performance. However, it still requires further performance improvement before consumer distribution, though a straightforward GUI-based prototype is constructed for demonstration.