The growing demand for healthy food, driven by population growth and the prevalence of plant diseases, is a significant concern. Farmers are increasingly forced to cultivate their fields continuously, often relying on pesticides that degrade soil quality and contribute to the spread of various diseases. Disruptive technologies, such as automated cassava leaf disease detection using deep learning models, can play a vital role in promoting agricultural sustainability. While numerous deep learning methods have been explored, challenges remain, particularly in developing lightweight models that can quickly and accurately identify disease class variations while also being suitable for deployment on electronic devices. This research focuses on evaluating lightweight deep learning models, specifically CNN and transformer networks, for cassava leaf disease detection. Based on our experiments,  we propose an ensemble model consisting of lightweight models (CNN and transformer) for automated early disease detection using raw images. We have utilized the ResNeXt, EfficientNet-B5, and TinyViT lightweight models, optimizing the ensemble process through brute force approaches. Additionally, we experimented with center-crop and multi-crop image transformations to test these models, aiming to enhance performance and classify images based on the global information derived from the whole image. Our combined approach achieved state-of-the-art results with an overall recall of \(90.35\%\) on the unseen test cassava leaf disease dataset. With high accuracy, fewer parameters, and low computation time, our model is well-suited for deployment on mobile devices.