Leaf disease detection and classification play a vital role in maintaining food security, economic stability, and promoting sustainable agricultural practices. The timely identification of cassava leaf diseases is essential for avoiding irreversible damage to crops. Conventional disease detection methods often involve manual inspection, which is labour-intensive, laborious and prone to human error. In this paper, we present a deep convolutional neural network architecture, namely MiXceptionLeaf, specifically developed for the automated detection and classification of four major cassava diseases. Our approach utilizes transfer learning, leveraging the powerful XceptionNet CNN, pretrained on ImageNet, as the fundamental feature extractor. To improve the model’s classification performance, we incorporate a classifier head based on the ConvMixer architecture. We’ve also introduced an extra loss term to complement the primary classification loss, achieving dual optimization. This novel approach enhances both classification accuracy and attention diversity, promoting unique feature extraction. We have evaluated our model on a cassava leaf disease dataset that has been meticulously compiled and consists of 11,414 images. Our model has outperformed other state-of-the-art techniques in terms of precision, recall, accuracy, and F1-score, with an average precision of 98.78%, average recall of 98.85% and an average F1-score of 98.80%.