This study introduces a hierarchical classification framework for automated grading of Alphonso mangoes, segmenting them into four critical categories: unripe, ripe, overripe without internal defects, and overripe with internal defects. A fine-tuned Convolutional Neural Network (CNN) with 6.4 million parameters forms the core of the model, offering a balance between computational efficiency and classification accuracy. The approach incorporates five data augmentation strategies rotation, flipping, brightness adjustment, and distortion to expand the dataset and improve generalization. Beyond deep learning implementation, the study also analyzes the impact of these augmentation techniques on conventional machine learning models, revealing their limitations in handling transformed handcrafted features and highlighting the superiority of deep learning-based feature extraction. Classification proceeds through progressive binary classifiers in a hierarchical structure, simplifying complex decisions into interpretable sub-tasks. The proposed method is benchmarked against leading architectures, including Vision Transformers (Base_16, Large_16, Huge_14) and CNNs (VGG16, InceptionV3, EfficientNetB0, ResNet50), with and without fine-tuning. Evaluation on 979 original and 2,935 augmented images yields an average classification accuracy of 97.95%, significantly outperforming prior conventional methods. These results confirm the model’s robustness, efficiency, and real-world applicability in resource-constrained agricultural settings. The work lays a solid foundation for scalable, accurate, and cost-effective fruit grading systems with future scope for real-time deployment.