In the agricultural supply chain, precision in fruit sorting and grading is paramount for maintaining quality standards and operational efficiency. Traditional manual sorting methods are often challenged by inconsistencies, lack of scalability, and susceptibility to human error, necessitating the adoption of automated, machine-based solutions. This study presents a novel framework for tomato grading using deep learning techniques tailored for implementation in agricultural sorting machinery, facilitating real-time processing capabilities. The developed framework utilizes a thoroughly annotated dataset comprising tomato images classified into five specific categories: Grade 1, Grade 2, Grade 3, Immature, and Spoiled. To optimize classification accuracy, semantic segmentation techniques were applied to remove irrelevant background elements, thereby enhancing the clarity and focus of the extracted features. We sequentially improved the ConvNeXt architecture by adding in Squeeze-and-Excitation (SE) blocks for explicit channel-wise attention, performing pruning on the model to reduce its size, and adjusting the classifier via the use of Global Average Pooling (GAP) and LayerNorm to obtain better generalization and computational efficiency. Notably, while introducing these architectural enhancements, we ensured that the average inference time per sample was minimally affected. The original ConvNeXt-Tiny model had an inference time of 0.113 s, and the enhanced model maintained a similar speed, with only a slight increase to 0.116 s. This highlights that the proposed system not only achieves a 95.69% accuracy rate but also retains its scalability and efficiency for real-time and lightweight implementations in edge-based devices. This enables reliable and automated grading processes, addressing the demands of modern agricultural supply chains.