With the advancement of agricultural modernization, machine vision technology has garnered increasing attention in the agricultural product sorting domain. This study focuses on addressing the developmental requirements of the Chinese specialty fruit industry, with a particular emphasis on the high-value fruit - loquat. We propose an integrated system design scheme for an automated sorting system that incorporates defect detection, ripeness assessment, and fruit diameter classification. Employing deep convolutional neural networks (DCNN), we achieve end-to-end feature learning and defect detection in loquat images. The maturity of loquats is determined via color space transformation and threshold segmentation methods, while machine learning algorithms are employed to classify and grade extracted fruit diameter data. In terms of system design, we present a comprehensive approach to integrate various technologies into a unified system, thereby enhancing overall sorting efficiency and accuracy. Test results demonstrate significant improvements in production efficiency, quality stability, and reductions in labor costs and misjudgment rates. This study offers novel insights and solutions for the advancement of smart agriculture.