Accurate identification of local rice varieties is critical for maintaining agricultural value chains, yet traditional methods are inefficient for large-scale use. This study proposes a deep learning-based solution for classifying four Indonesian rice varieties (pandan wangi, IR64, Rojolele, and glutinous rice) using images captured from standard smartphones. We created a novel dataset of 1,474 images and evaluated two distinct object detection architectures. The first model, RetinaNet, was optimized for maximum precision, leveraging a custom classification head and Focal Loss to handle subtle inter-class differences. RetinaNet achieved a Total Accuracy of 0.9227 (92.27%), demonstrating its robustness in fine-grained classification tasks. The second, SSDMobileNetV2, was evaluated for on-device deployment. Our experimental framework establishes the viability of using accessible mobile technology for this fine-grained agricultural classification. The results demonstrate that RetinaNet achieves high accuracy suitable for laboratory analysis, while the performance of the lightweight SSDMobileNetV2 highlighted the challenges of this task, underscoring the need for the higher accuracy provided by RetinaNet for reliable classification.