Every year, a large number of papaya farmers suffer significant losses due to diseases affecting their crops. Unfortunately, many of these farmers lack the knowledge and tools to detect these issues early on. Often, by the time the disease is noticed, the fruits are already damaged and can’t be saved. Because of this recurring problem, some farmers have even become hesitant to grow papaya again. To help address this issue, researchers have turned to deep learning technologies to develop a system that can automatically detect and classify the condition of papaya fruits. In this study, papaya samples were collected from farms in Vijayawada, Andhra Pradesh. The goal was to classify the fruits into three categories: unripe, ripe, and defective. A dataset containing 4500 images (1500 each category) was created. These images were pre-processed through steps like resizing, normalization, and label encoding to get them ready for model training. Several deep learning models were developed from scratch, including CNN, AlexNet and VGG16, to perform the classification task. Among these, a custom CNN model with 12 convolutional layers and 3 fully connected layers delivered the best results, achieving a test accuracy of 94.22%. This research demonstrates how deep learning can play a key role in automating agricultural processes, especially for tasks like fruit classification and quality checking.