Produces are manually inspected and graded, which is a laborious process. The manual inspection includes the color, texture, shape, and size for assessing the cleanness and grading of fruits. It is quite a complicated, tedious and labor-intensive task. This process is also prone to error due to human intervention. Hence, to overcome the aforementioned error and assess the vegetables and fruits quality in an effective manner, this work presents a model based on deep learning, called SDNet, for evaluating the quality of fruits and vegetables. The proposed model also utilizes the transfer learning to detect defects in fruits while incorporating key attribute based on color, shape, size, and texture. In addition to it, an attention-based feature fusion technique is utilized to figure out the most prominent characteristics for defect detection. The performance of SDNet is evaluated using apple and mango fruit datasets, and results are compared with MobileNet, ResNet50, VGG16, EfficientNet, support vector machine with Deep Features, Improved YOLOv4, and Vision Transformer (ViT). The experimental results demonstrated that SDNet outperforms existing models in the context of an average higher accuracy rate of 92.08%, precision rate of 95.01%, recall rate of 92.60%, and an F-score rate of 93.79%. The findings show that, in comparison with alternative methods, the suggested SDNet yields more accurate and trustworthy results. Hence, the suggested model also shows a more reliable and expandable way to calculate the quality of produce.