In order to ensure better product quality, manual inspection is used in many small-scale manufacturing industries. However, human error causes lower productivity, high penalties and lack of effectiveness. For an autonomous defect detection in real time applications, object detect based models have shown promising results. Contrary to conventional image processing techniques, object detection framework can also be trained for classification of the type of defects to identify the root cause. For effective training of the model, we have collected defective and non-defective samples of M6 and M8 sized nuts from manufacturing industries and created a labeled dataset using an industrial grade machine vision camera. There are 976 images in the training set and 244 images in the test set. To evaluate and ensure the usability of proposed setup in real-time settings, an inference dataset comprising 41 images is also collected from different sources. The dataset comprises three surface defect classes (dent, scratch, and crack) and a non-defective class. SSD MobileNet V2 FPNLite 640x640, SSD MobileNet V2 FPNLite 320x320, EfficientDet D1 640x640, and YOLOv5 are trained and compared by confusion matrix. YOLOv5 outperforms the other three models with the accuracy of 94.67%, recall 93.02%, precision 98.68%, and F1 score of 95.65%. Further, YOLOv5 is also found to be the most suitable for deployment in real-time scenarios for the proposed defect detection setup. It also has the highest confidence score for each detected class and minimum inference time.