Defects, such as stains and scratches, in the answer document image can degrade the image quality and seriously reduce the accuracy of text and mark recognition. The existing methods to detect these defects are difficult due to weak ability to suppress interference from the complex background, and result in lower detection accuracy. This paper proposes a new method that deeply explores visual features to detect defects using deep learning. Firstly, the Oriented Bounding Box (OBB) is used to minimize the bounding rectangle for abnormal objects with flexible shapes. Secondly, a new convolution module combining the Space-to-Depth Convolution (SPDConv) is designed to improve the feature extraction ability for objects with complex backgrounds. In addition, the attention mechanism with Multi-Scale Context Aggregation (MSCA) is employed to effectively capture multi-scale information and establish dependent relationships between remote pixels. In order to detect long-line defect objects, the P6 detector head is introduced to expand the receptive field range. Finally, the proposed method is integrated into YOLOv8 to verify defect detection. The experimental results show that the proposed method can effectively detect the four types of defects in answer document images, and the \({mAP}_{50}\) reaches 92.1% on our dataset (The dataset is available a https://github.com/micangdao/defectanswersheetimage ).