The conveyor belt has a wide range of applications in industrial settings, while the defect in the conveyor belt is a critical factor that affects normal production, making the detection of the corresponding defect particularly important. However, traditional YOLO-based conveyor belt defect detection methods struggle to balance real-time performance and speed, limiting their practical application. To address this issue and provide higher detection accuracy and speed, in this work, we introduce DefectFormer, a fast and accurate real-time end-to-end object detector based on transformers. We perform conveyor belt defect detection experiments using data collected from a line scanning camera, analyzing both the real-time performance and accuracy of the detection. The results demonstrate that, compared to other detectors, DefectFormer strikes a balance between inference efficiency and accuracy, better meeting the demands of industrial environments.