DNA origami technology uses DNA molecular self-assembly to construct nanostructures, which is widely used in many fields. Still, DNA origami molecular detection faces challenges such as small data sets, small targets, and close to the background color. This study proposes YOLORIG algorithm, which is improved based on YOLOv8 framework. Its Backbone introduces iAFF convolution layer and GAM attention mechanism layer to enhance feature extraction and focus on key regions; its Neck uses FPN-like structure to fuse multi-scale features; and its Head adds MSDA attention mechanism layer to optimize prediction. In the experiment, AFM color image and TEM gray scale image data sets were used to compare YOLOv5n, YOLOv8n and other models. The comparison experiment shows that YOLORIG is better than other models in mAP index and the reasoning speed is similar. Qualitative tests show that it has good detection and generalization ability on different data sets and clip-scale images. In general, while maintaining the detection speed, YOLORIG algorithm improves TEM data set by 4.2% and YOLOv5 by 5.8% on mAP, respectively. Compared with YOLOv5s in AFM data set, YOLORIG algorithm improves multiple indexes and decreases the number of parameters and GFLOPs, achieving a good balance between model performance.