Object detection faces numerous challenges in complex scenarios, especially in situations with occlusions and small objects. To address this problem, SCS-YOLO (Self-attention Convolution with Soft-NMS based on YOLO12) is proposed in this paper, which is based on YOLO12 and integrates the mix of Self-Attention and Convolution (ACmix) module and the Soft Non-Maximum Suppression (Soft-NMS) loss function. Firstly, Dynamic fusion of multiscale features and local feature extraction through convolution can be achieved by the ACmix module. Secondly, the Gaussian attenuation mechanism of Soft-NMS is introduced into the model. Last, computational parameters can be decreased effectively due to SPPF-LSKA module is added into the network. The model’s mAP@0.5 is increased from the base-line of 0.713 to 0.762 on VOC2012. Moreover, the detection accuracy for small object categories such as bottles and pottedplants has been improved by 24.24% and 42.89%. Additionally, the recall rate for occluded objects in dense scenes has been increased by 11.5%. Furthermore, its robustness is verified on the COCO 2014 dataset. The mAP@0.5 metrics of SCS-YOLO is on average 7.03% higher than the other models on the COCO 2014. This research provides a new method for combining feature enhancement and post-processing optimization improvement strategies which can be used in complex environments and some industrial detection.