Unmanned driving technologies occupies a significant position in the field of open-pit mine. The intelligent detection technology plays an important role in unmanned driving application. However, in the complicated mine scenario, object detection accuracy is pretty poor. For achieving good detection accuracy in the open-pit mine scenario, this work created an open-pit mine dataset and analyzed it from different dimensions. Secondly, we proposed a hybrid attention module composed of multi-head attention and single-head attention module to focus on the interest region. Besides, an extra branch was designed with CNN network combined with the CBAM block to strengthen small objects information. And the convolutional layer was substituted with depthwise separable convolution to decrease calculation parameters and realize real-time detection. As a result, the detection result demonstrates that our proposed network has a 2.1% higher mAP than the classic YOLOv4 model. Furthermore, the heatmap indicates that our proposed network could help in capturing rich discriminative feature representations and improve the expression of targets. Experimental results validate that our proposed algorithm could achieve good performance in complex open-pit mine scenario.