<p>Maritime surface object detection based on unmanned surface vessels (USV) is a key technology for the safe navigation of USV. However, maritime surface object detection algorithms frequently encounter challenges such as large parameter sizes, complex maritime surface image backgrounds, and uncertain object sizes, which significantly affect the deployment and detection accuracy of object detection algorithms on unmanned vessels. In response to these challenges, this paper introduces a lightweight maritime surface object detector named MLViT-YoloX. In this work, initially, a lightweight backbone network called MLViT is designed to enhance the feature extraction capability. Specifically, a new convolutional self-attention fusion mechanism is proposed to replace traditional self-attention, which fully utilizes the locality of convolution, capture static correlations between local information, to guide the learning of self-attention matrices. Secondly, lightweight cross-correlation module (LCCM) and path aggregation network (PANet) are integrated as the neck of detector to capture the interdependencies between adjacent scale features, and decoupled heads are used as the prediction part. Additionally, multiple training strategies such as the Simple Optimal Transport Assignment (SimOTA) strategy and multiple data augmentation are integrated. Finally, the SeaShips dataset and the Singapore Maritime Dataset were used to evaluate and compare MLViT-YoloX with other mainstream detectors. The results indicate that MLViT-YoloX has significant advantages in terms of <i>mAP</i> and computational speed. Compared to the baseline YoloV7-Tiny, MLViT-YoloX achieved a 10% and 6.5% increase in <i>mAP</i> on the two datasets, respectively, and also demonstrated faster computational speeds, showing significant effectiveness.</p>