Accurate and rapid image inpainting is of great significance for environmental perception in unmanned surface vehicles (USVs) with soiled lenses. However, existing methods still struggle to handle large-scale missing regions with complex semantics due to the lack of sufficient prior guidance. To address this problem, we propose a novel reference-guided image inpainting network based on progressive reference alignment and fusion for USVs, called RAFNet. Specifically, we devise a new reference alignment (RA) module to locally align the reference features with the corrupted features, thereby generating aligned features to guide image inpainting. Moreover, a cross-stage fusion (CF) module is presented to adaptively fuse multi-scale aligned features from different network stages. Building upon the RA and CF modules, a progressive reference alignment and fusion decoder is designed to gradually aggregate and refine contextual information from both the reference image and the corrupted image, ultimately yielding the inpainted result. Experiments on our maritime stereo image (MSI) dataset show that RAFNet outperforms the state-of-the-art baseline by 13.9% in PSNR and 6.4% in SSIM, while running at over 28 fps on an NVIDIA RTX 4090D GPU.