In recent years, the development of self-supervised Multi-View Stereo based on deep learning has effectively alleviated the reliance of supervised methods on ground-truth depth maps. However, existing methods often suffer from high memory consumption, which limits their practicality in real-world deployment. To this end, this paper proposes an efficient dual-branch self-supervised MVS Network, called WGMVSNet for short, which reduces memory usage while improving the accuracy of 3D reconstruction. In the overall architecture, the normal branch and the image enhancement branch are combined to refine the self-supervised signal and improve reconstruction quality. During feature extraction, we design wavelet attention block and dynamic upsampling block to capture high-frequency information in the frequency domain, enhancing the quality of image features. For cost volume regularization, we replace conventional 3D CNNs with a combination of 3D Ghost convolution and 3D CBAM attention, which reduces computational cost while maintaining accurate similarity representation in the cost volume. In terms of loss design, a refined depth smoothness loss is proposed to achieve higher accuracy in depth estimation. Qualitative and quantitative experiments confirm the proposed method’s effectiveness in achieving a favorable balance between reconstruction quality and memory efficiency.