The rapid development of digital payment systems has brought great convenience but also introduced significant fraud risks. Current fraud detection systems face two major challenges: difficulty with identification of overlapping samples due to data imbalance, and low sensitivity to emerging new fraud patterns. To address these issues, this paper proposes an intelligent fraud detection system based on a dual-module architecture (OS-DFDA). The main innovations include: (1) Design of an OV-WGAN data balancing preprocessing module that precisely models samples in overlapping regions by integrating Wasserstein distance and overlapping coefficients, effectively improving the quality and reliability of balanced samples; (2) Development of a dual-module detection framework that combines Decision Tree and autoencoders, enabling collaborative detection of both known and unknown fraud patterns while maintaining high precision and significantly improving recall rate. Experiments on real-world digital payment datasets demonstrate that the system achieves an F1-score of 86%, outperforming the existing mainstream methods.