In the current complex and dynamic network environment, traditional security protection methods are difficult to effectively respond to the constantly evolving security threats. This paper aims to improve the accuracy and efficiency of threat detection. Firstly, a GAN framework based on large-scale network traffic data is constructed to collect multi-dimensional data features including normal traffic and malicious traffic, and then the raw data is processed through data cleaning, feature extraction and dimensionality reduction techniques. Secondly, supervised learning algorithms are used for classification, identifying malicious traffic, and combining unsupervised learning algorithms to detect unknown threats. In order to further improve system performance, this paper introduces reinforcement learning to optimize intrusion detection systems, achieving dynamic adjustment of detection rules and resource allocation. In the experiment, the KDD Cup 1999 dataset is used for model evaluation, and the threat detection accuracy of GAN reached 99.8%. GAN has significant potential for application in network security and can effectively improve threat detection capabilities and system optimization levels.