BFDet: A Method for Detecting Malicious Traffic with Ultra-low False Positive Rate Based on TLS Behavior Flow
摘要
With the widespread adoption of encryption technologies, the proportion of encrypted network traffic has surged, posing new challenges for detecting encrypted malicious traffic. Although existing methods have made some progress in this field, they still face the following challenges in real-world network environments: (1) the detection performance on minority class samples is unsatisfying under imbalanced data distribution; (2) the false positive rate (FPR) for benign traffic is too high to meet the detection needs in large-scale traffic scenarios. To address these issues, we propose a novel Transport Layer Security (TLS) behavior flow-based malicious traffic detection method, which is abbreviated as BFDet. The method employs a three-layer neural network architecture specifically designed for feature aggregation from packet to flow, feature aggregation from flow to behavior flow, and further optimization of behavior flow feature vector. Through hierarchical feature aggregation, BFDet effectively exploits the potential relationships between features of different granularities, improving the detection performance of TLS malicious traffic. Experimental results show that BFDet while maintaining high detection accuracy, reduces the average FPR of benign traffic in a real network traffic dataset to 0.005%. Compared to the FPR of existing methods: AppScanner’s 3.2%, FS-Net’s 0.2%, and ET-BERT’s 0.1%, the reduction rates are 99.84%, 97.5%, and 95%, respectively, outperforming state-of-the-art methods.