Detecting Malware in Encrypted Network Traffic Using Machine Learning and TLS Fingerprints
摘要
The increasing use of encryption protocols such as Transport Layer Security (TLS) has led to enhanced privacy and security for internet users. However, this development has also enabled malware to conceal its communications within encrypted channels. In this work, we explore the application of machine learning techniques to detect malware in encrypted network traffic. To this end, we compare two distinct approaches: one based on statistical flow features and the other one based on TLS fingerprinting (JA4+). In order to accomplish this objective, we have developed and evaluated two state-of-the-art solutions—a MalDIST-inspired model and JA4+ fingerprint-based classification. The experimental results, based on a curated dataset, show that both models exhibit high levels of accuracy. Notably, JA4+ fingerprinting offers a favorable trade-off between accuracy metrics and overall processing speed compared to the MalDIST-inspired model, making it a promising candidate for deployment in real-world environments. Furthermore, we introduce JA4TS, another fingerprinting technique that focuses on TCP SYN-ACK packets, enhancing the capability of the JA4+ framework to predict malware by identifying TCP/IP stack characteristics, a subject of particular relevance as SNI encryption becomes more prevalent. These findings underscore the efficacy of lightweight, metadata-based models for malware detection in encrypted traffic, particularly in the context of IoT and IIoT networks, where privacy and efficiency are paramount.