Enhancing DoH Traffic Classification Using LLM Embeddings: Evaluation of Traditional, LLM-Based, and Hybrid Models
摘要
The growing use of DoH protocol poses a significant challenge for network security systems. The encryption makes traditional traffic inspection techniques ineffective. In this study we evaluates the effectiveness of transformer based Large Language Models(LLMs) in comparison with conventional Machcine Learning (ML) and Deep Learning (DL) modles in classification of encrypted DoH traffic. We explore three kinds of design models; conventional ML/DL classifiers, standalone LLM classifiers using pretrained embeddings from models like DistilBERT, BERT and GPT2 and a hybrid LSTM classifier using LLM embeddings. Experiments are conducted on the CIRA-CIC-DoHBrw-2020 dataset, it demonstrate that LLM models achieve up to 99.9% accuracy, reducing false positives/negatives by over 60% compared to traditional DL methods. Among these, DistilBERT offers a 3–5 \(\times \) reduction in training time with negligible loss of accuracy. The LSTM+LLM hybrid architecture achieves F1-scores above 99% while lowering the computational overhead, making it suitable for deploying in resource-constrained environments. Also the t-SNE visualizations confirms that LLM generated embeddings enhance the class separability and generalization. These results support pretrained transformer embeddings for intrution detection and open doors for further research on encrypted protocols beyond DoH.