Behavioral malware detection in cloud environment using system calls
摘要
Malware detection on cloud platforms encounters serious challenges as attackers employ more obfuscation methods such as polymorphism and fileless infections to avoid discovery. Current methods based on n-gram frequency and Term Frequency-Inverse Document Frequency (TF-IDF) analysis of system calls lack effectiveness in spotting contemporary threats; they focus more on repeated patterns but do not consider the inherent randomness of obfuscated malware, such as non-systematic sequences of system calls provoked by encrypted payloads or code injections. Overlapping patterns between benign and malicious behavior also accelerate the False Positive Rate (FPR). To overcome these limitations, we introduce novel entropy-driven method for anomaly detection. In our methodology, first, we model program behavior by examining the Shannon entropy of its system calls n-grams. Second, we isolate the most critical indicators of malicious activity using a powerful, Extreme Gradient Boosting (XGB) coupled feature selection technique. This core methodology was validated on the VMM Malware and ADFA-LD datasets. Rather than proposing a new classifier, we demonstrate the fundamental strength of our feature engineering by showing its superior performance across a wide range of standard machine learning algorithms including Logistic Regression (LR), Random Forest (RF), Naïve Bayes (NB), K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and XGB. k-fold cross-validation was employed to rigorously train and validate each model, ensuring robust and generalizable performance. The proposed approach outperforms existing state-of-the-art methods by significantly reducing false alarms and missed detections. This work offers a solution for identifying unknown malware and intrusions in cloud environments.