Malware detection is a critical task in ensuring the security of mobile devices. However, the growing prevalence of obfuscated malware poses challenges to traditional detection methods. In this study, we propose a new approach to detecting obfuscated malware based on Smali bytecode analysis, instruction categorization into families, N-gram vectorization and machine learning techniques. We collected a dataset comprising 8,000 obfuscated APKs from VirusTotal, with half labeled as malicious and half as benign. The dataset includes recent samples from 2024, ensuring the relevance of our results. We extracted the Smali bytecode from each APK, categorized the instructions into families, then transformed them into ordered sequences of n-grams. Our method relies on the use of ensemble learning algorithms to combine the strengths of various models. In particular, we used Random Forest and XGBoost in combination with an n-gram vectorization process of categorized Smali instructions. This approach enabled us to achieve an average F1-score of 0.96 (± 0.01) in cross-validation and a precision of 0.98 for detecting malware and 0.94 for detecting benign software on the test set, with an average recall of 0.96, the overall F1 score also reached 0.96. Our approach, which integrates Smali bytecode analysis, instruction family categorization, and ensemble learning, has demonstrated promising results in detecting obfuscated malware. These findings underscore the importance of a detailed analysis of Smali bytecode and the effectiveness of ensemble learning algorithms in identifying obfuscated malware.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Detecting Obfuscated Android Malware Through Categorized Smali N-Gram Instructions and Ensemble Learning

  • Adam Talbi,
  • Christophe Rodrigues,
  • Nga Nguyen

摘要

Malware detection is a critical task in ensuring the security of mobile devices. However, the growing prevalence of obfuscated malware poses challenges to traditional detection methods. In this study, we propose a new approach to detecting obfuscated malware based on Smali bytecode analysis, instruction categorization into families, N-gram vectorization and machine learning techniques. We collected a dataset comprising 8,000 obfuscated APKs from VirusTotal, with half labeled as malicious and half as benign. The dataset includes recent samples from 2024, ensuring the relevance of our results. We extracted the Smali bytecode from each APK, categorized the instructions into families, then transformed them into ordered sequences of n-grams. Our method relies on the use of ensemble learning algorithms to combine the strengths of various models. In particular, we used Random Forest and XGBoost in combination with an n-gram vectorization process of categorized Smali instructions. This approach enabled us to achieve an average F1-score of 0.96 (± 0.01) in cross-validation and a precision of 0.98 for detecting malware and 0.94 for detecting benign software on the test set, with an average recall of 0.96, the overall F1 score also reached 0.96. Our approach, which integrates Smali bytecode analysis, instruction family categorization, and ensemble learning, has demonstrated promising results in detecting obfuscated malware. These findings underscore the importance of a detailed analysis of Smali bytecode and the effectiveness of ensemble learning algorithms in identifying obfuscated malware.