Malware developers disguise their malicious codes using different sophisticated techniques, making analysis more difficult when malware analysts examine the disassembled binary they have obtained through reverse engineering. Although numerous studies have been published in the literature, most focus on the entire binary to extract features and fail to capture the contextual information of the binary. Thus, it is necessary to develop a model that extracts features from potential regions, which helps extract robust features and capture the binary’s contextual information. This paper proposes a model that identifies the significant APIs that distinguish between benign and malware binaries and extracts byte sequences from the regions with significant API presence. To capture the contextual information of the binary, we use the Bidirectional Encoder Representations from Transformers (BERT) model. These contextual features are passed to different state-of-the-art traditional ML models, such as Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), Support Vector Classifier (SVC), Linear Discriminant Analysis (LDA), and K-Nearest Neighbors (KNN). The proposed model achieves more than 99.93% accuracy with a 0% False Positive Rate (FPR). Further, the proposed model outperforms the state-of-the-art models, which do not focus on potential regions and contextual features.