Machine learning (ML) is commonly used in cybersecurity to detect threats in binary applications. These techniques can identify patterns in large datasets where traditional logic and outdated fingerprints may fall short. Since skilled hackers can bypass static analysis through various means, security products typically employ a multilayered approach. The final layers often include sandboxing and execution monitoring. In this paper, we present a ML-based method for clustering and classifying binary execution traces obtained through Dynamic Binary Instrumentation (DBI). Our approach enhances sandbox detection by classifying execution traces from new applications. While binary instrumentation may introduce some overhead, adding a few seconds to minutes of processing time and additional analysis artifacts, we explore potential optimizations to mitigate these drawbacks in Sect. 2.4. Our experiments indicate that DBA (Dynamic Time Warping Barycenter Averaging) distance in k-Means finds the initial six clusters (original threat families gathered by common Antivirus labels), while tweaking a deep-learning Convolutional Neuronal Network (CNN) with support for Long Short-Term Memory (LSTM) model, we obtained a 99.83% test accuracy.