In modern applications, identifying unusual patterns is crucial to maintaining security and stability. Unusual activity can be a sign of malware, which can lead to breaches or complete system failure. In this study, we explore the effectiveness of various machine learning techniques that specialize in detecting these novel patterns. Specifically, we evaluate methods including Isolation Forest, One-Class SVM, Local Outlier Factor, and Autoencoder. Through a detailed comparative analysis, we have found that the Autoencoder model stands out by identifying anomalies more effectively than the other approaches. Our findings not only highlight the potential of these algorithms in identifying malicious behaviour in software but also shed light on their respective performance metrics in terms of efficiency, accuracy, and overall reliability. This research underscores the importance of advanced detection techniques in the ongoing fight against malware, with the aim of improving the security of digital applications. Furthermore, the study offers guidance in cybersecurity research.