In today’s rapidly evolving digital landscape, robust cybersecurity measures are imperative to protecting digital assets and data integrity against constantly emerging threats, particularly malware. The primary objective of the proposed work is to establish a proactive malware detection system utilizing custom YARA rules tailored to identify distinct malware characteristics and patterns. The key contribution involves enhancing the system’s ability to detect specific types of spyware, particularly custom PDF spyware, by implementing specialized rules and patterns. The yaraGenerator, yarGen, and yabin tools are used to check validation of proposed YARA rules. The yarGen tool yielded comparatively better detection results. After adding the import hashing technique, the generated YARA rules using the three chosen tools, yarGen, yaraGenerator, and yabin, are improved, and their efficacy on Advanced Persistent Threats, Ransomware, Malware Strings, Miner Strings, and Stealer is reassessed. This detection mechanism is prompting the development of unique methods for scrutinizing PDF files for malicious elements such as concealed scripts, deceptive content, and documented vulnerabilities.