Malware detection is one of the most challenging tasks in the domain of Cybersecurity. The use of machine learning models as detectors significantly improved the performance. At the same time, adversarial modeling of malware can introduce a small noise into the malware data to generate adversarial malware, which can evade the machine learning based detector. In this work, we have proposed a detection framework for adversarial malware. We have considered SLIPNER and MalGAN datasets to generate adversarial malware using an adversarial-based attack algorithm. Our findings show that our approach is extremely effective in detecting and eliminating adversarial malware.