A decentralized, self-organized collection of mobile devices that construct networks is known as a Mobile Ad-hoc Network (MANET). The mobile ad hoc network provides flexible communication in a dynamic environment. However, their open and topology-less nature makes them highly susceptible to malicious attacks, imposing significant security challenges. To overcome malicious attack detection, numerous traditional security systems are designed, which often struggle with the adaptive and evolving nature of cyber threats in mobile ad hoc networks. This article proposes a machine learning-based decision tree malicious detection system (DML-DT) to enhance security in MANETs. The system utilizes a decision tree classifier to analyze network traffic patterns and identify malicious activities with high accuracy. By implementing a supervised learning technique, the proposed model effectively differentiates between normal and malicious network behavior, reducing false positives and improving detection efficiency. Decision tree capable for adapting to large network with the ability to handle large datasets in real-time networks and filtered out malicious data from the network using a node-blocking mechanism. In the result section, evaluations demonstrate that the proposed DML-DT system achieves a high detection ratio, accurate classification of malicious and normal nodes, and efficient performance in terms of the percentage of data received, throughput, and delay. The comparative analysis takes place with ad hoc on-demand distance vector malicious detection (AODV-M), dynamic source routing malicious detection (DSR-M) routing and concludes that the proposed machine learning decision tree (DML-DT) approach is strengthening the MANET security against malicious threats. This research contributes to the enhancement of intelligent, adaptive, and automated malicious detection mechanisms that ensure the reliability and safety of mobile ad hoc networks.