This research work aims to develop models and algorithms for detecting suspicious transactions for payment systems based on artificial intelligence. In this work, algorithms for detecting suspicious transactions in payment systems based on machine learning and their analysis are described in detail. Today, the proliferation of payment technologies has heightened the risk of suspicious transactions inflicting substantial harm on financial institutions; hence, the identification of such transactions is very important. The effectiveness of several machine learning methods in identifying questionable data was assessed. This article is crucial for guaranteeing security and dependability in sectors such as payment systems, banking, and healthcare.