The significant challenge of the imbalanced datasets is the impact on the model’s ability to learn from and correctly classify the minority class. To improve machine learning classification models for identifying unbalanced financial fraudulent transactions, this study explores and compares five data balancing techniques, including under-sampling, oversampling, and hybrid methods such as Random Under Sampler, SMOTE, SMOTE Tomek, SMOTEENN, and ADASYN. These techniques are evaluated for their impact on three machine learning models: Logistic Regression, Complement Naïve Bayes, and Decision Tree Classifier. As a result, applying these algorithms has improved the accuracy of the models, which necessitates applying optimization through parameter tuning and ensemble methods, where two approaches have been utilised: AdaBoost and Random Forest, which in turn have positively impacted the measured outcome. This process has led to improving and identifying the best fit technique resulting from the hybrid for enhancing accuracy and addressing regularization. The findings offer valuable insights into the most effective balancing techniques for improving machine learning modelling. SMOTEENN, combined with Random Forest and parameter tuning for Decision Tree and Logistic Regression, outperformed all other methods, achieving a balanced trade-off accuracy of 82% compared to undersampling with only 64%. This highlights the significant potential of the hybrid balancing framework in improving fraud detection performance for unbalanced datasets.