Social Media Bots (SMBs) pose significant challenges to Online Social Networks’ security and integrity by manipulating online discourse and spreading misinformation and spam content. However, where real users significantly outnumber bots, imbalanced datasets often hinder the performance of Machine Learning (ML) and Deep Learning (DL) models used for SMB detection. This paper proposes a novel approach that goes beyond focusing solely on accuracy to optimize social media bot detection. This approach incorporates these key elements to enhance the model’s effectiveness: 1) extracting a large range of features from a recent dataset of SMBs; 2) Selecting the effective features for SMB detection using Recursive Feature Addition (RFA), which is a hybrid feature selection technique; 3) Investigating the effectiveness of different techniques to address the imbalanced nature of social media bot detection datasets. We explored under-sampling, over-sampling, and combined approaches to ensure the model is trained on a more balanced representation of real and bot-generated content; 4) Train the models using DL and ML algorithms and stratified 5-fold cross-validation. Our results demonstrate that using hybrid feature selection and combined data balancing leads to significant improvements in performance metrics beyond just accuracy. We achieved superior performances compared to the baseline model scores exceeding 99% for accuracy, recall, precision, F1-score, and AUC-ROC in both training and test stages.