Aphasia is one of the brain disorders that affects the part of the brain that is responsible for communication. This work deals with the detection of aphasia from speech features. The features extracted from the speech samples are MFCC, mel spectrogram, zero crossing rate, chroma, energy, spectral centroid, spectral roll-off, fundamental frequency and utterance duration. These different speech features were then ranked based on the SHapley Additive exPlanations (SHAPs), and the most important features were retained in an optimized feature set that was given to various machine learning models like support vector machine and decision trees and deep learning models like LSTM, LSTM with GRU, Bi-LSTM and Bi-LSTM with GRU. These models were then evaluated based on their train and test set accuracy, F1 score, recall, precision, plots of training loss and validation loss and confusion matrix. Based on the analysis of the performance metrics, Bi-LSTM with GRU was found to be the best model among all the models used. The model’s performance was evaluated using metrics like accuracy, F1 score, recall and precision when trained and tested with the optimal feature set which were 87.5%, 79.7%, 90% and 75%, respectively.