Accurately identifying emotions from a speaker’s voice is essential for enhancing user experience in various applications. A significant challenge in speech emotion recognition (SER) is acquiring balanced and diverse training data. This research explores the effectiveness of data augmentation (DA) techniques on both balanced and imbalanced datasets. A ConvLSTM model, which merges convolutional neural networks (CNN) with convolutional long short-term memory (LSTM) networks, effectively captures both spatial and temporal dependencies in speech signals. Using Analysis of Variance (ANOVA), the study identifies the most informative features. The model’s accuracy improved progressively with the addition of each DA technique, achieving optimal results with all four techniques applied to the imbalanced RAVDESS, EMO-DB, and SAVEE datasets, reaching 96.63%, 98.46%, and 96.74%, respectively, and 99.06% for the combined R+E+S+T datasets. In contrast, the balanced TESS dataset achieved 99.67% accuracy without DA, showing only minor improvements when DA was employed, reaching 99.99%.