Speech recognition refers to the process of transforming human speech into text or other machine-readable formats. This paper presents a comprehensive exploration of Bangla Numeral Recognition from Speech Signals. This work features a combination of authenticity and augmentation that significantly reduces the dependency on raw data, especially for a low-resource language like Bangla. This study uses Transfer Learning and Convolutional Neural Networks (CNNs) to create a Bangla number identification system based on voice signals. The dataset includes 19,906 individual audio files for (0–199) Bangla numerals where numerals (0–99) are taken from a pre-existing dataset, while numbers (101–199) are augmented. The methods are designed around the extraction and analysis of log Mel spectrograms, capitalizing on their ability to represent complex audio structures in a manner that is both comprehensive and conducive to CNN architectures including DenseNet, ResNet, and VGG. Among our experiments, ResNet performed most significantly with an accuracy of 95.58% recognizing (0–199) Bangla spoken numbers across the entire dataset.