Speech recognition has become ubiquitous in modern AI applications. It converts spoken words into text using learning algorithms to make human-machine interaction more convenient. Conventional methods that use mel-frequency cepstral coefficients (MFCC) features for speech recognition often lack efficiency in the presence of environmental noise. Advanced deep learning algorithms have the potential to address this challenge. This article presents an empirical study for a holistic comparison among EfficientNet-based speech recognition models. EfficientNet architectures deal with the trade-off between performance and computational efficiency through compound scaling, mobile inverted bottleneck convolution, and squeeze and excitation blocks. The presented work trains and tests EfficientNetB0-B7 with mel-spectrogram images extracted from the benchmark AudioMNIST dataset. The original acoustic signals in the dataset are superimposed with babble and Gaussian white noise to produce noisy data of varying intensities. The results show that all EfficientNet architectures have outperformed traditional methods, achieving above 90% accuracy even in noisy environments without the need for any noise removal method.