Infant cry recognition systems are used to identify an infant’s communication and emotional transmission from cry signals. However, the high degree of variability in infant cry signals, which are auditorily similar and different in their physiological genesis, makes this a challenging task. This paper presents an efficient deep learning framework for handling the infant cry recognition. The Baby Chillanto (BC) and Donate a Cry Corpus (DCC) datasets were utilized to gather infant cry signals, and each dataset has five different kinds of cry signals. Spectrogram and scalogram images obtained from raw cry signals are converted into deep feature vectors using five pre-trained convolutional neural network (CNN) models, namely DenseNet201, ResNet101, VGG19, InceptionV3, and Xception. The ReliefF algorithm is used to select descriptive features from high-dimensional deep features. The selected deep feature vectors are transferred to the ensemble learning model consisting of extreme learning machine (ELM), support vector machine (SVM), and random forest (RF) algorithms to classify infant cry types. Experimental results and findings show that the proposed spectrogram image-based InceptionV3 deep features and ensemble learning model achieve 99.95% and 99.69% classification accuracy for BC and DCC datasets, respectively. The results show that the proposed infant cry recognition framework outperforms existing methods and can be used as a tool that can help parents understand their infant’s needs and emotions.