Nonverbal vocal sounds, frequently used in human oral communication, are very helpful in understanding the speaker’s emotional state. In recent years, there have been notable advances in emotion recognition in audio, driven by developments based on transformer architecture. However, emotion recognition in non-linguistic sounds remains largely unexplored. In this work, we explore the use of the Whisper encoder with frozen weights–thereby preserving its transcription capabilities–as a feature extractor for emotion classification in vocal bursts (VB). Different lightweight classifiers were tested using features extracted from multiple depths of the Whisper encoder on the publicly available EmoGator dataset. We employ speaker-independent 5-fold cross-validation to determine the best configuration for each architecture. The proposed approach substantially outperformed the state-of-the-art with an accuracy of 66.5% on 10 classes. Furthermore, we found that the point within the encoder from which features are extracted significantly influences the accuracy of the classifiers. These results demonstrate the feasibility of performing emotion recognition in VB using pre-trained transformers without altering the original model weights, suggesting that such models can be effectively integrated into human-computer interaction systems while preserving their original capabilities.