Persian Aut emotional speech database: machine learning-based evaluation and cost-efficient labeling approach
摘要
Speech emotion recognition (SER) plays a pivotal role in different areas, such as the evolution of automated customer service systems, enabling a more sophisticated understanding of user sentiments. In the context of the Persian language as a low-resource language, the absence of natural emotional speech datasets poses a challenge for developing effective systems. This paper addresses this gap by engaging 28 non-professional university students in creating a semi-natural dataset as a basic representative of real-world interactions. This dataset called AutESD having four emotions of neutrality, happiness, anger, and sadness could also be adapted to be used for on-line academic evaluations in future since we have gathered it from both male and female students in the age range of 23 to 30. The main advantages of AutESD are its semi-natural characteristic, larger number of speakers compared to previous Persian datasets, and its low cost in the collection procedure. Moreover, we propose and develop a labeling software to streamline the labeling process which uses 16 persons including both speech experts and non-experts to label the emotion classes remotely. The inter-rater reliability analysis yielded Fleiss’ Kappa values between 0.70 and 1.00, indicating substantial to almost perfect agreement among annotators. It establishes a framework to optimize both labeling expenses and time as well as accuracy. Subsequently, to show the efficiency of AutESD in SER task, we evaluate it utilizing four machine learning models of K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Deep Neural Network (DNN)-based model which uses a Convolutional Neural Network (CNN) architecture, and a transformer. The results show that AutESD achieved Unweighted-Average-Recall (UAR) of up to 79% using CNN by MFCC features on the test dataset using 5-fold cross validation in a speaker independent scenario.