Task-Specific Knowledge Distillation for Scalable Sentiment Classification in Low-Resource Settings
摘要
Knowledge distillation (KD) is a effective method for creating efficient sentiment analysis models by transferring knowledge from a large, high-performing teacher model to a smaller, faster student model. This study compares three models: the teacher, student, and a distilled model that leverages KD to combine the strengths of both. The teacher model achieved the highest accuracy due to its large capacity to capture complex sentiment cues and contextual nuances but required significant computational resources, limiting its use in real-time or resource-constrained environments. Conversely, the student model was lightweight and fast, making it suitable for deployment on devices with limited resources, but it struggled with accuracy, especially in cases involving indirect or subtle sentiment expressions. The distilled model, trained through KD, effectively bridges this gap by approaching the teacher’s accuracy while maintaining the efficiency of the student model. When compared to traditional machine learning models that rely on hard labels, the distilled model better handled mixed, nuanced, and context-dependent sentiments, demonstrating superior generalization. This makes KD-based models practical for scalable sentiment analysis applications, particularly in mobile and edge computing scenarios. Additionally, this approach can be extended to multi-class sentiment classification tasks and integrated with sentiment reasoning models to enhance emotional understanding. The full implementation and code are publicly available at https://github.com/Abishethvarman/KD-Sentiment-Analysis .