Analyzing sentiment in Open-Source Software (OSS) discussions is crucial, but manual labeling for large datasets is prohibitively expensive. We address this annotation bottleneck with a novel hybrid active learning framework. Our approach uses a fine-tuned DistilBERT model and a pool-based strategy that leverages both prediction entropy and breaking-ties margin to efficiently select the most informative comments for annotation. We also include a rigorous, multi-faceted convergence analysis that integrates uncertainty metrics, loss curves, performance trends, and the OracleAcc-MCS stopping criterion. The final model was evaluated on a separate held-out test set, achieving a high accuracy of 0.96 and demonstrating robust generalization. When applied to the full dataset, the converged model revealed a predominant neutral sentiment (66.4%), followed by positive (22.2%) and negative (11.4%) expressions, all with high prediction confidence. This work significantly reduces labeling effort while maintaining high performance, offering a scalable and efficient solution for large-scale sentiment analysis in OSS communication. The fine-tuned DistilBERT model is publicly available to foster further research.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Boosting Sentiment Analysis in OSS: A Hybrid Active Learning Strategy Using Uncertainty Metrics

  • Mohit Kaushik,
  • Kuljit Kaur Chahal

摘要

Analyzing sentiment in Open-Source Software (OSS) discussions is crucial, but manual labeling for large datasets is prohibitively expensive. We address this annotation bottleneck with a novel hybrid active learning framework. Our approach uses a fine-tuned DistilBERT model and a pool-based strategy that leverages both prediction entropy and breaking-ties margin to efficiently select the most informative comments for annotation. We also include a rigorous, multi-faceted convergence analysis that integrates uncertainty metrics, loss curves, performance trends, and the OracleAcc-MCS stopping criterion. The final model was evaluated on a separate held-out test set, achieving a high accuracy of 0.96 and demonstrating robust generalization. When applied to the full dataset, the converged model revealed a predominant neutral sentiment (66.4%), followed by positive (22.2%) and negative (11.4%) expressions, all with high prediction confidence. This work significantly reduces labeling effort while maintaining high performance, offering a scalable and efficient solution for large-scale sentiment analysis in OSS communication. The fine-tuned DistilBERT model is publicly available to foster further research.