Comparison of Cognitive-Based and Classic Optimization-Based Annotator Selection Approaches
摘要
Active Learning (AL) is an effective Machine Learning (ML) strategy that improves training efficiency by selecting the most informative examples from unlabeled data. It is especially valuable in scenarios where data labeling is costly or time-consuming, as it reduces the amount of labeled data required to achieve strong model performance. However, traditional AL is based on unrealistic premises, namely, on an oracle that is always present and correct. Although new approaches have been proposed to address these inaccuracies, most still overlook internal factors that affect the productivity of human annotators. This paper builds on a new approach that enhances AL by considering the productivity of human annotators. It introduces a Recommendation System (RS) that, for each instance, recommends the most appropriate annotator at that specific time using information about their mood, fatigue level, and historical accuracy, as well as information about the instance queried. That study demonstrated that it is beneficial to consider the mood and fatigue levels of annotators, in addition to historical accuracy, since it achieves marginally higher accuracy (compared to relying solely on past performance) and decreases misclassifications and uncertainty, lowering overall training costs. In this study, we compare this research to an optimization-based strategy using the same simulated annotators and show that the RS approach used for query-annotator pair selection (considering annotators’ past accuracy, mood, and fatigue levels) demonstrates no significant difference in performance compared to the best pairs possible with the given annotators.