This study evaluates the performance of various large language models (LLMs) combined with six Active Learning (AL) query strategies–including a random query strategy as a baseline, three diversity-based query methods, and two uncertainty-based query approaches–for the task of Named Entity Recognition and Classification (NERC). The evaluation spans seven datasets that vary in size, uncertainty, and perplexity. Our findings indicate that AL algorithms perform poorly on datasets with high uncertainty or perplexity, while showing promising results on more simple datasets. Additionally, we show that the effectiveness of each AL strategy is highly dependent on both the specific dataset and the LLM used, revealing considerable variability and limiting the reliability of these methods in practical applications. These results highlight the urgent need for more robust AL query strategies that can better capture relevant features in complex, high-uncertainty data–thereby improving the consistency and generalizability of AL systems across different domains.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Perplexity, Uncertainty, and the Limits of Active Learning

  • Pablo Turón,
  • Montse Cuadros

摘要

This study evaluates the performance of various large language models (LLMs) combined with six Active Learning (AL) query strategies–including a random query strategy as a baseline, three diversity-based query methods, and two uncertainty-based query approaches–for the task of Named Entity Recognition and Classification (NERC). The evaluation spans seven datasets that vary in size, uncertainty, and perplexity. Our findings indicate that AL algorithms perform poorly on datasets with high uncertainty or perplexity, while showing promising results on more simple datasets. Additionally, we show that the effectiveness of each AL strategy is highly dependent on both the specific dataset and the LLM used, revealing considerable variability and limiting the reliability of these methods in practical applications. These results highlight the urgent need for more robust AL query strategies that can better capture relevant features in complex, high-uncertainty data–thereby improving the consistency and generalizability of AL systems across different domains.