As recommender systems play an increasingly central role in shaping information exposure on platforms like YouTube, understanding the nature of the content they promote, especially in sensitive contexts, requires scalable and reliable labelling methods. This paper investigates the use of Large Language Models (LLM) to label YouTube videos based solely on their metadata. We propose a committee-based approach that aggregates predictions from an ensemble of seven state-of-the-art LLMs through majority voting. Using a novel dataset collected via simulated user interactions on YouTube, we analyse model agreement, labelling behavior, and the influence of model size. To assess label reliability, we also investigate the semantic coherence of label assignments. Our results show that LLM committees produce highly consistent labels in low-disagreement settings. These findings highlight both the promise and limitations of LLM-based annotation for auditing social networks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Can LLMs Reliably Label YouTube Videos? A Committee-Based Evaluation

  • Adriano Mourthé,
  • Carlos Eduardo Mello,
  • Alípio Jorge

摘要

As recommender systems play an increasingly central role in shaping information exposure on platforms like YouTube, understanding the nature of the content they promote, especially in sensitive contexts, requires scalable and reliable labelling methods. This paper investigates the use of Large Language Models (LLM) to label YouTube videos based solely on their metadata. We propose a committee-based approach that aggregates predictions from an ensemble of seven state-of-the-art LLMs through majority voting. Using a novel dataset collected via simulated user interactions on YouTube, we analyse model agreement, labelling behavior, and the influence of model size. To assess label reliability, we also investigate the semantic coherence of label assignments. Our results show that LLM committees produce highly consistent labels in low-disagreement settings. These findings highlight both the promise and limitations of LLM-based annotation for auditing social networks.