Scholarlink

The rapid development of large language models (LLMs) has unlocked new opportunities for tackling complex multi-label classification tasks across diverse domains. Open-source LLMs, in particular, offer advantages in terms of data privacy, customization, and flexibility. A representative benchmark for evaluating multi-label classification performance is the Sustainable Development Goal (SDG) mapping task. In this study, we systematically evaluate the multi-label classification capabilities of several LLMs, including Mixtral, LLaMA 2, LLaMA 3, Gemma, Qwen2, and GPT-4o-mini, using GPT-4o as a baseline. To provide a comprehensive assessment, we measure model performance using micro-averaged F1 score, precision, and recall, derived from the confusion matrix. Additionally, performance curves are plotted at varying decision thresholds to analyze model behavior under different conditions. Our results demonstrate that while models like LLaMA 2 and Gemma show promise for improvement, other models exhibit competitive performance in multi-label classification tasks. All model outputs are available on Zenodo ( https://doi.org/10.5281/zenodo.12789375 ) to facilitate further research.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating the Performance of Large Language Models on a Multi-label Classification Task

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Hui Yin,
  • Amir Aryani,
  • Nakul Nambiar,
  • Qi Zhong

摘要

The rapid development of large language models (LLMs) has unlocked new opportunities for tackling complex multi-label classification tasks across diverse domains. Open-source LLMs, in particular, offer advantages in terms of data privacy, customization, and flexibility. A representative benchmark for evaluating multi-label classification performance is the Sustainable Development Goal (SDG) mapping task. In this study, we systematically evaluate the multi-label classification capabilities of several LLMs, including Mixtral, LLaMA 2, LLaMA 3, Gemma, Qwen2, and GPT-4o-mini, using GPT-4o as a baseline. To provide a comprehensive assessment, we measure model performance using micro-averaged F1 score, precision, and recall, derived from the confusion matrix. Additionally, performance curves are plotted at varying decision thresholds to analyze model behavior under different conditions. Our results demonstrate that while models like LLaMA 2 and Gemma show promise for improvement, other models exhibit competitive performance in multi-label classification tasks. All model outputs are available on Zenodo ( https://doi.org/10.5281/zenodo.12789375 ) to facilitate further research.