Scholarlink

Context <p>Automated requirements classification is crucial for software quality assurance, yet it remains a challenging task, particularly for multi-label classification (MLC) where requirements can span multiple quality attributes. While traditional machine learning (ML) and deep learning (DL) methods have been widely studied, the true potential of modern Large Language Models (LLMs) in this complex domain remains largely unexplored.</p> Objective <p>This paper presents the first comprehensive empirical study to systematically evaluate generative LLMs for multi-label requirements classification. We benchmark them against reproduced state-of-the-art traditional methods and dissect the impact of three core drivers of LLM performance: model architecture, Parameter-Efficient Fine-Tuning (PEFT), and prompting strategies.</p> Method <p>To ensure robust external validity, our evaluation leverages two distinct datasets: the widely-used, imbalanced <b>EMSE benchmark</b> and a newly constructed, balanced <b>App Reviews Balanced Dataset (ARBD)</b>. We assess performance across various scenarios, including zero-shot, few-shot, and fine-tuned settings, using a suite of standard classification metrics.</p> Results <p>Our findings reveal critical, context-dependent insights. First, we establish that <b>data distribution dictates the optimal paradigm</b>: while Deep Learning (BERT) and Classical ML (SVM) define the “accuracy ceiling” in imbalanced and balanced supervised settings respectively, LLMs offer decisive advantages in high-recall scenarios (critical for requirements discovery) and are the only viable solution in low-data environments. Second, we demonstrate that architectural suitability trumps raw scale, as a moderately-sized open-source LLM (Qwen3-14B) consistently outperforms massive frontier models in F1-score. Finally, we show that PEFT (LoRA) serves a dual role dictated by data balance—acting as a precision-recall trade-off lever in imbalanced scenarios, but as a robust performance amplifier in balanced ones.</p> Conclusion <p>LLM-based approaches represent a powerful and versatile new paradigm. Our findings provide a foundational guide and a clear decision framework for practitioners, highlighting how data characteristics (balance) and project goals (accuracy vs. coverage) determine the choice between Traditional Models and LLMs. This study paves the way for a more effective and context-aware application of LLMs in requirements engineering.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Machine learning, deep learning, or large language models: An empirical study on multi-label requirements classification

  • 期刊论文

  • 发表日期：

    2026年5月22日

查看全文
  • Wenhao Wang,
  • Jiaxi Peng,
  • Hongbin Xiao,
  • Yang Hua,
  • Yufei Zhou,
  • Zhi Li,
  • Xiaoli Wang

摘要

Context

Automated requirements classification is crucial for software quality assurance, yet it remains a challenging task, particularly for multi-label classification (MLC) where requirements can span multiple quality attributes. While traditional machine learning (ML) and deep learning (DL) methods have been widely studied, the true potential of modern Large Language Models (LLMs) in this complex domain remains largely unexplored.

Objective

This paper presents the first comprehensive empirical study to systematically evaluate generative LLMs for multi-label requirements classification. We benchmark them against reproduced state-of-the-art traditional methods and dissect the impact of three core drivers of LLM performance: model architecture, Parameter-Efficient Fine-Tuning (PEFT), and prompting strategies.

Method

To ensure robust external validity, our evaluation leverages two distinct datasets: the widely-used, imbalanced EMSE benchmark and a newly constructed, balanced App Reviews Balanced Dataset (ARBD). We assess performance across various scenarios, including zero-shot, few-shot, and fine-tuned settings, using a suite of standard classification metrics.

Results

Our findings reveal critical, context-dependent insights. First, we establish that data distribution dictates the optimal paradigm: while Deep Learning (BERT) and Classical ML (SVM) define the “accuracy ceiling” in imbalanced and balanced supervised settings respectively, LLMs offer decisive advantages in high-recall scenarios (critical for requirements discovery) and are the only viable solution in low-data environments. Second, we demonstrate that architectural suitability trumps raw scale, as a moderately-sized open-source LLM (Qwen3-14B) consistently outperforms massive frontier models in F1-score. Finally, we show that PEFT (LoRA) serves a dual role dictated by data balance—acting as a precision-recall trade-off lever in imbalanced scenarios, but as a robust performance amplifier in balanced ones.

Conclusion

LLM-based approaches represent a powerful and versatile new paradigm. Our findings provide a foundational guide and a clear decision framework for practitioners, highlighting how data characteristics (balance) and project goals (accuracy vs. coverage) determine the choice between Traditional Models and LLMs. This study paves the way for a more effective and context-aware application of LLMs in requirements engineering.