[Context and Motivation] Online user feedback provides valuable information to support requirements engineering (RE). However, analysing online user feedback is challenging due to its large volume and noise. Large language models (LLMs) show strong potential to automate this process and outperform previous techniques. They can also enable new tasks, such as generating requirements specification. However, existing work largely focuses on large proprietary models. Consequently, lightweight open-source LLMs remain underexplored for feedback-driven RE. [Question/Problem] In particular, existing studies offer limited empirical evidence, lack thorough evaluation, and rarely provide replication packages, undermining validity and reproducibility. [Principal Idea/Results] We evaluate five lightweight open-source LLMs on three RE tasks: NFR classification, user request classification, and requirements specification generation. Classification performance was measured on two feedback datasets, and specification quality via human evaluation. LLMs achieved moderate-to-high classification accuracy (F1 \(\approx \) 0.47–0.68) and moderately high specification quality (mean \(\approx \) 3/5). [Contributions] We newly explore lightweight open-source LLMs for feedback-driven requirements development. Our contributions are: (i) an empirical evaluation of lightweight LLMs on three RE tasks, (ii) a replication package, and (iii) insights into their capabilities and limitations for RE.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

From Online User Feedback to Requirements: Evaluating Large Language Models for Classification and Specification Tasks

  • Manjeshwar Aniruddh Mallya,
  • Alessio Ferrari,
  • Mohammad Amin Zadenoori,
  • Jacek Dąbrowski

摘要

[Context and Motivation] Online user feedback provides valuable information to support requirements engineering (RE). However, analysing online user feedback is challenging due to its large volume and noise. Large language models (LLMs) show strong potential to automate this process and outperform previous techniques. They can also enable new tasks, such as generating requirements specification. However, existing work largely focuses on large proprietary models. Consequently, lightweight open-source LLMs remain underexplored for feedback-driven RE. [Question/Problem] In particular, existing studies offer limited empirical evidence, lack thorough evaluation, and rarely provide replication packages, undermining validity and reproducibility. [Principal Idea/Results] We evaluate five lightweight open-source LLMs on three RE tasks: NFR classification, user request classification, and requirements specification generation. Classification performance was measured on two feedback datasets, and specification quality via human evaluation. LLMs achieved moderate-to-high classification accuracy (F1 \(\approx \) 0.47–0.68) and moderately high specification quality (mean \(\approx \) 3/5). [Contributions] We newly explore lightweight open-source LLMs for feedback-driven requirements development. Our contributions are: (i) an empirical evaluation of lightweight LLMs on three RE tasks, (ii) a replication package, and (iii) insights into their capabilities and limitations for RE.