<p>Estimating the effort required for software development is challenging due to the inherent complexity of software. The use of Large Language Models (LLMs) could facilitate this process for development teams. This study aims to explore the capability of LLMs to estimate the effort needed to complete requirements expressed as user stories in software development. The estimates of 10 LLMs and 42 developers were evaluated using the agile T-shirt Sizing technique for 12 user stories. The preservation of the expected logical order, less effort with greater experience and optimism were analyzed using statistical tests. Subsequently, absolute accuracy was compared with the actual effort obtained from post-mortem developments of a small e-commerce business. Human developers and several LLMs showed similar accuracy in their estimates compared to the actual post-mortem effort. Large models, those with hundreds of billions of parameters, consistently preserved the expected order in estimates, reducing estimated effort by increasing experience and optimism. The experience level showed a greater influence than the degree of optimism. Some smaller models, commonly referred to as Small Language Models (SLMs) ranging from four and eight billion parameters, showed inconsistent patterns, so model size was more decisive than reasoning capability for the quality of estimates. These findings suggest that the effort estimates from some LLMs are more accurate than those from human developers when the developer’s experience level and degree of optimism are properly incorporated.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Large language models for agile effort estimation: a post-mortem study incorporating developer experience and optimism

  • Carlos Villarrubia,
  • Enrique Adrian Villarrubia-Martin,
  • Juan Manuel Vara,
  • David Granada

摘要

Estimating the effort required for software development is challenging due to the inherent complexity of software. The use of Large Language Models (LLMs) could facilitate this process for development teams. This study aims to explore the capability of LLMs to estimate the effort needed to complete requirements expressed as user stories in software development. The estimates of 10 LLMs and 42 developers were evaluated using the agile T-shirt Sizing technique for 12 user stories. The preservation of the expected logical order, less effort with greater experience and optimism were analyzed using statistical tests. Subsequently, absolute accuracy was compared with the actual effort obtained from post-mortem developments of a small e-commerce business. Human developers and several LLMs showed similar accuracy in their estimates compared to the actual post-mortem effort. Large models, those with hundreds of billions of parameters, consistently preserved the expected order in estimates, reducing estimated effort by increasing experience and optimism. The experience level showed a greater influence than the degree of optimism. Some smaller models, commonly referred to as Small Language Models (SLMs) ranging from four and eight billion parameters, showed inconsistent patterns, so model size was more decisive than reasoning capability for the quality of estimates. These findings suggest that the effort estimates from some LLMs are more accurate than those from human developers when the developer’s experience level and degree of optimism are properly incorporated.