This research evaluates the effectiveness of various LLM prompt engineering techniques in analyzing acceptance criteria in Gherkin for forming functional requirements in Rimay. Effective application of these techniques would assist the human-expert process of analyzing acceptance criteria to develop requirements. The prompt techniques we evaluate are few-shot learning, chain-of-thought and role-play, with each having either a low or high LLM temperature. Precision-recall metrics provide evaluations for each. The structured language Gherkin defines user stories in Agile software development for defining acceptance criteria. Rimay defines functional requirements in a standardized controlled natural language. The results show that the prompt technique few-shot learning with low LLM temperature gave the best results for translating from Gherkin to Rimay. The chain-of-thought technique combined with a high LLM temperature gave reasonable results. Role-play gave the least accurate results.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Prompt Engineering for Analyzing Acceptance Criteria for Functional Requirements

  • Lloyd Rutledge,
  • Koen van der Kruk

摘要

This research evaluates the effectiveness of various LLM prompt engineering techniques in analyzing acceptance criteria in Gherkin for forming functional requirements in Rimay. Effective application of these techniques would assist the human-expert process of analyzing acceptance criteria to develop requirements. The prompt techniques we evaluate are few-shot learning, chain-of-thought and role-play, with each having either a low or high LLM temperature. Precision-recall metrics provide evaluations for each. The structured language Gherkin defines user stories in Agile software development for defining acceptance criteria. Rimay defines functional requirements in a standardized controlled natural language. The results show that the prompt technique few-shot learning with low LLM temperature gave the best results for translating from Gherkin to Rimay. The chain-of-thought technique combined with a high LLM temperature gave reasonable results. Role-play gave the least accurate results.