As modern IT systems grow increasingly complex, User Interface (UI) testing has become a critical component of quality assurance. Crafting UI test scripts from scratch can be both time-consuming and error-prone. In this paper, we explore the use of large language models (LLMs) for automated test generation. We adopt the semi-formal Gherkin language as an intermediary specification to address the gap between test requirements and executable test scripts. Thus, we propose and evaluate an approach that leverages an LLM to transform Gherkin specifications into Selenium scripts. We freeze the version of the system under test (SUT) via Docker containers and measure statement-level test coverage (C0) and branch-level test coverage (C1) using JaCoCo, in order to assess the feasibility and baseline performance of generating executable Selenium scripts from Gherkin with an LLM. Our experimental results show that LLM-generated test scripts, after an average of 13 min of manual refinement, can achieve around 16% C0 coverage, compared to 18% from a reference suite developed by a senior-level test engineer. While a coverage gap remains, the automatically generated scripts demonstrate good correctness and structural quality once minor issues such as selector inaccuracies and assertion insufficiencies are corrected. These findings suggest that, even in an out-of-the-box configuration, LLM-based UI test generation offers a promising starting point for reducing repetitive test engineering tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Baseline Evaluation of LLM-Facilitated UI Test-Case Generation from Gherkin Specifications

  • Alexander Poth,
  • Olsi Rrjolli,
  • Huiyu Wang,
  • Klaus Schmid

摘要

As modern IT systems grow increasingly complex, User Interface (UI) testing has become a critical component of quality assurance. Crafting UI test scripts from scratch can be both time-consuming and error-prone. In this paper, we explore the use of large language models (LLMs) for automated test generation. We adopt the semi-formal Gherkin language as an intermediary specification to address the gap between test requirements and executable test scripts. Thus, we propose and evaluate an approach that leverages an LLM to transform Gherkin specifications into Selenium scripts. We freeze the version of the system under test (SUT) via Docker containers and measure statement-level test coverage (C0) and branch-level test coverage (C1) using JaCoCo, in order to assess the feasibility and baseline performance of generating executable Selenium scripts from Gherkin with an LLM. Our experimental results show that LLM-generated test scripts, after an average of 13 min of manual refinement, can achieve around 16% C0 coverage, compared to 18% from a reference suite developed by a senior-level test engineer. While a coverage gap remains, the automatically generated scripts demonstrate good correctness and structural quality once minor issues such as selector inaccuracies and assertion insufficiencies are corrected. These findings suggest that, even in an out-of-the-box configuration, LLM-based UI test generation offers a promising starting point for reducing repetitive test engineering tasks.