Scholarlink

<p><b>Context:</b> In recent years, advanced Artificial Intelligence (AI) tools, such as Generative Pretrained Transformers (GPTs), have shown remarkable potential across diverse application areas, including academic research. One of the promising applications of using these tools for conducting Systematic Literature Reviews (SLRs), which are effort intensive and time-consuming process in software engineering research. </p><p><b>Objective:</b> This study investigates the use of GPT-4o to support SLR activities in the context of software engineering. </p><p><b>Methods:</b> We employed a mixed research method to explore the significance of GPT-4o in conducting sub-SLR activities, such as formulating research questions, generating Boolean strings, and performing quality assessments. Four benchmark SLR studies were selected and used to replicate the SLR activities using GPT-4o, which were further assessed by six domain experts. </p><p><b>Results:</b> In the results, GPT-4o demonstrated consistency in generating Boolean search strings, identifying themes, and quality assessment criteria across all the benchmark studies, and showed close alignment with expert evaluations. The model effectively automated key SLR activities, including literature search string formulation, validation of primary studies, quality assessment, thematic analysis, and taxonomy development. However, GPT-4o was less accurate in identifying relevant studies across databases, occasionally missing domain-specific literature. </p><p><b>Conclusions:</b> The findings indicate that GPT-4o can effectively support several SLR tasks in software engineering when integrated within a domain expert-in-the-loop workflow. However, the study evaluates selected activities under controlled settings rather than a full end-to-end SLR execution. Future research should extend this evaluation to additional datasets and alternative LLMs to further assess generalizability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An exploratory study on effectiveness of GPT-4o in conducting sub-tasks of systematic literature reviews

  • 期刊论文

  • 发表日期：

    2026年5月13日

查看全文
  • Mohammad Shameem,
  • Mohammad Alshayeb,
  • Mahmood Niazi,
  • Sajjad Mahmood,
  • Aakash Ahmad,
  • Jehad Al Dallal

摘要

Context: In recent years, advanced Artificial Intelligence (AI) tools, such as Generative Pretrained Transformers (GPTs), have shown remarkable potential across diverse application areas, including academic research. One of the promising applications of using these tools for conducting Systematic Literature Reviews (SLRs), which are effort intensive and time-consuming process in software engineering research.

Objective: This study investigates the use of GPT-4o to support SLR activities in the context of software engineering.

Methods: We employed a mixed research method to explore the significance of GPT-4o in conducting sub-SLR activities, such as formulating research questions, generating Boolean strings, and performing quality assessments. Four benchmark SLR studies were selected and used to replicate the SLR activities using GPT-4o, which were further assessed by six domain experts.

Results: In the results, GPT-4o demonstrated consistency in generating Boolean search strings, identifying themes, and quality assessment criteria across all the benchmark studies, and showed close alignment with expert evaluations. The model effectively automated key SLR activities, including literature search string formulation, validation of primary studies, quality assessment, thematic analysis, and taxonomy development. However, GPT-4o was less accurate in identifying relevant studies across databases, occasionally missing domain-specific literature.

Conclusions: The findings indicate that GPT-4o can effectively support several SLR tasks in software engineering when integrated within a domain expert-in-the-loop workflow. However, the study evaluates selected activities under controlled settings rather than a full end-to-end SLR execution. Future research should extend this evaluation to additional datasets and alternative LLMs to further assess generalizability.