Scholarlink

<p>Artificial Intelligence (AI) tools are increasingly applied to support systematic literature reviews (SLRs), but researchers lack structured guidance for evaluating and integrating these tools into their workflows. This study proposes and applies a rubric-based framework for assessing AI toolchains across core SLR tasks, including search, screening, and synthesis. A mixed-methods evaluation was conducted with 63 postgraduate student researchers, representing early-stage users who are primary adopters of such workflows. Findings indicate that rubric-guided workflows improved participant-perceived usefulness of AI tools, supported structured criteria for comparison, and facilitated greater transparency in tool adoption. While formal inter-rater reliability statistics (e.g., Cohen’s or Fleiss’ Kappa) could not be computed because participants did not apply the rubric in parallel due to the absence of a common benchmark set, rubric consistency was qualitatively validated through pilot testing, iterative refinement, and structured participant discussions. This highlights interpretive alignment while acknowledging the need for future replication with independent evaluators. The study contributes a validated evaluation rubric, an illustrative workflow for AI-assisted SLRs, and reflections on implications for research training, reproducibility, and cross-domain adaptation.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating AI toolchains for systematic literature reviews using a rubric-based workflow approach

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年12月29日

查看全文
  • Nisha Biju,
  • Alisha Sinha,
  • Sarthak Saini,
  • Ganesh Beniwal,
  • Syaamantak Das

摘要

Artificial Intelligence (AI) tools are increasingly applied to support systematic literature reviews (SLRs), but researchers lack structured guidance for evaluating and integrating these tools into their workflows. This study proposes and applies a rubric-based framework for assessing AI toolchains across core SLR tasks, including search, screening, and synthesis. A mixed-methods evaluation was conducted with 63 postgraduate student researchers, representing early-stage users who are primary adopters of such workflows. Findings indicate that rubric-guided workflows improved participant-perceived usefulness of AI tools, supported structured criteria for comparison, and facilitated greater transparency in tool adoption. While formal inter-rater reliability statistics (e.g., Cohen’s or Fleiss’ Kappa) could not be computed because participants did not apply the rubric in parallel due to the absence of a common benchmark set, rubric consistency was qualitatively validated through pilot testing, iterative refinement, and structured participant discussions. This highlights interpretive alignment while acknowledging the need for future replication with independent evaluators. The study contributes a validated evaluation rubric, an illustrative workflow for AI-assisted SLRs, and reflections on implications for research training, reproducibility, and cross-domain adaptation.