Maturity models are widely used as instruments for assessing the current state of transformations within organizations. However, traditional assessment approaches are often resource-intensive, subjective, and difficult to scale across domains. Recent advances in large language models (LLMs) and structured reasoning techniques, such as Chain of Thought (CoT) prompting, offer new possibilities for automating and augmenting such assessments. Despite this potential, systematic approaches that leverage LLMs for maturity level determination remain underexplored. In this paper, we propose a concept for LLM-based maturity assessment grounded in a custom-developed maturity model. We design and evaluate a two-stage proof-of-concept pipeline in which a BERT-based encoder performs category-level classification, followed by a fine-tuned LLM decoder, adapted using LoRA+ and Parameter-Efficient Fine-Tuning (PEFT), that generates structured Chain of Thought reasoning sequences to derive maturity ratings and also explains the determination of the maturity level. We further critically examine the conditions under which such a system can and should be deployed, addressing the question of whether full automation is appropriate or whether hybrid human-AI oversight models are preferable. Our results demonstrate the technical feasibility of the proposed approach while surfacing important limitations regarding data quality. We argue that CoT-supported reasoning can improve the verifiability of model outputs for human evaluators and can serve as a key mechanism for calibrating confidence in partially or even fully automated maturity assessments. This work contributes an initial research paper that fundamentally addresses the application of LLMs in the determination of maturity levels.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

LLM-Supported Excerpt-Level Maturity Assessment

  • Wesley Preßler,
  • Patrick Seidel,
  • Steffen Späthe

摘要

Maturity models are widely used as instruments for assessing the current state of transformations within organizations. However, traditional assessment approaches are often resource-intensive, subjective, and difficult to scale across domains. Recent advances in large language models (LLMs) and structured reasoning techniques, such as Chain of Thought (CoT) prompting, offer new possibilities for automating and augmenting such assessments. Despite this potential, systematic approaches that leverage LLMs for maturity level determination remain underexplored. In this paper, we propose a concept for LLM-based maturity assessment grounded in a custom-developed maturity model. We design and evaluate a two-stage proof-of-concept pipeline in which a BERT-based encoder performs category-level classification, followed by a fine-tuned LLM decoder, adapted using LoRA+ and Parameter-Efficient Fine-Tuning (PEFT), that generates structured Chain of Thought reasoning sequences to derive maturity ratings and also explains the determination of the maturity level. We further critically examine the conditions under which such a system can and should be deployed, addressing the question of whether full automation is appropriate or whether hybrid human-AI oversight models are preferable. Our results demonstrate the technical feasibility of the proposed approach while surfacing important limitations regarding data quality. We argue that CoT-supported reasoning can improve the verifiability of model outputs for human evaluators and can serve as a key mechanism for calibrating confidence in partially or even fully automated maturity assessments. This work contributes an initial research paper that fundamentally addresses the application of LLMs in the determination of maturity levels.