Scholarlink

<p>Large language models (LLM) make it possible to systematically evaluate text material from therapy sessions and convert it into standardized ratings. The present study investigates whether the LLM-based multi-agent system Psychological Rater Artificial Intelligence (PsyRAI) can be developed for the automated assessment of basic clinical skills. The starting point is the clinical micro-skill training scale (CMST), which is used to assess the communicative, interpersonal, process-related and time management skills of prospective psychotherapists. Analogous to the training of human raters, the CMST assessment rules were transferred to a&#xa0;prompt structure and evaluated iteratively. This evaluation was based on a&#xa0;development sample of 150 video-based therapist responses that had been automatically transcribed beforehand. The agreement between artificial intelligence (AI) and human ratings was determined using intraclass correlations (ICC) and validated against random fluctuations using a&#xa0;permutation-based null model. The system’s baseline prompts initially achieved lower agreement with human ratings but were significantly improved through iterative prompt optimization (ICC ≈ 0.60). It was found that different CMST items require different prompt structures. The results suggest that LLM-based ratings of clinical micro-skills are possible in principle but require careful instructional specification and psychometric evaluation. The PsyRAI system can thus provide a&#xa0;basis for scalable competency feedback in the training and continuing education of psychotherapists, the effectiveness and perceived usefulness of which should be further examined.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Entwicklung eines KI-gestützten Systems zur Bewertung klinischer Basisfertigkeiten

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月29日

查看全文
  • Antonia Vehlen,
  • Jana Bommer,
  • David Riebschläger,
  • Jeanne Hierse,
  • Steffen Eberhardt,
  • Christoph Maerz,
  • Wolfgang Lutz

摘要

Large language models (LLM) make it possible to systematically evaluate text material from therapy sessions and convert it into standardized ratings. The present study investigates whether the LLM-based multi-agent system Psychological Rater Artificial Intelligence (PsyRAI) can be developed for the automated assessment of basic clinical skills. The starting point is the clinical micro-skill training scale (CMST), which is used to assess the communicative, interpersonal, process-related and time management skills of prospective psychotherapists. Analogous to the training of human raters, the CMST assessment rules were transferred to a prompt structure and evaluated iteratively. This evaluation was based on a development sample of 150 video-based therapist responses that had been automatically transcribed beforehand. The agreement between artificial intelligence (AI) and human ratings was determined using intraclass correlations (ICC) and validated against random fluctuations using a permutation-based null model. The system’s baseline prompts initially achieved lower agreement with human ratings but were significantly improved through iterative prompt optimization (ICC ≈ 0.60). It was found that different CMST items require different prompt structures. The results suggest that LLM-based ratings of clinical micro-skills are possible in principle but require careful instructional specification and psychometric evaluation. The PsyRAI system can thus provide a basis for scalable competency feedback in the training and continuing education of psychotherapists, the effectiveness and perceived usefulness of which should be further examined.