Formative feedback on early-stage project ideas is essential in software project management education, as it helps students refine their concepts and establish a basis for subsequent coursework. In large-enrollment courses, limited tutoring capacity and inconsistencies in assessment create challenges for maintaining feedback quality. However, as large language models (LLMs) become more integrated into educational settings, examining their role in supporting tutor assessments of student project ideas is increasingly relevant. To explore its potential for formative assessment, a within-subject crossover study was conducted in which six tutors assessed 76 student software project ideas, each paired with either an LLM- or expert-generated pre-assessment. The results indicate that LLM-based pre-assessments affect tutor ratings, whereas expert-generated pre-assessments lead to higher consistency and closer alignment across raters. Pre-assessment ratings significantly predicted tutor ratings across all six evaluation dimensions. Although tutors engaged with both types of feedback, expert-generated pre-assessments produced higher inter-rater reliability. Reflections captured in a post-study questionnaire revealed mixed perceptions of LLM-generated assessments. Some were considered useful, while others were criticized for lacking contextual awareness and depth. These findings suggest that LLMs can support formative assessment processes, but expert input remains vital for ensuring coherence and reliability.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Supporting Tutors in Assessing Student Software Project Ideas with LLM-Based Pre-assessments

  • Sebastian Gürtl,
  • Jonathan Maier,
  • Daniel Scharf,
  • Marcus Gugacs,
  • Christian Gütl,
  • Alexander Steinmaurer

摘要

Formative feedback on early-stage project ideas is essential in software project management education, as it helps students refine their concepts and establish a basis for subsequent coursework. In large-enrollment courses, limited tutoring capacity and inconsistencies in assessment create challenges for maintaining feedback quality. However, as large language models (LLMs) become more integrated into educational settings, examining their role in supporting tutor assessments of student project ideas is increasingly relevant. To explore its potential for formative assessment, a within-subject crossover study was conducted in which six tutors assessed 76 student software project ideas, each paired with either an LLM- or expert-generated pre-assessment. The results indicate that LLM-based pre-assessments affect tutor ratings, whereas expert-generated pre-assessments lead to higher consistency and closer alignment across raters. Pre-assessment ratings significantly predicted tutor ratings across all six evaluation dimensions. Although tutors engaged with both types of feedback, expert-generated pre-assessments produced higher inter-rater reliability. Reflections captured in a post-study questionnaire revealed mixed perceptions of LLM-generated assessments. Some were considered useful, while others were criticized for lacking contextual awareness and depth. These findings suggest that LLMs can support formative assessment processes, but expert input remains vital for ensuring coherence and reliability.