RECE: A Family of Explainable Rubrics for Autograding Refute Problems
摘要
A Refute problem provides students with (1) a programming task and (2) a buggy solution (typically, a function) for that task. A student answers a Refute problem by specifying an input on which the function does not return the expected result (as per the task). This paper makes three contributions. First, by identifying inconsistencies in scores to Refute problems produced manually by Computer Science faculty, we demonstrate that it is challenging to manually score such problems. Second, we identify a key category of near-correct responses whose importance seems to be under-appreciated by faculty in our study. Together, these findings indicate a need for rubrics that can support automated grading and are easy to explain. As a third contribution, we propose RECE (rhyming with peace), a family of simple rubrics for autograding Refute problems. RECE rubrics reward near-correct responses from the above category, but they otherwise align with the scores suggested by faculty. We believe that these contributions will aid faculty in introducing Refute problems as assessment items in programming courses.