Automatic Scoring of Constructed Responses in PISA Using the Machine-Supported Coding System
摘要
This chapter illustrates how a machine supported coding system (MSCS) was developed and implemented for automatic scoring of constructed responses in the Programme for International Student Assessment (PISA) and presents its performance. As an international large-scale survey, PISA 2018 collected data from students across more than 80 countries and in about 50 languages in the computer-based mode. Conceptually, the MSCS can be viewed as a supervised learning of natural language data that is generalizable across multiple languages to be used for PISA data. Without extracting language-dependent features, the MSCS utilizes the exact-matching method and automatically applies a verified label to a subset of incoming uncoded responses, which guarantees the highest accuracy in the prediction. In the PISA 2018 assessment cycle, the MSCS has shown to alleviate scoring burden as much as 25% on average across all participating countries and across domains. This chapter starts with the motivation for developing the MSCS to benefit a large-scale setting with multiple languages of assessment. We also present the findings from an analysis of PISA 2018 data and explore possible avenues for continuing to improve.