Technical Platforms for the Improvement of Coding Reliability: OECS and OERS
摘要
In the 2018 administration of the Programme for International Student Assessment (PISA), there were over 7-million constructed responses collected from students on items in the major domain of Reading alone, the majority of which were assigned to 1300 human coders, across 139 unique country-by-language groups. This sheer volume of data requiring human coding necessitates computer-based coding systems to facilitate independent coding and support coding teams. This chapter focuses on PISA’s international coder training, efficient coding designs, and most importantly, the coding systems employed to manage the vast number of responses assigned to coding teams working across the globe during test administration. The Open-Ended Response System (OERS) and Open-Ended Coding System (OECS) provide national coding teams with formative statistics on coder agreement and overall coding reliability throughout the coding process, giving them insight and awareness of how accurately codes are being applied. These systems automatically alert lead coders to low levels of coding agreement and flag patterns of inaccurate coding. With such coding systems in-place, not only can large amounts of constructed response data be effectively managed in international large-scale assessments, but measures of coding accuracy and reliability can be calculated and reviewed for areas of future system and assessment improvement.