This chapter examines rater-mediated assessments with a focus on how the Many-Facet Rasch Model (MFRM) extends the basic Rasch model to account for multiple sources of variability that may include raters, tasks, and rating scale categories. Unlike traditional assessments where item difficulty and person proficiency are the primary parameters, rater-mediated contexts introduce additional facets—most notably rater severity/leniency—that can systematically influence scores. The chapter outlines the conceptual foundations of MFRM with an emphasis on invariant measurement principles across facets. A practical implementation is demonstrated using the Facets software, and examples showing how to estimate and interpret rater severity, task difficulty, and category threshold parameters. Diagnostic tools are available including model-data fit statistics, Wright Maps, and interaction analyses. These tools are discussed as methods for identifying misfitting raters, inconsistent scoring patterns, and potential bias in ratings. The chapter also addresses design considerations for rater-mediated assessments, including rater training, monitoring, and the importance of linking designs for comparability. By integrating rater, task, and scale effects within a unified measurement framework, MFRM provides a robust approach to enhancing score validity, reliability, and fairness in performance-based, observational, and other rater-mediated assessment contexts.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Rater-Mediated Assessments with Explanatory Rasch Models

  • George Engelhard,
  • Stefanie A. Wind

摘要

This chapter examines rater-mediated assessments with a focus on how the Many-Facet Rasch Model (MFRM) extends the basic Rasch model to account for multiple sources of variability that may include raters, tasks, and rating scale categories. Unlike traditional assessments where item difficulty and person proficiency are the primary parameters, rater-mediated contexts introduce additional facets—most notably rater severity/leniency—that can systematically influence scores. The chapter outlines the conceptual foundations of MFRM with an emphasis on invariant measurement principles across facets. A practical implementation is demonstrated using the Facets software, and examples showing how to estimate and interpret rater severity, task difficulty, and category threshold parameters. Diagnostic tools are available including model-data fit statistics, Wright Maps, and interaction analyses. These tools are discussed as methods for identifying misfitting raters, inconsistent scoring patterns, and potential bias in ratings. The chapter also addresses design considerations for rater-mediated assessments, including rater training, monitoring, and the importance of linking designs for comparability. By integrating rater, task, and scale effects within a unified measurement framework, MFRM provides a robust approach to enhancing score validity, reliability, and fairness in performance-based, observational, and other rater-mediated assessment contexts.