<p>Randomized controlled trials (RCTs) are the gold standard for evaluating intervention effectiveness. In education research, students are often screened to identify those with reading difficulties (SWRD) who are eligible to participate. However, criteria for identifying SWRD vary widely across studies and are often shaped by practical constraints such as budget or statistical power. Common approaches include using publisher-defined risk categories, percentile-based thresholds, or selecting a fixed number of the lowest-performing students per classroom. These study-specific decisions, though necessary, may introduce unintended bias. In this simulation study, we examined how different eligibility criteria, applied to the same underlying population with a fixed “true” effect, can influence estimated intervention effects. Results showed that all tested methods produced some degree of bias in effect size estimation, but the magnitude and direction of bias were inconsistent and unpredictable. These findings suggest that variation in how researchers define SWRD can distort the field’s understanding of intervention efficacy. We suggest future areas of research to guide the field toward a solution that is both statistically valid and practically feasible.&#xa0;</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

The Influence of Study Selection Criteria on Statistical Conclusion Validity: A Simulation Study

  • Allyson L. Hayward,
  • Jeffrey A. Shero,
  • Jessica A. R. Logan

摘要

Randomized controlled trials (RCTs) are the gold standard for evaluating intervention effectiveness. In education research, students are often screened to identify those with reading difficulties (SWRD) who are eligible to participate. However, criteria for identifying SWRD vary widely across studies and are often shaped by practical constraints such as budget or statistical power. Common approaches include using publisher-defined risk categories, percentile-based thresholds, or selecting a fixed number of the lowest-performing students per classroom. These study-specific decisions, though necessary, may introduce unintended bias. In this simulation study, we examined how different eligibility criteria, applied to the same underlying population with a fixed “true” effect, can influence estimated intervention effects. Results showed that all tested methods produced some degree of bias in effect size estimation, but the magnitude and direction of bias were inconsistent and unpredictable. These findings suggest that variation in how researchers define SWRD can distort the field’s understanding of intervention efficacy. We suggest future areas of research to guide the field toward a solution that is both statistically valid and practically feasible.