Scholarlink

Introduction <p>Biases in the general surgery interview selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Several studies have questioned the reliability of the interview process. We aimed to improve the resident interview structure through structural improvements in the interview and increasing the validity of the selection methodology.</p> Methods <p>We performed a prospective comparative analysis of our general surgery residency interview process over two consecutive years, academic year (AY) 2022 and 2023. We used descriptive statistics and a mixed-effects ordered logit model to measure bias and guide process improvement.</p> Results <p>Eighty total students were interviewed. We found numerous statistical biases, including leniency bias exacerbated by the unequal distribution of evaluators, significant differences in the variance from evaluator’s scoring (<i>p</i> &lt; 0.0001), and non-normal distribution (<i>p</i> &lt; 0.001). Rater reliability was “good” in both years: 0.69 (CI 0.51–0.82). Using raw means, on average, each student was statistically different from 11.5 ± 1.1 other students. Using our model, we improved the number of statistically distinct groupings as each student now differed by 30 ± 1.0 others (<i>p</i> &lt; 0.0001). In AY 2023, we restructured the interviews so that all the evaluators scored every student, which significantly improved the interview accuracy using the same mixed model: R-squared of 0.95 versus 0.70, and a smaller percent of students had a change in their rank using the improved structure (57% vs. 90%, <i>p</i> &lt; 0.05), compared to AY 2022.</p> Conclusions <p>Among a pool of students applying for general surgery residency at a single institution, our study shows that using the unadjusted evaluators’ impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection.&#xa0;Due to numerous statistical biases, there is little differentiation between students and thus little validity in scoring students using raw mean scores. This can be overcome by developing a structured interview and correcting for statistical biases. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others’ impressions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Improving the resident interview process with structural and statistical bias correction

  • 期刊论文

  • 发表日期：

    2026年2月2日

查看全文
  • Sultan Abdelhamid,
  • Zaid Haddadin,
  • Molly Casey,
  • Emery Cuellar,
  • Moshumi Godbole,
  • Ramsey Dallal

摘要

Introduction

Biases in the general surgery interview selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Several studies have questioned the reliability of the interview process. We aimed to improve the resident interview structure through structural improvements in the interview and increasing the validity of the selection methodology.

Methods

We performed a prospective comparative analysis of our general surgery residency interview process over two consecutive years, academic year (AY) 2022 and 2023. We used descriptive statistics and a mixed-effects ordered logit model to measure bias and guide process improvement.

Results

Eighty total students were interviewed. We found numerous statistical biases, including leniency bias exacerbated by the unequal distribution of evaluators, significant differences in the variance from evaluator’s scoring (p < 0.0001), and non-normal distribution (p < 0.001). Rater reliability was “good” in both years: 0.69 (CI 0.51–0.82). Using raw means, on average, each student was statistically different from 11.5 ± 1.1 other students. Using our model, we improved the number of statistically distinct groupings as each student now differed by 30 ± 1.0 others (p < 0.0001). In AY 2023, we restructured the interviews so that all the evaluators scored every student, which significantly improved the interview accuracy using the same mixed model: R-squared of 0.95 versus 0.70, and a smaller percent of students had a change in their rank using the improved structure (57% vs. 90%, p < 0.05), compared to AY 2022.

Conclusions

Among a pool of students applying for general surgery residency at a single institution, our study shows that using the unadjusted evaluators’ impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection. Due to numerous statistical biases, there is little differentiation between students and thus little validity in scoring students using raw mean scores. This can be overcome by developing a structured interview and correcting for statistical biases. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others’ impressions.