<p>The intent of the residency algorithm is to match medical graduates and programs based on their ranked preferences to improve applicant-program fit and patient care. The algorithm has been used for more than a generation without an external validation that can provide feedback to improve it. This validation study runs the algorithm 2300 times using randomized applicant and program ranking inputs. The study finds that the algorithm’s matching outcomes are related to the input preferences of programs but unrelated to those of applicants. Odds ratio of matching related to the program’s ranking of applicants was 0.64 (95 CI 0.63, 0.55; <i>P</i> &lt; 0.001), suggesting a significant, lower odds of matching. The odds ratio of matching related to the program’s number of positions was 2.10 (95 CI 2.06, 2.15; <i>P</i> &lt; 0.001), again suggesting a significant, larger odds of matching to larger programs. The odds ratio of matching related to the applicants’ rank of programs was 1.01 (95 CI 0.99, 1.03; <i>P</i> &lt; 1), indicating that the influence of applicants’ ranks of programs have a significant, but small effect on odds of matching. Odds ratio of matching related to the size of the applicant’s rank-order-list, is 1.04 (95 CI 1.02,1.06; <i>P</i> &lt; 0.01), which shows that while the relationship between the variables is statistically significant, the change in odds of matching for an applicant with a larger list is only 1.04 times the original odds.</p><p>The results suggest ways of improving the algorithm to achieve better applicant-program fit and higher applicant matching odds.</p>