USMLE Step 1 change has unintended effects on residency selection
The quantifiable nature of the United States Medical Licensing Examination Step 1 score lends itself to secondary uses, including medical education assessment and evaluating students for promotion and graduation.
Scores between 200 and 210 have been found to correlate with passing the board examination in many specialties, however, no correlation between higher (United States Medical Licensing Examination) USMLE Step 1 scores, and performance during residency has been found. Although the evidence supporting the utility of the Step 1 exam to predict success in residency training is scant, historically, a high Step 1 score has been a critical factor in obtaining a residency interview, regardless of medical specialty.
Some believe medical students’ well-being has been negatively affected by the importance placed on the Step 1 result, with nearly one in four students reporting moderate to high anxiety surrounding the exam. There is also a belief that the time spent preparing for this examination detracts from important aspects of students’ personal and clinical development, such as community service, research, shadowing, exercise and other activities, to reduce stress and promote health.
Competition for residency positions can be high. Data from the 2018 National Resident Matching Program (NRMP) showed the number of applicants per residency position exceed the available positions in almost all specialties. The Step 1 scores are used to screen residency applicants to make the review of thousands of applications more manageable. This practice is seen in all programs from those with historically high chances of matching, such as pediatrics (98.8% match rate), to the more competitive specialties, such as orthopedic surgery (82.4% match rate).
In February 2020, the National Board of Medical Examiners (NBME) and Federation of State Medical Boards (FSMB) announced that, starting in 2022, the USMLE Step 1 score would change from a numeric score with associated percentiles to reporting only pass/fail. A major reason behind this change was to lessen the “Step 1 climate” in medical school, where it was believed too much focus had been diverted from clinical development to USMLE Step 1 preparation. Although those in favor of the pass/fail change believe it will have a positive effect on stress/well-being, promote a holistic application review, and increase diversity, there are also arguments that switching to pass/fail will have the opposite effects.
Stress on applicants
Medical school is a time that students should not only learn medicine and patient care, but also how to manage stress effectively. High-stakes examinations are not foreign to medical students as the Medical College Aptitude Test (MCAT) is required for entry into medical school. Learning to deal with stress effectively should be an integral part of students’ education and should not be avoided. Students will be ill prepared for the future as these stressors do not end after medical school. It is well researched that dealing with stress, deadlines, balancing work and home life continues after medical school. With the USMLE Step 1 pass/fail change, medical schools will be doing students a disservice by not preparing them for what lies ahead. Furthermore, emphasis will be placed on the USMLE Step 2 numeric score, merely pushing this stress back 1 year. The concerns about stress are simply postponed.
Holistic approach
Although a goal of the grading change was to motivate holistic application review instead of relying on a numeric score as a filter, programs may be challenged to evaluate the applications in a holistic manner in a limited amount of time, resulting in greater competition for residency interviews due to an increased number of applications.
With the pass/fail change, the number of applications per program will likely increase and result in even more competition for an interview position. It should be noted, that for the 2021 cycle, almost every specialty had an increase in applications per applicant. In the 2022 application cycle, there were 8,389 students who applied to internal medicine with an average of 35 applications per student (an increase from 2018, 2019 and 2020). Orthopedic surgery had the highest number of applications per applicant at 97 compared with 86 applications per applicant in 2020 (total applicants 1,283).
There could be tens of thousands of applications requiring a review for two to 30 spots depending on the specialty.
Recommendations including formal training to address implicit bias, quantification of criteria such as clinical skills, professionalism, leadership skills, community service and advocacy may make the residency selection process more holistic. These recommendations do not address how to sift through the increased number of applications in a finite period, nor do these address the fact that the score can be an efficient screening tool for low outliers at risk for not passing their specialty boards.
The pass/fail change will result in the loss of an objective data point to compare applicants equally which may lead programs to weigh other variables when deciding on granting an interview. Factors, such as tier of medical school, USMLE Step 2, tier of undergraduate education, MCAT and phone calls made by programs, mentors, and friends of applicants in the specialty may be weighted higher. Other than the USMLE Step 2 score, none of these variables have been shown to correlate with performance during residency and some would argue are fraught with inequities and biases that run counter to gains in diversity and inclusion.
Opinions of medical students and residents regarding USMLE Step 1 numerical vs. pass/fail grading has been a discussion for almost a decade. A 2011 study surveying medical students and residents found that only 26% of respondents were in favor of a pass/fail grading system, citing that the numerical score is advantageous for resident selection. A recently published study of applicants applying to orthopedic surgery showed only 21.2% of applicants were in favor of the pass/fail change. This disparity brings up an important question – Why are applicants, especially those interested in competitive specialties, largely not in favor of transitioning to a grading system that is intended to decrease the burden of stress related to the residency selection process, allow for more time with family, wellness, research and shadowing? Most orthopedic applicants surveyed believe the consequences of the pass/fail change will have the opposite effect.
USMLE Step 2
Most program directors (80.7%) from a New England Journal of Medicine study report that USMLE Step 2 Clinical Knowledge scores will become more important in resident selection. Although the USMLE Step 2 exam is thought be more indicative of performance during residency, the examination is administered at different times during the academic year, depending on the institution. Some students may feel pressured to take the examination prior to completing their core clerkships and yet will be tested in those specialties they have not yet been formally taught. For other students, the test may not be administered before residency applications are due, so these students will not have scores to submit for their residency application. The shift of focus to the USMLE Step 2 will result in testing stress to be pushed back a year and more importantly will not be equitable to all students based on test timing as compared with the USMLE Step 1.
Improved diversity?
Although some contend that the pass/fail change will be beneficial for historically underrepresented medical students, especially those applying to more competitive specialties that are generally the least diverse in medicine, the pass/fail change may have the opposite effect.
Furthermore, with the pass/fail change comparing students from schools without grades/Alpha Omega Alpha (AOA) status/class rank, as seen with many of the upper-tier medical schools, to students from middle- or lower-tier medical schools where there are class rank/clerkship grades/AOA may be even more challenging. There are no objective data points, only narrative evaluations which may be fraught with bias.
The USMLE Step 1 numeric score can serve to “level the playing field” for applicants, as it can compare students regardless of where they attended medical school or college. How can students be compared equally if one attends a medical school with grades and the other student attends a medical school without grades? Who will have the advantage?
The USMLE numeric score helped to give the middle- and lower-tiered students a chance to compete with those from upper-tiered medical schools. The students who will suffer with this change are the historically underrepresented medical students, international medical graduates, osteopathic students, and students of middle- and lower-tiered medical schools.
It is our fear that residency review committees may overlook some applicants who would have otherwise stood out due to their Step 1 scores and may now be bypassed. The goal of the USMLE Step 1 pass/fail change to increase diversity will fall short.
Opportunities for improvement
Although well-intentioned, the pass/fail change will not create a more equitable process and will have unintended consequences. The following are some options to address the concerns raised by the proponents of the USMLE Step 1 pass/fail change:
Medical schools can take an active role and include curricula/strategies to help give students the necessary skills to deal with stressors both during and after medical school which may help with the excessive stress students are currently feeling.
Although a holistic approach to residency selection is ideal, it is impossible if objective data points are removed as the number of applicants will only increase making a holistic approach impossible. To holistically review thousands of applications is impractical; the number of applications must be decreased to a manageable number. As there are no limits to the number of programs students can apply to, programs will turn to other subjective data to decrease the number of applications, many of which are fraught with inequities. Programs, however, are starting to have supplemental essays or pre-interview video questions, or some specialties have begun using “signaling” a finite number of programs to decrease the number of applications and allow for more holistic review. The system feeds upward and addressing the inequities at the entry point may help the diversity in medical school and residencies. Making the USMLE Step 1 test pass/fail should not be seen as the cure to a problem that starts earlier.
Medical schools should also agree to reveal applicants’ Shelf examination scores and agree to have class ranks to compare applicants fairly as it is difficult to compare applicants from schools with no grades to schools with grades.
Making the test pass/fail will not improve stress levels, will not lead to a holistic review and will not allow for fair comparisons of applicants from different backgrounds and medical schools. The hopes are that medical students will graduate medical school prepared for their future and the ability to deal with the stressors of life as a physician, for programs to be able to provide a holistic review of applicants, for applicants from all programs to be compared fairly, and for the diversity that is lacking in all specialties of medicine to improve; however, making the USMLE Step 1 pass/fail is not the answer and will only exacerbate these issues..
References:
Al Khalili K, et al. World Neurosurg. 2014;doi:10.1016/j.wneu.2013.08.019.
Arguello A, et al. J Am Acad Orthop Surg. 2022;doi:10.5435/JAAOS-D-21-00615.
Armstrong A, et al. Mil Med. 2007;doi:10.7205/milmed.172.6.640.
Bernstein AD, et al. J Bone Joint Surg Am. 2002;doi:10.2106/00004623-200211000-00026.
Chen DR, et al. Acad Med. 2019;doi:10.1097/ACM.0000000000002565.
Dillon GF, et al. J Grad Med Educ. 2013;doi:10.4300/JGME-D-12-00205.1.
Green M, et al. Acad Med. 2009; doi:10.1097/ACM.0b013e3181970c6b.
Green M, et al. Clin Teach. 2016;doi:10.1111/tct.12386.
Hoffman KI. Acad Med. 1993;doi:10.1097/00001888-199310000-00003.
Jones MD, Jr., et al. BMC Med Educ. 2018;doi:10.1186/s12909-018-1138-x.
Kenny S, et al. Med Educ. 2013;doi:10.1111/medu.12234.
Lee M, et al. Med Teach. 2018;doi:10.1080/0142159X.2018.1430353.
Lewis CE, et al. J Grad Med Educ. 2011;doi:10.4300/JGME-D-10-00121.1.
Makhoul AT, et al. N Engl J Med. 2020;doi:10.1056/NEJMp2006148.
McCaskill QE, et al. Ambul Pediatr. 2007;doi:10.1016/j.ambp.2007.01.002.
McGaghie WC, et al. Acad Med. 2011;doi:10.1097/ACM.0b013e3181ffacdb.
Prober CG, et al. Acad Med. 2016;doi:10.1097/ACM.0000000000000855.
Prober CG. N Engl J Med. 2020;doi:10.1056/NEJMp2003880.
Program NRM. National Resident Matching Program, Results and Data: 2020 Main Residency Match. Washington DC. 2020:128.
Scoring I-ICoU. Change to pass/fail score reporting for Step 12020.
USMLE Step 1 transition to pass/fail only score reporting. USMLE. Available at: https://www.usmle.org/usmle-step-1-transition-passfail-only-score-reporting.
Verret CI, et al. Clin Orthop Relat Res. 2021;doi:10.1097/CORR.0000000000001457
Weissbart SJ, et al. Urology. 2015;doi:10.1016/j.urology.2014.12.041.
Williams RG. Acad Med. 1993;doi:10.1097/00001888-199310000-00004.
Collapse
Read more about