USMLE Step 1 score reporting changes offer opportunity to improve diversity, well-being
Click Here to Manage Email Alerts
The recent change in score reporting to pass/fail for the USMLE Step 1 exam is creating understandable concern among residency program directors as this will require changes to the process of selecting applicants to interview.
The selection process has become even more challenging recently given the increase in the number of applications per program, now that all or at least most interviews are conducted in a virtual format. There has historically been reliance on a three-digit Step 1 score when reviewing residency applications, as it has been demonstrated that student performance on that exam correlates with the likelihood of eventually passing the American Board of Orthopaedic Surgery (ABOS) Part I exam. However, this change in score reporting provides a reminder that United States Medical Licensing Examination (USMLE) exams were developed and are intended to identify physicians who can be safely licensed to practice medicine, not for the secondary intent of identifying appropriate residency applicants.
Opportunity to reflect
This change in score reporting provides opportunity for the orthopedic community to reflect on what we are looking for in an applicant, how we define a successful orthopedic resident and practicing orthopedic surgeon, and if there are better ways to evaluate applications and applicants.
Utilizing three-digit scores allows for defined cutoffs for application review can be a clear benchmark to students and their advisors, but are we not paying sufficient attention to students who would make outstanding orthopedic surgeons? And is achieving a specific score to improve the odds of receiving an interview and matching in a competitive field such as orthopedic surgery worth the impact on education and on health and wellbeing while preparing for and taking Step 1? And what does the use of the score mean for improving the diversity of the profession?
The USMLE Step 1 exam has been seen by students, and in many cases accurately so, as “the only test you’ll ever take in your entire life where you can totally eliminate future career options if you don’t perform well on test day.” Students have also noted an impact, usually negative, on their education while they are preparing for an exam that has such significant emphasis placed on it by residency programs. “People spend time studying for this test and miss out on cases in the hospital and actually learning how to be a good doctor,” according to a study in the Journal of Medical Regulation.
In addition, given the importance of the exam score, especially for students planning on applying to competitive specialties such as orthopedic surgery, some students experience high levels of stress and isolation or loneliness. It is unknown if these resolve once the test is taken or if these responses contribute to future burnout and loneliness. However, given the rates of burnout and depression among orthopedic residents, especially the former among more junior residents, and the rates of depression and suicidal ideation among practicing orthopedic surgeons, it is imperative that we prioritize orthopedic surgeon well-being and mental health, starting with experiences in medical school. Changing the score reporting system for USMLE Step 1 may just transition stressors to Step 2 (clinical knowledge) CK, but this needs future study. In addition, Step 2 is more clinically based, and three-digit scores for this examination may provide information more relevant to training programs.
Improve diversity
The change in score reporting also provides opportunities for orthopedic surgery to continue to address and improve diversity in the profession. It has been estimated that it will take more than 2 centuries for orthopedic surgery to be on par with the rest of medicine in terms of gender diversity, and there is even less racial diversity in the profession. While it has been assumed that criteria we use to assess and rank applicants, especially medical school grades, (Alpha Omega Alpha) AOA status, and USMLE Step 1 scores, are objective, in reality, many are not. Unconscious biases can be reflected in subjectivity in clinical grades and AOA status, as well as in letters of recommendation, disadvantaging underrepresented racial minority students. However, there are also differences in USMLE exam scores based on student demographics. Women have been reported to have lower USMLE Step 1 and Step 2 scores. This is a complex issue and may reflect that women often face conflicts between their dual roles as student and societal-gendered expectations at home that can impact the time available to study for high-stakes exams. Historically underrepresented students can face issues with stereotype perception threat related to the USMLE Step 1 examination. “If you’re of this minority [group], you need to have a higher score because you’re going to be judged differently,” according to the study in the Journal of Medical Regulation. Stereotype perception threat has been found in other arenas to negatively impact performance.
The increased emphasis on USMLE Step 1 scores for residency applications, continued escalation in score, and differences in curricula among schools has led to an “arms race” in obtaining test prep materials, especially for students who plan on applying to competitive specialties. These materials can cost more than $500 each, with students usually utilizing more than one. The proliferation of test prep materials may help to explain, in part, the increases in USMLE Step 1 scores during the past several years, especially among students who match into orthopedic surgery residencies.
Financial resources
However, the ubiquitous and increasing use of test prep materials creates another bias in the exam process, with students with more financial resources being able to afford more test prep resources and courses, likely resulting in higher scores and improving their odds of matching into the residency and specialty of their choice. Those with fewer resources attempt to keep up in the use of test prep materials, but this adds financial concerns to other stressors related to the exam. Students with more financial resources have likely had advantages during most or all of their education journeys, and being able to afford additional USMLE Step 1 test prep materials is just one more advantage in reaching their career goals. No longer reporting a three-digit score may hopefully remove the drive to purchase increasing numbers of test prep materials and help to remove financial resources as an indirect metric used in the residency application review process.
Level the playing field
The change in USMLE Step 1 score reporting provides an opportunity to level the playing field among potential orthopedic residency applicants and offers opportunities to improve diversity and well-being in orthopedics. The change can be seen as an invitation to reassess what we are looking for among students and residents. Are we placing disproportionate emphasis on applicants who can or have learned how to do well on a test? Do we also need to assess qualities in interpersonal interactions? If so, how can this be reflected in residency applications?
Levels of grit have been shown in other professions to be indicative of rates of success, and applicants to orthopedic surgery residencies have been found to have higher levels of grit than most of the population. How much emphasis should we place on psychometric variables, such as grit, in identifying applicants who will do well in residency training and in practice? Are characteristics such as interpersonal skills and grit just as or more important than the ability to score well on a test? These characteristics are harder to measure and may be more nebulous than a 3-digit exam scores, but they if are important to being a successful orthopedic surgeon, should we spend time in finding ways to consistently evaluate these across training programs?
Assess attributes
We have historically relied on several metrics, including the USMLE Step 1 three-digit score, to assess applicants because we assumed these metrics were objective. We now know most are not.
We are also developing a better understanding of the impact on students in achieving these metrics.
While challenging, now is the time for conversations to identify how we move forward as a profession to more holistically evaluate residency applicants. Changes in this evaluation process may lead us to interviewing outstanding applicants that previously would not have met our criteria and can help to improve well-being and diversity in orthopedic surgery.
References:
Boatright D, et al. JAMA Intern Med. 2017;doi:10.1001/jamainternmed.2016.9623.
Camp CL, et al. J Am Acad Orthop Surg. 2019;doi:10.5435/JAAOS-D-17-00545.
Gauer JL, et al. Adv Med Educ Pract. 2018;doi:10.2147/AMEP.S152684.
Lichstein PM, et al. Clin Orthop Relat Res. 2020;doi:10.1097/CORR.0000000000001310.
Poon SC, et al. Clin Orthop Relat Res. 2022;doi:10.1097/CORR.0000000000001553.
Stein MK, et al. J Bone Joint Surg Am. 2022;doi:10.2106/JBJS.21.00666.
Templeton K, et al. Journal of Medical Regulation. 2022;doi:10.30770/2572-1852-108.2.7.
Collapse
Read more about
Click Here to Manage Email Alerts