Black IM residents face assessment bias despite changes to ratings system

Key takeaways:

• The Milestone ratings system was linked to some improvements in assessment bias for most diverse groups.
• Still, researchers found substantial bias against Black residents after the ratings system was adopted.

United States-born Black internal medicine residents continued to be rated lower on knowledge assessments compared with white residents, even after the installation of the Milestone rating system, researchers found.

In 2014, the Accreditation Council for Graduate Medical Education replaced the Resident Annual Evaluation Summary with the Milestone ratings system to address several flaws, such as a lack of standardized evaluation criteria across programs, Bradley M. Gray, PhD, a health services researcher at ABIM, and colleagues wrote in Annals of Internal Medicine.

The researchers examined the impact of this system among residents from diverse groups “because they report experiencing bias in medical education and are underrepresented in academic leadership positions in medicine,” Gray and colleagues wrote.

“Understanding sources of bias is especially important for Black and Latino physicians because they are vastly underrepresented in medicine,” they added.

To do this, the researchers used ABIM administrative data to analyze residency ratings from 2008 to 2013, before the Milestone system was implemented, and 2015 to 2020, after Milestone was implemented. The analysis included 59,835 U.S. internal medicine (IM) residents.

Overall, estimates of knowledge ratings bias during the pre-Milestone period were:

• –0.48 standard deviations (SDs; 95% CI, –0.58 to –0.38) for U.S.-born Black residents;
• –0.44 SDs (95% CI, –0.54 to –0.34) for non-U.S.-born Black residents;
• –0.43 SDs (95% CI, –0.56 to –0.3) for non-U.S.-born Latino residents;
• –0.36 SDs (95% CI, –0.45 to –0.27) for non-U.S.-born Asian residents;
• –0.24 SDs (95% CI, –0.3 to –0.18) for U.S.-born Asian residents; and
• –0.23 SDs (CI, –0.32 to –0.14) for U.S.-born Latino residents.

After the adoption of Milestone ratings, Gray and colleagues found that the estimates for bias decreased to less than –15 SDs for all groups except U.S.-born Black residents (–0.26 SDs; –0.36 to –0.17).

“This finding is particularly concerning given their already low representation in clinical care,” they wrote. “In our sample, only 6% of IM residents identified as Black and 34% of Black residents were U.S.-born, in contrast to 90% of the U.S. Black population being born domestically.”

The researchers concluded that the consequences of substantial bias that existed before Milestone was implemented are still felt today “because most practicing physicians trained during the pre-Milestone period, when bias estimates against all of the minoritized groups that we studied were large.”

“Past research indicates that even small levels of bias could lead to an ‘amplification cascade,’ with fewer opportunities for awards and advancement,” they wrote. “Therefore, the pre-Milestone bias we observed, especially in terms of receipt of a top rating, might partially explain why so few minoritized physicians are in academic and other leadership positions.”

In a related editorial, Andrea Anderson, MD, MEd, an attending physician in the ED at Children’s Hospital of Philadelphia, and Chavon Onumah, MD, MEd, MPH, an associate professor of medicine at George Washington University School of Medicine and Health Sciences, wrote the study “is an important step in unraveling and combating the persistent nature of bias in the evaluation of internal medicine residents of color.”

“Bias is difficult to confront and must be continually and proactively addressed to counter its unconscious and negative effects,” they wrote. “Further systematic exploration of bias in trainee assessment and its unintended consequences is warranted as we endeavor to improve health workforce diversity and health outcomes for all.”

References:

• Anderson A, Onumah C. Ann Intern Med. 2023;doi:10.7326/M23-3141.
• Gray B, et al. Ann Intern Med. 2023;doi:10.7326/M23-1588.