Changes in online ranking methodology should be transparent, inform orthopedic community
Click Here to Manage Email Alerts
Online rankings have become a normal part of life and help inform people in making many daily decisions. When buying a car, check Edmunds or Kelley Blue Book. When taking a trip, log on to Trip Advisor to see where to stay and eat.
Rankings have also become influential in higher education and health care decisions. When searching online for hospital rankings, U.S. News & World Report will be at the top of your search engine. After more than 30 years, U.S. News & World Report has become the most publicly visible ranking source for health care institutions. However, in recent years, the U.S. News rankings have been criticized by numerous leading institutions. Some top academic centers in the United States have withdrawn participation in the process with strong suggestions that the results provide artificially skewed perceptions and distorted incentives.
Ranking method
For orthopedic programs, recent rankings have demonstrated significant annual variation, suggesting nontransparent modifications to the methodology used to achieve the final ranking. A close examination reveals surprising details about the ranking method that tarnishes the impact of the U.S. News rankings.
The first step of performing a ranking analysis is to obtain consistent data among various institutions that will comprise the source of the analysis and conclusions. U.S. News data are largely obtained from the American Hospital Association, including patient satisfaction scores and CMS data. CMS data are limited to Medicare patients, with 86% of the patients being aged 65 years or older.
If you are looking for a procedure almost exclusively performed on patients younger than 65 years, including many surgeries related to sports medicine, hand or foot and ankle surgery, the online ranking may have little correlation to the competency of these orthopedic subspecialties. Less than 20% of the U.S. population is enrolled in Medicare, which suggests that the orthopedic care provided to the other 80% is not appropriately reflected in the ranking system.
The most common orthopedic procedures within the Medicare population are hip and knee replacement. Astonishingly, these procedures are excluded from the U.S. News ranking algorithm. The outcomes and complications of these procedures are similar across the entire Medicare population, especially at the top health care institutions evaluated with U.S. News rankings.
Due to the limited variability, the results of the most common elective orthopedic procedures in the Medicare population are not reflected in the overall rankings. If 50% or more of the Medicare procedures being performed are removed from the analysis, along with the absence of procedures in the non-Medicare population, the analysis and subsequent ranking represent less than 10% of the overall orthopedic procedural volume, providing minimal evidence for conclusions related to overall orthopedic program quality and outcomes.
Mortality, sites of service
Another flaw in the algorithm is using risk-adjusted mortality as a proxy for quality. Mortality is a “worst-case” approximation of outcome. In contrast, understanding outcomes in orthopedic literature is largely determined by patient-reported outcome measures (PROMs), which assess pain, function, satisfaction and quality of life. However, PROMs are not reported to CMS or any accrediting body and, thus, are not available for use. Mortality should be adjusted for patient comorbidities to be adjudicated. Otherwise, hospitals and physicians who care for the sickest patients will be penalized for higher mortality rates. The current methodology does not take into consideration the severity of the illness of the patient and the inherent risk of mortality and therefore has the potential to penalize the health care system caring for the sickest patients. Patients are likely to prefer treatment at a facility where the medical teams frequently care for the most complex situations.
Possibly the most significant flaw in the methodology is the inability to attribute all the care provided by a single orthopedic program across multiple sites of service. With the rapid acceleration of site of service from the hospital to ASCs, including major medical centers, the U.S. News data are limited to facilities with the same tax ID as the hospital. In several top programs, these facilities are joint ventures and have unique tax IDs.
These outside facilities and the care provided are typically orthopedic-specific centers focusing on quality, outcomes, efficiency and maximizing both patient and surgeon satisfaction. These may represent the best level of care provided by the orthopedic program. The U.S. News analysis can include orthopedic procedures in the Medicare population performed in ASCs with the same tax ID as the core hospital. The lack of inclusion of all cases provided by the surgeons of the orthopedic program distorts the data available for ranking, which should be more accurate to consumers and others seeking information to inform their decisions.
Transparency
U.S. News & World Report has historically been perceived as a well-respected and insightful publication. Recent changes in the methodology, with significant consequences to the rank list, should be transparent and understood by consumers and health care professionals. The limited data set, bias toward older patients yet not including the most common procedures, inability to adjust risk and the inability to include a representative body of work from some programs should encourage constructive changes to future methods, as well as the use of multiple ranking sources in addition to U.S. News data to inform decisions.
U.S. News & World Report should also seek an alternative funding model to avoid possible conflict of interest concerns. While the online rankings are essentially free to consumers, the hospitals and orthopedic programs are charged a fee to be listed and identified as being ranked in the publication. Transparency throughout the process will help to keep data relevant for informed consumers making health care decisions.
- References:
- Murrey G. Olmsted MG, et al. Methodology. U.S. News & World Report 2023-2024 Best Hospitals: Specialty Rankings. https://health.usnews.com/media/best-hospitals/BH_Methodology_2023-2024
- https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Beneficiary-Snapshot/Downloads/Bene_Snaphot.pdf
- For more information:
- Anthony A. Romeo, MD, is the Chief Medical Editor of Healio | Orthopedics Today. He can be reached at Healio, 6900 Grove Road, Thorofare, NJ 08086; email: orthopedics@healio.com.
- Nikhil N. Verma, MD, a Healio | Orthopedics Today Editorial Board Member, is professor and director, Division of Sports Medicine, and director, Sports Medicine Fellowship for Rush University Medical Center, Midwest Orthopaedics at Rush, and head team physician for the Chicago White Sox.
Collapse
Read more about
Click Here to Manage Email Alerts