What’s wrong with the 5-star rating system for the renal community

The end-stage renal disease program has led American health care quality measurement by utilizing robust and transparent metrics. Using this data to drive change, there have been year over year improvements in intermediary outcomes and resultant reduced mortality over time for the ESRD population.

The Dialysis Facility Compare (DFC) website, launched by the Centers for Medicare & Medicaid Services in 2001, has become a prominent destination for those interested in ESRD quality. Historically, various quality measures have been displayed there. In 2011, with the inception of the ESRD Quality Improvement Program (QIP), the website was updated to include only QIP measures. In 2013, three non-QIP measures were added: Standardized Mortality Ratio (SMR), Standardized Hospitalization Ratio (SHR) and Standardized Transfusion Ratio (STrR).

On July 10, CMS announced the transformation of the site into a star rating system, ranging from 5 stars (excellent) to 3 stars (average) to 1 star (poor). Star ratings were forced into a bell curve distribution with 10%, 20%, 40%, 20% and 10% for 5, 4, 3, 2 and 1 star, respectively.

Providers, patient groups, and others have expressed serious concerns regarding the creation of a new facility quality measurement system, which differs from the legislatively mandated QIP, as well as the utility of the star methodology itself for consumer use.

ESRD facilities are required to post a CMS-issued QIP certificate annually and have done so for years, with the goal of informing patients about the success of the facility in meeting QIP metrics. Whether achieving success on QIP metrics truly indicates the quality of care in a facility remains a subject of intense debate, the creation of the 5 star system, a second and potentially discordant measure of ESRD facility quality, will be confusing to patients.

The 5 star program uses most of the existing QIP metrics with the addition of three additional measures, and uses a bell curve scoring system, so results may differ from what is shown on the QIP certificate. Facilities that may have stellar QIP scores will not necessarily have 5 stars on DFC. It is unclear how patients will interpret two differing CMS-endorsed assessments of quality for the same dialysis facility.

The use of the forced bell curve is inconsistent with many other rating systems with which consumers are familiar. Outside of health care, systems used for hotels or restaurants use the attainment of a given quality standard to award stars. This allows the consumer to understand the level of expected quality with meaningful differentiation. If the standard is met there can be as many 5 star restaurants or hotels that achieve that level of quality. None of those rating systems uses a bell curve. A forced bell curve also assumes that the differences between star tiers for dialysis facilities are clinically meaningful for patients, but to date no validation data to substantiate that assertion has been provided. Therefore a patient choosing to drive further to a 5 star vs a 4 star unit on DFC may be doing so for no benefit at all. In other CMS 5-star systems a bell curve is not used.

A good comparison is the CMS Nursing Home Compare. There the 5 star % has doubled from 2009 to 2014 and is now at 25% of facilities. This enables the star system to be seen as a tool to drive change, to the extent you buy into the metrics used. Using a bell curve, there will never be the ability to see change occurring.

The use of the broad SMR, SHR and STrR measures is problematic. There have been concerns raised for some time on the lack of transparency in the methodology used to calculate these ratios and the need for verifiable validation of their accuracy and predictive value. In addition, many factors which contribute to these metrics may be out of a dialysis facility’s control. For example, virtually no transfusions are given in the dialysis unit, yet the STrR is included as a measure of facility quality. To include this in a metric is the equivalent of reducing a restaurant’s star rating based on the amount of traffic, a factor clearly out of the restaurant’s control.

Interestingly, CMS has requested comments in the current proposed rule for the Prospective Payment System and the QIP for some of these very measures and to date, unlike the other DFC measures, none of these have been endorsed by the National Quality Forum. Such metrics are more applicable to a coordinated care model advocated by the community (i.e., the forthcoming Comprehensive ESRD Care model), than for individual dialysis facilities.

What could be done to prevent the likely confusion by the patients between the QIP scores and the Star Ratings? The core issue is the 2013 expansion of DFC beyond the QIP measures. One easy solution to harmonize this is to simply transform the 5 QIP penalty tiers into stars. (i.e. the best performing facilities, those with 0% QIP penalties, would be awarded 5 stars, etc.) This simple solution would address many issues and be consistent with consumer rating systems. The ESRD community, having worked closely with CMS on measure development and validation in the QIP, will be advocating for this approach in an effort to have one unified measure to allow facilities, providers, and patients to understand relative quality. After all, in today’s information overloaded world, isn’t simpler better? -by Mahesh Krishnan, MD, MPH, MBA, FASN; Allen R. Nissenson, MD, FACP