Misleading patients: CMS blindly charges ahead with 5-Star

This blog post was originally published Sept. 15, 2014, for the Acumen Nephrology Blog.

“When buying a product or service, looking at ratings can often help narrow down the choices. Some websites offer ‘star’ ratings that give information about the quality of the products and services they offer. Wouldn’t it be helpful to have the same kind of ratings when choosing a health care provider?”

Sounds like a great idea doesn’t it? Create a rating system for health care that rolls up a lot of complex health care quality data into a star “rating” system that’s easy for consumers to understand. It makes perfect sense to me.

The quote is from a blog post by Dr. Patrick Conway in mid-June of this year. For those of you who do not know Dr. Conway, he is the Chief Medical Officer at CMS. A pediatrician by training, Dr. Conway is well respected within many circles. Unfortunately, during the translation of his idea from concept to reality, CMS fell asleep at the wheel and the resulting train wreck is a travesty.

Frequent readers of this blog have been introduced to the perils of the Dialysis Facility Compare 5-Star “ratings” system. Last week, in spite of overwhelmingly negative response from the renal community, CMS announced they are moving forward with the release of the 5-Star system in January of 2015. The list of organizations opposed to 5-Star as written is far too long for a single blog post, but it includes the likes of the Medicare Payment Advisory Committee (MedPAC), Kidney Care Partners, the American Kidney Fund, the National Kidney Foundation, Dialysis Patient Citizens, and the Renal Support Network, to name a few. Perhaps more telling, other than CMS, no one supports this program.

So why is everyone up in arms about 5-Star? At a high level there are two things that must work reasonably well in order for a program like this to provide value to patients:

Data integrity: The data used to formulate the rating system must accurately reflect what’s happening “in the field.”
Methodology: Once you have accurate data, what you do with it must result in a solution that permits patients to reliably compare the quality of care delivered by health care providers—dialysis facilities in this case.

The truth of the matter is that 5-Star fails miserably on both counts.

Data integrity
This sounds like a simple task, but in fact it is not. There are approximately 75 million in-center dialysis treatments administered each year in the nation’s 6,000+ dialysis facilities. How does CMS collect 5-Star data from dialysis facilities? The answer is a beast by the name of CROWNWeb. Over a decade in the making, CROWNWeb is the solution in place today to harvest quality data from dialysis facilities and it remains a work in progress. Setting aside the fact that there are multiple ways a dialysis clinic can ship data via CROWNWeb, it remains remarkable to me how often the patient-level data transmission fails. As the recent fiasco surrounding healthcare.gov reminds us, information technology is not one of Health and Human Services’ core competencies. But folks, let me assure you, Healthcare.gov smells like a rose compared with the odor emanating from CROWNWeb. The contractors in charge seem to change frequently and in spite of over ten years of effort, the majority of dialysis facilities see patient data transmission failure rates as high as 10%.

Why is this important? As noted below, many of the measures that make up 5-Star are so tightly bunched, there is very little separation in several measures across the country. At Fresenius we discovered this the hard way. Due to a technology glitch, around 30,000 dialysis treatment data points did not make the trip to CMS last year. Sounds like a small number compared to the 75 million treatment figure I mentioned above, doesn’t it? When we went back and recalculated the “stars” for the impacted clinics, the result was that over 600 dialysis facilities were rated incorrectly. Folks, that’s 10% of the country’s dialysis clinics—ranked incorrectly because of a single data integrity issue from a single dialysis provider.

While we are on the subject of data integrity, let’s talk about missing data. As part of this program, CMS expected the 6,033 dialysis facilities in this country to report seven individual measures (five if you only do PD). Remarkably almost 1 in 5 (1,130 facilities to be precise) were unable to send at least one of the seven individual measures. Undaunted, CMS pushed ahead, assigning an “average” value for half the clinics impacted and leaving the other half completely out of the program. What other rating system moves forward when almost 20% cannot report data?

As a patient, all I see is a star rating. I am blind to the incompetence behind the system, and absent full disclosure I will be misled in the best-case scenario, deceived at worst.

Methodology
As a technology guy, I actually think with additional time, effort, and money, we can solve the data integrity issue. But that leaves us with the biggest failure of 5-Star and the one that will truly mislead our patients: a seriously flawed methodology. There are many problems with the 5-star methodology, but I know I only have your attention for a few minutes so let’s call out three:

Forcing a bell curve
Bastardizing standardized ratios
Inept measure selection

Bell curve
We discussed the multitude of problems the bell curve creates several weeks ago, but it bears repeating. By forcing every dialysis clinic into a normal distribution, the program does not create the “rating” system Patrick Conway envisioned, but instead it creates a “ranking” system. It assumes that quality is a zero sum game, that there will be as many winners as there are losers. The misleading nature of this single flaw is mind boggling. How can anyone involved in health care believe that exactly 10% of providers are failures and 10% are excellent? That another 20% are below average and precisely the same number above average? That exactly 40% are average? Are they taking this path because it’s the easy way out? How can this not result in a program that dangerously misleads patients with ESRD, families of patients, and the public?

Standardized ratios
The misuse of standardized ratios is another major defect within 5-Star. One of the three measurement domains consists of three standardized ratios (mortality, hospitalization, and transfusion). Ignoring for the moment the flawed logic that including these ratios in 5-Star assumes that every death, hospitalization, and transfusion occurs because of the care delivered by the dialysis facility, I will try to explain the defect using an example my colleague Dr Eduardo (Jay-r) Lacson recently shared.

Dialysis Facility Compare appropriately reports the standardized mortality ratio (SMR) using one of three easy to understand categories:

Better than expected
As expected
Worse than expected

The standardized ratios are reported like this because each comes with a confidence interval. Suppose your facility’s Standardized Hospitalization Ratio (SHR) is 1.00 this year and that you experienced 8 patient hospital stays. The confidence interval around that SHR could be 0.66 – 2.00. This means that if your facility experienced between 4 and 12 hospitalizations during the year, we can be 95% confident the number of hospitalizations was “as expected.” Anything less than 4 is better than expected and anything above 12 is worse than expected. This vast range reflects the uncertainty inherently present within standardized ratios. Now suppose instead of 8 hospitalizations, 10 patients receiving treatment in your facility were hospitalized during the year. Your SHR would increase, perhaps to 1.25. Because of the built-in uncertainty in this statistic, your SHR appropriately remains in the “as expected” range. Remarkably, the architects of 5-Star ignore this uncertainty by reporting the standardized ratios as a discrete number (not a range). Effectively they are sweeping the complexity of these ratios right under the rug, burying them from view. Try explaining that to your patients. This fallacy has been repeatedly brought to CMS’ attention, but it has fallen on deaf ears.

Measure selection
Last, but certainly not least, some of the measures that make up 5-Star border on the ridiculous. I alluded to the problems with butchering the standardized ratios above. The much easier target though is hypercalcemia. Honestly, I am not sure where to start with this one. The architects of this program somehow determined that among the multitude of laboratory values we track as we deliver care to this vulnerable patient population, the presence of elevated calcium would be a key component of the program. How important is hypercalcemia to 5-Star? It is more important than their measure for death and more important than their measure for hospitalization. Seriously, I could not make this up if I tried. Think about just how ridiculous this is. Now I know most of the folks who read this blog either practice nephrology or know someone who does, but it is very clear whoever decided to include hypercalcemia has never had the privilege of caring for a patient with renal failure.

Every nephrologist treating patients with ESRD carefully follows three lab values related to bone and mineral metabolism. Ask any one of them which of the three are the most important and none of them would pick calcium. Yet the National Quality Forum has conveniently sanctioned an ESRD-specific calcium measure. So, there you go, let’s include it in 5-Star. Really? Can you imagine the unintended consequences of ranking dialysis facilities using a measure of remarkably limited clinical value? Providers may well begin to “teach to the test” by reducing dialysate calcium or prescribing more Sensipar, not because it’s good for the patient but because it will lower the patient’s calcium. Honestly, I cannot think of a more worthless measure to include. What were these people thinking? I have heard some of them say they did not want “perfect to be the enemy of good.” But this program is about as far from “good” as you can get. It is fundamentally wrong and it will certainly mislead our patients.

Patient deception begins in four months
In spite of the overwhelming criticism from patient groups, nephrologists, and dialysis providers, CMS is charging ahead with 5-Star. They announced a delay of a few months to ‘educate’ patients and try to deal with some of the data integrity issues that have plagued the system, but they fully intend to stick with this flawed methodology and expose the Stars on Dialysis Facility Compare in January. What’s missed by many is this flawed approach not only misleads our patients, it puts every nephrologist on the hot seat, as each of us will at some point be asked to explain our facility’s stars to a variety of audiences.
By the way, dialysis may be first, but the plan is to extend these deceptive rankings to hospitals and home health agencies. And I guarantee you, in the not too distant future Physician Compare will have its own 5-Star “ratings” system.

I just don’t understand how it’s possible that everyone in the renal community can be highlighting the dangers and flawed logic of this program, and yet CMS continues to plunge forward. Is there some warped political motive at play here? Does someone in the agency have an axe to grind? How can a collection of intelligent people screw up this badly? If you have an answer, drop us a comment. I’d truly like to know. In the meantime, get out and beat the bushes. If CMS or Congress is listening, perhaps there is hope that we will wake up in January and this nightmare will be just that … a very bad dream. -by Terry L. Ketcherseid, MD, MBA

Misleading patients: CMS blindly charges ahead with 5-Star

Find more Viewpoints posts