Analysis: Pooled-cohort risk equation overestimates CV risk in real-world cohort

Donald M. Lloyd-Jones, MD, ScM, FACC, FAHA

I believe in the pooled-cohort risk equation, and it is doing what we asked it to do. There has been controversy from day 1 about its performance in different types of populations. This is exactly what should happen. Any time we use anything in a guideline based on science, people should figure out what its strengths and weaknesses might be. This is certainly not the first nor will it be the last paper to evaluate the risk assessment tool in varied populations. This is the way we move science forward. I’m quite certain that this will evolve in the next set of guidelines, as it should.

That said, not all science is equal. There are some very significant methodological flaws in this paper that preordained what was found. It’s important to understand the decisions the researchers made in designing this study that led to the results. If the question is how well the risk score works in the population for whom it’s intended — patients potentially at risk for MI or stroke — they did almost everything they could to remove all the people actually at risk for MI or stroke in the study design. For example, Figure 1 in the paper shows that they excluded anyone who received lipid-lowering therapy in the prior 5 years. Those are people whose doctor thought they might be at risk for MI or stroke. That’s a huge chunk of people in the Kaiser system, because they do a very good job of identifying risk in their patients. Then, there were more exclusion criteria, effectively removing all the people likely to be at higher risk for MI or stroke. For example, people treated with a statin during follow-up; they’re treated with a statin because somebody thinks they’re at risk for MI or stroke in the future. By removing those people, you’re taking away the natural history that the risk score is supposed to predict.

There are two glaring pieces of evidence in the paper telling that this is exactly what happened. The first evidence of selection bias is that the population had a prevalence of diabetes of 1.4%. In the United States, the prevalence of diabetes is 9% to 10%. That’s important because diabetics are at very high risk for MI or stroke. If you’ve removed the vast majority of diabetics, of course the risk score is going to overpredict. It’s supposed to predict on a representative population. The second piece of evidence is that the researchers report an overall event rate of 1.4 CV events per 1,000 person-years. That event rate is a historically low, never-before-seen event rate for any U.S. population. Maybe it’s true, but several factors suggest it is a very biased sample. In the REGARDS cohort, which represents white and black men and women across 48 states and where the risk score performed very well, the event rate was 7 to 8 per 1,000 person-years, five to six times higher than seen in the present study cohort. The mismatch between event rates is striking. Also odd is that this group has reported that the event rate for just MI during this time period is 2.1 per 1,000 person-years. The event rate for all CVD is lower than their previously published event rate for just MI; that math doesn’t work at all. I’m puzzled about how to reconcile those two things.

All that said, it would be expected that the risk score would overpredict given the exclusion criteria. These patients exist in the real world, but we’ve lost sight of what the question was. When you skew the sample, as the researchers have done pretty substantially, then you see results like this. This analysis is one piece of information. It is certainly not the final word. The overall pattern that we see across all the papers that have been published is that, in some cases, there is a mismatch where the risk score overpredicts, and that tends to be in populations with people who are very healthy or very engaged with the health system, where we would expect that they are toward the lower end of the risk spectrum. We see just as many papers about populations of patients with conditions like HIV or rheumatoid arthritis who are at higher risk for CVD and in whom the risk score underpredicts. The more representative the sample is of the kinds of patients presenting to primary care offices, without weeding out large swaths of them, the much better the risk score performs.

The 2013 ACC/AHA Guideline on the Treatment of Blood Cholesterol to Reduce Atherosclerotic Cardiovascular Risk in Adults says to use the pooled-cohort equation as a starting point for a conversation, and then bring your patient’s individual characteristics to the table to help you make a decision. The risk score puts you in the ballpark to understand the patient’s risk, but then you have to consider whether they are lean, active, eating a good diet, heavily engaged with the health system, with no family history of CVD, etc. If they meet those criteria, then you might consider them at the lower end of the risk spectrum and might not be as aggressive in prevention. Whereas if someone has a long history or overweight/obesity, a family history of CVD, heavy smoking, etc, I would expect the risk score is actually underestimating risk. That’s why you can’t think about the risk score in isolation. It has to be used in the way intended by the cholesterol guidelines. That’s medicine.