This article is more than 5 years old. Information may no longer be current.
Suboptimal evidence behind most ATS recommendations; ATS responds
Click Here to Manage Email Alerts
Almost 40% of the American Thoracic Society’s recommendations in clinical practice guidelines are designated as strong, but less than one in 10 are supported by high-quality evidence, according to findings published in JAMA Internal Medicine.
“The ATS issues clinical practice guidelines for the care of patients with pulmonary and critical care disease,” Ross C. Schumacher, MD, from the University of Texas Southwestern Medical Center, and colleagues wrote. “The utility of ATS guidelines depends on the quality of the evidence base underpinning recommendations and whether the guidelines permit the practice of evidence-based medicine. However, the extent to which ATS guidelines are substantiated by high-quality evidence and can be used to promote evidence-based medicine is unknown.”
To address this gap in knowledge, Schumacher and colleagues reviewed the ATS clinical guidelines for adults that were listed on their website as of Aug. 1, 2017. The researchers studied each guideline’s recommendation type, recommendation strength (using the Grading of Recommendations Assessment, Development, and Evaluation [GRADE] scoring system), quality of evidence (using GRADE), evidence-based medicine measures and patient context.
Recommendations were defined as including evidence-based medicine measures with at least one measure of test performance for diagnostic recommendations, such as sensitivity, specificity or likelihood ratio, as well as at least one measure of absolute benefit or harm for therapeutic recommendations, such as absolute risk reduction or increase, number needed to treat or harm or relative risk with incidence of the outcome for the control group, according to the researchers.
Patient context was defined as including any discussion of severity of illness or comorbidities, sociopersonal context, prognosis or personal preference and how they may affect the recommendation.
The researchers identified 222 recommendations from 16 guidelines. Of those, more than half (63.5%) were based on low-quality evidence. Only 8.6% of the recommendations were based on high-quality evidence, but 38.7% were labeled as strong recommendations.
Evidence of higher quality was more likely to receive a strong recommendation. About 20% of low-quality evidence recommendations, 66% of moderate-quality evidence recommendations and 84% of high-quality evidence recommendations were strongly recommended. However, 18.6% of strong recommendations were not supported by high-quality evidence, according to the researchers.
Half of the diagnostic testing recommendations reported the test’s sensitivity, specificity or likelihood ratios (26 of 52). Nearly half of the therapeutic recommendations reported the absolute benefits or harms of the treatment (76 of 165).
Overall, any discussion of the patient context was included in 45.5% of recommendations regardless of recommendation strength or quality of evidence. Severity of illness or comorbidities (37.8%) and sociopersonal context (23%) were the most commonly discussed patient context domains, while prognosis (6.3%) and patient preference (1.4%) were the least.
There was considerable variation in number, evidence base and strength of recommendations among the guidelines, according to the researchers.
“The ATS clinical practice guidelines are supported by suboptimal evidence and often are not presented in a manner suitable to optimize care for individual patients,” Schumacher and colleagues concluded.
“Most ATS guidelines should be cautiously applied and should not be considered standard of care, given the paucity of high-quality evidence,” they wrote. “Future randomized clinical trials could lead to reversal of many recommendations, even for tests and therapies that are currently strongly recommended. Further, standardizing poorly substantiated diagnostic tests or treatments can lead to inappropriate and harmful care.”
ATS appreciates the authors’ interest in the organization’s guidelines, Kevin C. Wilson, MD, professor of medicine at the Boston University School of Medicine and chief of documents and patient education at the ATS, told Healio Primary Care Today.
“However, we are disappointed in the quality of their work,” he said. “They missed 16 guidelines. More than one-third of the recommendations analyzed are from guidelines that pre-dated the 2011 [Institute of Medicine] Standards for Trustworthy Guidelines and ATS’s full implementation of the GRADE process; thus, they do not reflect the current era of ATS guidelines. We were surprised that there was no control group to which to compare their findings.”
Most of the criticisms in the study can be explained easily, he said.
“The many recommendations based upon low or very-low quality evidence reflect ATS’ preference that its guidelines address true clinical uncertainties,” Wilson said. “When there is uncertainty, there is often poor evidence; because, if there was good evidence, there would be less uncertainty. The strong recommendations based upon low- or very low-quality evidence were intended to avoid serious patient harm, a situation in which such grades are permissible within the GRADE framework.”
“Finally, evidence-based medicine measures are routinely provided in ATS guidelines when estimable but, due to page limitations, are often relegated to tables within an online supplement,” he added. “Thus, these guidelines reflect the ATS’ commitment to provide guidelines that are fully transparent, answer questions for which there is a clinical need, and avoid patient harm.” – by Alaina Tedesco
Disclosures: The authors of the study report no relevant financial disclosures. Wilson is employed by the ATS.
Perspective
Back to Top
Raed Dweik, MD
The ATS is one of the leading guideline-developing organizations in our field, so the JAMA Internal Medicine article caught my attention. A big deficiency of this study is that there was no control group to indicate how ATS guidelines fare compared with guidelines from other societies. The study also appeared to be dominated by older guidelines, which may not reflect the quality of more recent guidelines.
The results were not unexpected: the large proportion of recommendations based upon low quality evidence likely reflects the fact that questions typically arise in the context of scarce evidence. Pulmonary and critical illnesses are often so severe that strong recommendations may be warranted even when there is not a lot of supportive evidence due to the elevated risk of harm from inaction or inappropriate action.
A better study would have been one that looks at the ATS guidelines published since 2017 and compare them with those of other organizations.
Click Here to Manage Email Alerts