This article is more than 5 years old. Information may no longer be current.
Selective reporting of primary, secondary endpoints common in randomized oncology trials
Click Here to Manage Email Alerts
The selective reporting of outcomes has persisted in published reports of randomized clinical trials in oncology, despite public and reviewer access to trial protocols, according to the results of a meta-analysis.
Additional initiatives are needed to minimize selective reporting and to foster sound, evidence-based medicine, according to the researchers.
Michael J. Overman
“Randomized clinical trials represent the highest level of evidence and drive clinical care,” Michael J. Overman, MD, associate professor in the department of gastrointestinal medical oncology at The University of Texas MD Anderson Cancer Center, told HemOnc Today. “As such, it is imperative that the reporting of such trials is clear and concise, without the influences of bias or spin. Utilizing recent efforts by major journals to make available protocols to randomized studies has enabled us to compare the study reports to the true source document that drove a trial: the study protocol.”
Overman and colleagues conducted a meta-analysis of 74 oncology-based randomized trials published in Journal of Clinical Oncology (n = 55), The New England Journal of Medicine (n = 17) and The Lancet (n = 2) between March 1, 2012, and Dec. 31, 2012.
To determine the reliability of the reporting, the researchers compared the published reports with their respective protocols with regard to primary endpoints, non-primary endpoints and unplanned analyses,
The 74 randomized trials included in the analysis reported a total of 86 primary endpoints, with nine trials reporting more than one primary endpoint.
Nine trials (12.2%) had some discrepancy between their planned and published primary endpoints.
Sources of discordance between planned and published primary endpoints included the addition of a new primary endpoint (n = 1), failure to report on a primary endpoint (n = 1), a change in the reporting of a planned primary endpoint (n = 3) and change in the terminology of a planned primary endpoint (n = 4). Discrepancies in reporting occurred more frequently in trials initiated prior to 2005 (P = .03).
“Researchers conducting clinical trials need to be clear about the prespecfied questions that a study was designed to answer, as well as those questions that may have been raised and subsequently explored after the conduct of the study,” Overman said. “My hope is to have all endpoints reported in an appendix to randomized studies. This will help to minimize the selective reporting of only those endpoints that are positive, and provide a full representation of the data.”
The trials included in the study listed a total of 579 non-primary endpoints in their protocols (median per trial, 7), of which 64.4% (n = 373; median per trial, 5) were reported. Nineteen studies (25.7%) reported all of their planned non-primary endpoints.
The most common unreported non-primary endpoints included biomarkers (19.4%), quality-of-life measures (17.4%) and time-to-event endpoints (16%).
The researchers observed a significant positive correlation between the number of planned and the number of reported non–primary endpoints (Spearman r = 0.66). Trials with more than six planned non-primary endpoints had more nonreported non-primary endpoints than trials with six or fewer planned non-primary endpoints (P < .001).
Twenty-eight studies (37.8%) reported a total of 65 unplanned endpoints, 80% of which the study researchers did not identify as unplanned.
Further, 41.9% (n = 31) of trials reported a total of 52 unplanned analyses regarding primary endpoints and 25.7% (n = 19) of trials reported a total of 33 unplanned analyses involving non-primary endpoints. The researchers found that studies reported positive unplanned endpoints and unplanned analyses more frequently than negative outcomes in abstracts (unplanned endpoints, OR = 6.8; P = .002; unplanned analyses, OR = 8.4; P = .007).
“I hope this report will help create transparency in reporting and make sure that we are labeling endpoints and analyses that are preplanned and those that are post-hoc and exploratory in nature,” Overman said. “Both are extremely useful to understanding the data generated from clinical trials, but post-hoc and exploratory endpoints and analyses are more prone to bias and issues of multiplicity and thus need to be clearly labeled.” – by Cameron Kelsall
For more information:
Michael J. Overman, MD, can be reached at The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 426, Houston, TX 77030; email: moverman@mdanderson.org.
Disclosure: Overman reports research funding from Amgen, AstraZeneca, Bristol-Myers Squibb, Celgene, MedImmune and Roche, as well as consultant roles with Roche and Sirtex Medical. Further, he reports that an immediate family member holds stock ownership in Novartis. Please see the full study for a list of all other researchers’ relevant financial disclosures.
Perspective
Back to Top
Vinay Prasad, MD, MPH
Two things are true. Randomized trials are the best way to test whether a medical intervention improves clinical outcomes, providing conclusions far more reliable than nonrandomized evidence, which are riddled with residual confounding and far more vulnerable to alterations in statistical plan (also called P hacking), and the selective reporting bias. And, randomized trials themselves exist along a hierarchy with rigor determined by the statistical power of the study, the number of participating centers, the use of blinding, the choice of endpoints, the choice of controls, inclusion and exclusion criteria, presence of crossover and restrictions on the use of medications outside of the protocol. In short, even though randomized controlled trials (RCTs) are our best tool to assess medical practices, simply being an RCT is not enough.
Raghav and colleagues examine some of the ways RCTs can provide biased results. Researchers of RCTs may not report their primary outcomes or may change them, and researchers of RCTs may not report all of the endpoints that they set out to gather. Changing your primary outcome is an unforgivable sin, as trials are typically designed and powered to answer a specific question about the primary outcome, with all other analyses provisional and hypothesis-generating at best. Not reporting every outcome you gathered is also bad — but, unfortunately, it is not surprising. Modern clinical trials collect many secondary endpoints, and, as Raqhav and colleagues show, the more you gather, the more likely you are to not report some. Efforts to ensure the timely reporting of all outcomes are welcome.
Raghav and colleagues looked at 74 RCTs in three top journals that provided trial protocols — The Lancet, The New England Journal of Medicine and Journal of Clinical Oncology. But, with 55 (74%) studies coming from Journal of Clinical Oncology and just two from The Lancet — it probably does not do an adequate job of saying anything about trials and protocols generally in cancer, or anything beyond Journal of Clinical Oncology and The New England Journal of Medicine. But, this does not stop the authors from extrapolating, “We believe that our observations about selective reporting are not unique to oncology trials but instead reflect medical reporting in general.” I am not sure why they are so confident, particularly when the results here are different than a Cochrane analysis cited.
The major conclusion of the Raghav study is that nine out 74 trials (12%) have a discrepancy in the primary outcome. They don’t tell us which trials these were, and that is unfortunate because we cannot verify their interpretations. But, three instances involve changing DFS to relapse-free survival, PFS or recurrence-free survival. Frankly, my guess is that this is a semantic change, and not really a different concept. Three cases involve changing time to progression to PFS — which is a different endpoint entirely — one case involved changing a 4-year DFS to 5-year DFS, and two cases involve not reporting OS, or reporting it as primary endpoint when it was not a primary endpoint. In short, six out of 74 trials (8%) changed a primary endpoint in a way that warrants further investigation.
What do the results tell us about other journals? Unfortunately, nothing. Authors and sponsors know that they have to submit protocols along with manuscripts to these journals, and may be less likely to submit discrepant examples. Only a systematic review of all protocols from some collection of companies or investigators, deposited before the start of a trial, against subsequent publications, will be able to tell us just how prevalent the problem is. Clinicaltrials.gov has done a great job in preventing trials from being hidden. It is time to mandate that sponsors and investigators upload trial protocols prior to the start of the study, and journals must routinely check for discrepancies.
Reference:
Prasad V and Berger VW. Mayo Clin Proc. 2015;doi:10.1016/j.mayocp.2015.05.006.
Click Here to Manage Email Alerts