This article is more than 5 years old. Information may no longer be current.
‘Exaggerated’ early clinical trial results may create false hope
Click Here to Manage Email Alerts
Researchers at Mayo Clinic identified an “exaggerated treatment effect” in more than one-third of early-phase clinical trials designed to evaluate treatments and devices for cancer and other chronic conditions.
“This phenomenon of exaggerated early results was present in a whopping 37% of the studies we reviewed,” Fares Alahdab, MD, research fellow in Mayo Clinic’s Evidence-Based Practice Center, said in a press release. “Physicians and patients should be cautious about new or early clinical trial evidence.”
Nearly 51% of the U.S. adult population has one or more chronic conditions.
Encouraging results from some early-phase clinical trials designed to assess new treatments for these conditions have been found to fluctuate or moderate in later-stage trials.
Consequently, clinical and policy decisions based on what researchers have dubbed the Proteus effect — early trials that show significant or opposing results from subsequent trials — may be misguided and may be based upon incorrect estimates of benefits and harms, according to Alahdab and colleagues.
The researchers assessed whether early trials in chronic medical conditions show an effect size greater than that observed in subsequent trials.
Investigators pooled data from 70 meta-analyses published between Jan. 1, 2007, and June 3, 2015. The meta-analysis encompassed 930 clinical trials, and researchers focused their attention on those that evaluated drugs or devices for the treatment of chronic conditions, such as cancer, stroke and diabetes.
The prevalence of exaggerated early effect — or the proportion of meta-analyses with the largest effect or heterogeneity in the first two trials — was 37%. Early trials had an effect size that was, on average, more than twice as large as the overall pooled effect size (ratio of relative effects = 2.67; 95% CI, 2.12-3.37).
The presence of an exaggerated effect did not appear significantly associated with variables such as trial size, length of follow-up, number of events, number of study sites, outpatient vs. inpatient setting, or early trial cessation.
HemOnc Today spoke with Alahdab about the study findings, how physicians should communicate with patients to ensure realistic expectations, and the next steps for research into this topic.
Question: How did this study come about?
Answer: We always build upon the research of previous studies. This finding has been noted for quite some time; however, it was not clear how much or why it was happening. Our goal with this analysis was to answer these questions. Methodology research is at the heart of what we do.
Q: How did you conduct the study?
A: We started by searching the top 10 medical journals, including The New England Journal of Medicine, JAMA and others. We identified about 2,700 systematic reviews and meta-analyses but ended up using only about 70 of these. We chose meta-analyses because they capture a large body of evidence around a certain topic plotted over time. We then explored in detail each of the 930 randomized controlled trials included in the meta-analyses to look for all types of factors that we thought might help explain this phenomenon — for example, sample size, publication bias, risk for bias, and multicenter vs. single-center studies.
Q: What did you find?
A: We found this phenomenon present in more than one-third of the body of literature. Among 37% of studies published around a certain topic, the first two studies showed the largest effect size or the largest heterogeneity. Evidence that came later provided a lot more knowledge, certainty and precision about the effects of these interventions. We also found that the factors we evaluated could not explain this phenomenon. More research is needed to understand how this is occurring and where it is stemming from.
Q: What are the clinical implications of the findings?
A: Physicians who care for patients with these chronic conditions should be quite cautious when relying on evidence from early-phase clinical trials of novel drugs and treatments because the results can be exaggerated. We recommend looking at full bodies of evidence rather than single studies when trying to decide whether to implement a therapeutic intervention. We strongly recommend a shared decision-making process in which the unique situation of every patient, as well as the best available evidence, is taken into consideration.
Q: Can you talk about the Proteus effect?
A: The Proteus effect is when early studies about a new medication or intervention show extreme or opposing results compared with later studies. The analogy comes from Proteus, the Greek god of the sea. According to mythology, Proteus appeared and disappeared unexpectedly, evading being seen.
Q: Can you provide some practical advice for how clinicians can communicate with patients about reasonable expectations of trial results?
A: Explaining to patients what the medical literature actually says is important because this can be quite technical and challenging to many. On top of the complicated statistics, there is this phenomenon that is not clearly explained yet. Uncertainty in the evidence relevant to the medical management of patients should always be explained to them. We can explain that certainty in medical research changes over time. When we have a body of evidence that can be synthesized by summarizing a whole body of literature, this provides a more certain estimate to recommend an intervention rather than relying on a single study. In a situation in which such summary of the body of evidence is not available, or when the body of evidence itself has not developed yet, the uncertainty of the benefits or harms of a new therapy should be made clear to the patient. We also recommend and advocate that physicians consider the individual patient’s situation and preferences.
Q: What’s next for research on this topic?
A: We are trying to better understand the phenomenon, including its implications and origins.
Q: Would you like to mention anything else?
A: The main message is to always be cautious of early clinical trial results about new therapeutic interventions in the care of patients with chronic medical conditions. Relying on a body of evidence, rather than single studies, provides a more robust recommendation. – by Jennifer Southall
Reference:
Alahdab F, et al. Mayo Clin Proc. 2018;doi:10.1016/j.mayocp.2017.10.020.
For more information:
Fares Alahdab, MD, can be reached at Mayo Clinic, 200 1st St. SW, Rochester, MN 55905.
Disclosure: Alahdab reports no relevant financial disclosures.