Reproduction of published research results remains a challenge for physicians

Issue: June 2012

ByDouglas W. Jackson, MD

In recent months, there has been more public attention to the realization that many published studies have results that cannot be reproduced. This, along with how financial conflicts of interest can also potentially serve to impact study results, has the public and discerning readers at the crossroads of science and public policymaking.

The December 2011 issue of Science featured the topic of scientific reproducibility and replicability in detail. Jasny and colleagues wrote in the introduction, “The confirmation of results and conclusions from one study obtained independently in another is considered the scientific gold standard.” They continued, “new tools and technologies, massive amounts of data, long-term studies, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts, as are increased pressures on scientists to advance their research.”

Reproducible research infers that the published results can be duplicated in other laboratories and clinical settings. The question is what accounts for a published study when data and conclusions cannot be reproduced. One has to ask if the study is fraudulent or the results are from sloppy work or a selective presentation of data.

Possible dangers

Biomedical research, which has been estimated to be a $100 billion a year worldwide business, involves government funding, private industry, large institutions and multitudes of investigators. In his farewell presidential address in 1961, President Dwight D. Eisenhower warned of the risks of the emerging industrial-military complex that controlled large sums of money and numbers of people. He warned about the dominance of the federal government funding universities and their research laboratories. Increasing dependence for federal funds, he said, had the potential to compromise researchers’ search for truth as well as reduce the “scholar” to an “obedient employee.” He theorized that the need to obtain funding would necessitate researchers to obtain positive results, as well as please superiors with publication in recognized journals. Determinations for further funding, he said, could become highly political and based on solely grant reviewers’ decisions.

Douglas W. Jackson

On such a large scale, no one knows with certainty what percentage of published medical findings cannot be reproduced and validated. The orthopedic research system currently rewards positive results, particularly studies that enhance orthopedic surgeons’ approaches to patient care. There is little motivation or incentive to perform negative studies or studies that duplicate previously published results. These types of studies are much harder to publish and are most often performed by the institution when it wants to repeat or expand their own initial work.

Scope is unknown

It can be argued that more irreproducible results have been published than we would like to acknowledge. Reproducibility of published work is only occasionally challenged by knowledgeable scientists and clinicians who work in the same field or by industry when it tries to duplicate results before proceeding with product development. This problem is accepted to the point that the most successful venture capitalists have learned to reproduce results by independent observers before they commit to early-stage funding.

Outside of orthopedics, the pharmaceutical company Bayer recently reported an example of this problem. In September 2011, Bayer published an evaluation of 67 published studies in which they failed to duplicate two-thirds of the results with their in-house experiments.

As an orthopedic surgeon reading this commentary, you may say that this problem is primarily related to basic laboratory results with different reagents, equipment, personnel and the complexity of the studies. However, I believe reproducibility is a more insidious and pervasive problem than we realize, and it impacts our biomedical literature.

Variables

Many reported studies have intrinsic and multidisciplinary variables dependent on the interpretations of data, results and outcomes. Many orthopedic studies do not – and some cannot – involve investigators blinded to treatment. In addition, many orthopedic studies have small numbers and inadequate controls with short-term follow-up results. This allows authors to be more subjective and present their findings in a manner that supports positive results. For example, the published results and various studies on viscosupplementation and glucosamine remain controversial. In addition, there are many short-term device studies, as well as some with 10-year and longer results, in which it would be difficult to reproduce the high percentage of good-to-excellent results previously found.

A large number of orthopedic investigators need to produce successful studies and publications under their name. In addition, there are more scientific and professional journals to publish their work. It is my opinion that many large and small established orthopedic research facilities do not use stringent reviews of their researchers’ work after tenure is obtained. Many investigators, who were in the forefront early in their careers, have not kept up academically and technically, but are retained even though they are not doing reproducible work.

The complexity and multidisciplinary aspect of good research has significantly changed. As readers of published studies, physicians are dependent on research that is touted as reproducible. Studies are retracted on the basis of fraudulent or poor work, but physicians are rarely notified. This area of science deserves more open discussion, oversight and a mechanism to remove irreproducible studies and researchers.

Reference:

Jasny BR, Chin G, Chong L, Vignieri S. Again, and again, and again. Science. 2011; 334(6060): 1225. doi: 10.1126/science.334.6060.1225.

For more information:

Douglas W. Jackson, MD, is Chief Medical Editor of Orthopedics Today. He can be reached at Orthopedics Today, 6900 Grove Rd., Thorofare, NJ 08086; email: orthopedics@healio.com.