This article is more than 5 years old. Information may no longer be current.
An open letter to the FDA regarding the results of PROWL-1 and PROWL-2
Click Here to Manage Email Alerts
To the Editor:
The long-awaited and highly anticipated results of the U.S. Food and Drug Administration’s studies on quality of life after LASIK were presented recently by Dr. Malvina Eydelman at the American Academy of Ophthalmology meeting in Chicago. PROWL-1 was based on subjective responses to a survey completed by military personnel at 6 months postoperatively. PROWL-2 was based on subjective responses to the same survey provided by civilian subjects at 3 months postoperatively. PROWL-1 utilized wavefront-optimized and wavefront-customized treatments. PROWL-2 utilized conventional treatments in addition to wavefront-optimized and wavefront-customized treatments.
Overall, these studies confirm published reports of the outstanding safety and efficacy of LASIK. Ninety-nine percent of subjects in PROWL-1 and 96% of subjects in PROWL-2 were satisfied with their outcomes. There were, however, many aspects of study design, data reporting and interpretation of results that raise concerns.
R. Doyle Stulting
It is well known that patients undergoing LASIK do not reach maximum benefit from the procedure until more than 3 months have passed since surgery. For example, the recently approved T-CAT procedure with the WaveLight laser (Alcon) demonstrated a significant improvement in patient-reported symptoms at 1 year compared with 3 months after LASIK. The literature also clearly documents that symptoms related to tear deficiency are temporary, typically resolving between 6 and 12 months after surgery.
One must ask, therefore, why the FDA chose to measure outcomes at a time when visual symptoms were known to be maximal and temporary. Furthermore, it is difficult to understand why the protocols for PROWL-1 and PROWL-2 were designed to measure outcomes at different times postoperatively and why conventional treatments were allowed in PROWL-2 but not PROWL-1. These differences in study design would produce predictably better results from PROWL-1 than PROWL-2 and predictably worse results than patients experience long term.
One aspect of the study that might have escaped attention of the audience is the fact that only 83% of enrolled patients for each study submitted questionnaires for final analysis. This relatively high lost-to-follow-up rate would be seriously problematic for a sponsor submitting data to the FDA for product approval, so one must wonder why this deficiency and its potential for bias were not mentioned in the presentation.
Another major concern is the unbalanced, selective reporting of results. For example, ghosting, glare, halos and starbursts were uniformly much less prevalent at 3 months than they were preoperatively. Despite this striking overall improvement in visual symptoms related to LASIK, the FDA chose to present data on the number of subjects developing any new visual symptoms at 3 months compared with preoperatively.
Remarkably, the number of subjects in whom at least one symptom that was present preoperatively but was not present at 3 months was never mentioned or presented. Doing so would have allowed the audience to fairly evaluate the number of subjects developing new symptoms postoperatively by comparing them with the number of subjects in whom at least one symptom disappeared after LASIK. Text on the slide illustrating the data only states, “up to 35% of subjects developed new halos.”
Thus, the FDA selected the most prevalent symptom to highlight, selected the study with the greatest prevalence to quote, expressed the postoperative prevalence of this symptom as the percent of patients who did not have it preoperatively, and failed to present the appropriate balanced analysis of patients whose visual symptoms disappeared after LASIK. This biased analysis will obviously lead to misinterpretation of the data.
The distribution of corneal staining was presented at 3 months. Interestingly, this was not compared with the distribution of corneal staining preoperatively. Text on this slide states that “up to 3% of eyes had staining of Oxford grade 2 or more at 3 months,” again emphasizing undesirable effects of LASIK and failing to report the appropriate comparative data so that the true effect of LASIK could be determined. Thus, the FDA reported staining in a very small number of subjects, failed to compare postoperative staining with preoperative staining, and chose to emphasize postoperative staining, even it was rare and perhaps less prevalent that it was preoperatively.
Furthermore, the FDA failed to reference relevant literature and the fact that many of the signs and symptoms of tear deficiency would disappear in the next 6 months. Rather than conclude that “up to 3% of eyes had staining of Oxford grade 2 or more at 3 months,” the FDA should have concluded that dry eye signs occurred in a very small number of patients; these signs might have been less severe than they were preoperatively; dry eye signs that appear after LASIK are temporary in most subjects; and the PROWL studies were not designed to determine the long-term effects of LASIK. The agency should have presented preoperative data so that postoperative data could be appropriately interpreted.
The presentation focused on the three patients in PROWL-1 and the 10 patients in PROWL-2 who reported dissatisfaction (somewhat, very or completely dissatisfied) with their vision at 3 months. The most common explanation for this finding would be residual refractive error, which can easily be corrected. Appropriate analysis of the dissatisfied patients would include postoperative refractive error and whether symptoms were alleviated with appropriate refractive correction. If the investigators truly desired to design a study that would reflect the performance of LASIK in “real life,” they would have analyzed outcomes after enhancements, which were not allowed in these studies. In addition, the number of patients who were dissatisfied with their vision was many times higher before surgery and was not mentioned in the analysis of dissatisfied patients.
Interestingly, the presentation by Dr. Eydelman failed to mention significant conclusions from PROWL-1 and PROWL-2 that can be found on the FDA website (www.fda.gov/medicaldevices/productsandmedicalprocedures/surgeryandlifesupport/lasik/ucm190291.htm#results, accessed Oct. 15, 2014). Here, the FDA states, “The two initial studies found less than 1 percent of patients experience difficulty performing their usual activities following LASIK surgery. … given our findings from PROWL 1 and PROWL 2, a very large clinical study would be necessary to more accurately estimate the prevalence and find useful predictors for these patients, because there are so few. The FDA is unable to conduct a study of such size.” In other words, the FDA on its website concludes that the undesirable outcomes of LASIK are so rare that an unaffordably large study would be required to study them, yet in its AAO presentation, it had no problem reaching a final conclusion that “dissatisfaction and disabling symptoms may occur in a significant number of patients.”
Finally, one should question whether conclusions about the safety and efficacy of LASIK should even be drawn from the PROWL-2 studies. These studies were evidently designed only “to further validate the newly developed questionnaire in the general population,” according to presentations from Dr. Wilkinson and Dr. Eydelman’s own slides. Remarkably, Dr. Eydelman’s AAO presentation was virtually devoid of discussions of the questionnaire and focused on outcomes of LASIK.
One could certainly understand the rationale for measuring short-term results of LASIK if the purpose of the study was to validate a questionnaire, but it is readily apparent that the FDA presentations went much further. Scientifically and statistically, this type of post-hoc analysis of a study designed for another purpose makes misleading conclusions very likely. We need an explanation of why a study designed to validate a questionnaire has now become a study to determine the impact of LASIK on public health.
In summary, the study was designed to measure outcomes at the worst possible time after LASIK, reflecting visual symptoms that are known to be temporary. It was not designed to determine the long-term effects of LASIK. The analysis of data was clearly scientifically flawed and misleading. The summary and conclusions emphasize undesirable outcomes, many of which are temporary or easily correctable. The overall summary and conclusions appear to reflect bias to make outcomes of LASIK appear to be worse than they actually are. They were also designed to make the results of LASIK in the military appear to be better than the results of LASIK in the civilian population. It appears that the civilian study was designed only to validate a questionnaire, and the FDA is now reporting a post-hoc analysis of the impact of LASIK on public health without a study to support that analysis.
I, as a citizen, trained scientist, NIH grant recipient, career academician with more than 200 peer-reviewed publications and past editor of the journal Cornea, am upset that the FDA and National Institutes of Health invested government resources in executing a flawed study, presented a biased analysis of the data, and predictably will create misinformation and negative publicity about a procedure that offers so much to so many people. I am concerned that the military and our troops (and therefore our country) who benefit so greatly through the visual accuracy and freedom of LASIK may re-examine their position based on a flawed trial and misleading presentation of the data.
I hope that the study investigators in PROWL-2 will insist that the data be reanalyzed in a scientifically and statistically balanced way, deficiencies in study design be emphasized, and the obvious bias be removed from the discussion of the study. It also would be informative to follow the study cohorts longer to determine whether visual complaints are transient or long-lasting.
What was the cause for dissatisfaction in the PROWL study subjects? Can we treat them so they will be satisfied with their outcomes? Could we have identified them preoperatively and avoided surgery on them? These are the relevant questions that would further the FDA’s mission of enhancing public health.
Finally, one must wonder how the FDA will handle the obvious conflict created by the conclusions reached in its study. Why does the military branch of our government feel that it should prepare our soldiers for battle with a procedure when another branch of our government, the FDA, believes that the same procedure results in “dissatisfaction and disabling symptoms … in a significant number of patients?” If LASIK is good enough for young men and women who risk their lives for our country, why is it such a public health hazard for our citizens?
What should be done to rectify these serious flaws and scientific errors in the presentation of the study results?
First of all, the only data presented and discussed should be those relating to the stated purpose of the study: development and validation of a questionnaire to assess subjective outcomes of LASIK.
Second, any outcomes of surgery that are presented should be analyzed in an unbiased and scientifically balanced way. If the incidence of new symptoms is presented, the incidence of symptoms that disappeared should also be presented, for example.
Third, it should be clearly stated that many undesirable LASIK outcomes at 3 and 6 months are typically temporary. Any conclusions should be qualified with this disclaimer. If the FDA wishes to study long-term results of LASIK, it should extend the study to administer the questionnaire at 1 year after LASIK or afterward.
Fourth, the FDA should compare the outcomes of LASIK with the alternative: contact lens wear. These data can be obtained from the questionnaires administered preoperatively or from published data.
Fifth, the FDA needs to gather useful data from the individuals they have identified as being dissatisfied with their LASIK outcomes. Were they unhappy only because of residual refractive error? Were they unhappy because of symptoms of tear deficiency that will resolve with time? Were they more unhappy with their preferred method of correction before undergoing LASIK than they were after they underwent LASIK? Could they have been identified preoperatively so that an undesirable outcome could have been avoided?
Finally, all future presentations and publications should be scientifically valid and free of bias. The FDA website and all other modes of communicating the results of this clinical trial should be modified to reflect the comments and recommendations above.
R. Doyle Stulting, MD, PhD
Atlanta
Editor’s notes:
The FDA has declined the opportunity to respond.
Read OSN’s Jan. 25 cover story “Patients report high satisfaction rate in FDA LASIK outcomes questionnaire” for more on this topic.