‘Best Hospitals’ ranking for ophthalmology relies on subjective reputational scores

Whether U.S. News & World Report’s annual “Best Hospitals” ranking for ophthalmology as a specialty is a valid barometer or a beauty contest depends on who you ask.

Whereas 12 of the 16 medical specialties are evaluated on structure, process and outcomes for treating the most challenging patients, ophthalmology is one of the four specialties judged by reputation-only (the other three being psychiatry, rehabilitation and rheumatology) by a limited number of board-certified ophthalmologists.

 

Bascom Palmer Eye Institute in Miami extended its winning streak as the No. 1 ranked eye institute in the U.S. to 11 straight years; it has been ranked either No. 1 or No. 2 since the ratings’ inception in 1990.

“Our high rating speaks to what Bascom Palmer has always strived to do, which is to provide the best patient care possible in ophthalmology and provide the best education,” institute chairman Eduardo C. Alfonso, MD, said. “The ranking also reflects the institution’s continuing commitment to cutting-edge research.”

Being judged on reputation-only compels surveyed ophthalmologists “to digest information” that they read on outcomes, quality, patient safety care, and the type of education and research offered by a hospital, Alfonso said, with the ultimate result that such professionals would know best where to send a patient for treatment.

However, the likely most important ramification of Bascom Palmer’s ranking “is a sense of pride in coming to work every day and becoming the best we can be,” Alfonso said.

A significant increase in philanthropic contributions by donors is a direct result of the high ranking as well.

“These donors want to support programs that have been shown to be successful, and [this] continues to help us attract the best students in the country into our residency program,” Alfonso said. “Patients also look at these rankings for the best place for eye care.”

 

By excluding objective clinical or outcomes data, some question the validity of the rankings, despite favorable publicity for highly rated institutions. Overall, Alfonso believes the rankings are fair for ophthalmology.

“Currently, we do not have any other ways of developing metrics to judge ophthalmology care,” he said. “Eventually, though, I think there will be some objective criteria that can be statistically validated. But right now, some of the objective criteria is self-reporting, which becomes problematic because you only report things that make you look good.”

Ranked by reputation

No. 2 on the list is Wills Eye Hospital in Philadelphia, which, after holding the No. 3 spot since the survey began, moved up to No. 2 last year.

“We are proud of the fact that our peers reputationally score us at the very top and believe that we deserve to be there,” Julia A. Haller, MD, ophthalmologist-in-chief and OSN Retina/Vitreous Board Member, said. “We work every day to give what we believe is the No. 1 best eye care in the country, but we are not doing anything differently to achieve No. 1 status.”

 

When Haller started at Wills Eye Hospital 7 years ago, she was somewhat surprised by the “old-fashioned, slightly gentlemanly approach” to promotion. “The hospital was extremely modest, with very little advertising. This is a cultural characteristic that I’ve grown to become very fond of,” she said. “The hospital is very anti-commercial. If we are getting the word out in any way, it is what we do academically through our publications, our research and educational initiative.”

Haller believes that rankings based on reputation-only are a “complicated issue,” and even though objective measures are preferable, knowing the right measures to choose is difficult.

“Ideally, we as ophthalmologists would assign value to the outcomes we judged clinically meaningful,” she said.

She acknowledged, though, that some objective scores are hard to quantify, such as complicated cases that require multiple subspecialty interactions and teamwork.

“A vagary of ophthalmology is that we are such an outpatient-based specialty,” Haller said, whereas objective measures for most hospitals where business is primarily inpatient are more easily determined. “Many of those publicly reported figures are just not available to us, at least right now.”

‘Beauty contest’

Apart from the 12 hospitals ranked in the field of ophthalmology (Table), 23 other hospitals received nominations from one or more physicians, including the UPMC Eye Center, which received a 3.6% reputation score.

“I believe there are many things within ophthalmology that are quantitative and important and could and should be included in the assessment of ophthalmology programs or ophthalmology departments,” OSN Glaucoma Board Member Joel S. Schuman, MD, FACS, director of the UPMC Eye Center, said. “Being rated on reputation-only is essentially a beauty contest.”

 

Schuman also said that reputation tends to lag reality: “There are doctors who end up on best doctor lists who have not been practicing for years. But people know their name, so they end up on the list.” Conversely, it takes time for an unknown doctor to become noticed for increasingly praiseworthy accomplishments.

“[Even so], these rankings are important to our hospital, not only for the ‘feel good’ pride aspect but also for dollars in the door from patients,” Schuman said. “I do not think peer-reviewed grant research dollars are significantly affected by the rating; however, the rating definitively helps attract patients. Patients have very few ways of knowing whether a doctor is good or bad.” Schuman also said that UPMC’s administration “puts a lot of weight on the rankings,” including heavily marketing the distinction.

 

Schuman was a co-author of a 2013 Journal of Academic Ophthalmology article on the U.S. News & World Report criteria for evaluating the ranking of clinical ophthalmology. The article stated that the methodology is flawed because it excludes objective clinical or outcomes data. The “correlation of the current reputational process scores to objective measure scores is not significantly different from zero, indicating that no relationship exists between them,” the authors said in the article.

“The reputation-only evaluation does not necessarily reflect the quality of the department, whereas objective parameters that are important may actually represent the quality of the department,” Schuman said.

Bigger ranks better

New England Eye Center at Tufts Medical Center in Boston received a nomination and a 0.1% reputation score.

“I have no idea what that score means,” OSN Retina/Vitreous Board Member Jay S. Duker, MD, director at the New England Eye Center, said. “This type of ranking system is based quite a bit on previous reputation over many years and based on the number of graduates of the institution. Therefore, large programs that have been around for many years will naturally have an advantage because they have more alumni voting who are liable to vote for the place where they trained. Still, I think the rankings are of interest and certainly measure something. But we need to put the rankings in the context for what they are, which is a beauty contest.”

Even so, it is the nature of physicians, and ophthalmologists specifically, to be competitive, Duker said.

“Hence, from a personal level, I think we put a lot of credence into these rankings,” Duker said. “But on a more global scale, I am not sure that the rankings of institutions in surveys such as the U.S. News & World Report make much of a difference because our patients do not come to a specific doctor or go to a specific institution based on these rankings.”

In the real practice of ophthalmology, such ranking systems are not a truly critical way for patients and payers to make distinctions between doctors or institutions, Duker said. For example, payers such as Medicare and private insurance do not use these rankings as a differential for how certain physicians are reimbursed. In addition, Duker estimated that 99% of the staff and patients at the New England Eye Center are unaware of their ranking, so the survey rarely helps attract staff, research money or patients.

“Everyone would agree that a survey of clinical outcomes would be much more beneficial. The problem is that those outcomes are really hard to measure, and we do not have a way to measure them,” Duker said.

Newcomer to top 10

The yearly rankings, which were released in July, had a new entrant in the top 10: the USC Eye Institute at the University of Southern California, which debuted at No. 9.

“It is testimony to our great faculty and leadership,” OSN Retina/Vitreous Section Editor Carmen A. Puliafito, MD, MBA, Dean at the Keck School of Medicine of USC, said. Puliafito was chair at Bascom Palmer in 2004, when that institution first ascended to its top-ranked position.

 

“Although not perfect, the rankings do reflect the opinions of U.S. ophthalmologists about which programs have been academic leaders,” Puliafito said. “However, this is a very imperfect instrument. In fact, of the 200 ophthalmologists surveyed by U.S. News & World Report, only about 60 actually replied. With such a small sample size, the results do have to be taken with a grain of salt.”

Puliafito also noted that the rankings tend to reflect reputations gained over many decades “and may not accurately reflect how good an institution is at this time.” In addition, “many distinguished smaller departments, with fantastic regional reputations, are routinely ignored by the survey,” he said. Moreover, the survey fails to include “the outstanding private eye institutes which provide eye care to most Americans. Perhaps we need an independent, ophthalmology-specific survey.”

Puliafito believes the standings have “tremendous impact” for hospitals such as Bascom Palmer and USC.

“The ranking of your entire institution goes up,” he said. “It affects the way patients select care. It also affects your ability to get top residents. It is not just about the facilities — it’s about the people you attract, and you can attract some unbelievably talented people.”

The downside, though, is that the survey asks ophthalmologists where they would personally go for treatment if afflicted with a serious eye problem.

“In my opinion, this should be a regional survey,” Puliafito said. “What does an ophthalmologist in Bangor, Maine, know about care at Bascom Palmer in Miami?”

Impact of the ranking

Alfonso said that having a No. 1 ranking can help attract staff, research money and patients through ophthalmic referrals.

“Whenever you have a winning team, everyone rallies around that team. This is human nature,” he said. “As we continue to strive to be the best, we attract the best. This helps us to continue to fuel ourselves with energy.”

As the U.S. shifts to value-based purchasing of health care services, quality, safety and outcomes are important measures that hospitals need to have in order to continue to provide the best health care, Alfonso said.

“A hospital will be impacted positively if it is ranked highly in these areas.” Likewise, a hospital ranked poorly “will either need to improve or it will start suffering financially,” he said.

Bascom Palmer is not content to rest on its laurels.

“We continue to improve in patient care, education and research,” Alfonso said. The hospital is also committed to communicating to others its ongoing efforts, whether it is through media such as Ocular Surgery News or its own in-house communication department, which produces a number of publications throughout the year. Furthermore, faculty is sought for national and international ophthalmic meetings to educate and share new research findings. Medicare and private insurance are also more likely to refer patients to Bascom Palmer because of the high ranking, he said.

Haller said Wills Eye Hospital is humble about the ranking.

“It’s a tremendous accolade — a vote of confidence in our faculty and the superb quality of the care delivered at Wills Eye day in and day out,” she said. “We know we need to earn the respect of our colleagues and the trust and confidence of our patients — that’s what drives us.”

Benefactors and philanthropists are also likely to increase hospital funding due to the hospital’s high ranking.

“Rankings do impact on an institution,” Haller said. “Many of our constituency like to see that there is some sort of outside affirmation of excellence, and the U.S. News & World Report rankings are well-recognized, not only by patients and staff, but also by philanthropists and payers.”

Haller has some questions about rankings being used ubiquitously in some fields.

“For medical schools, the grade point average of their entering class is now very much a part of the ranking,” she said. “However, I would argue that GPA in and of itself is not necessarily a measure of who is going to be the top and most empathetic physician.”

As for attracting staff, “I think the sophisticated professional staff looks very hard at the reality to judge its correlation not only with the ranking but much more,” Haller said. “Atmosphere, teamwork, a culture of excellence and esprit de corps, for example, are much more important and unquantifiable qualities.”

Schuman also said that a hospital’s ranking has little influence on potential faculty hires, whether for clinical or basic science positions.

“But I do think rankings may have some influence as to residency applications,” he said.

Hospitals also use the rankings to further improve their reputation, according to Schuman.

“We are very aggressive in trying to get information out about what is happening in our department of ophthalmology,” he said, including sending out letters to fellow ophthalmologists around the country. “We try to overcome that merely word-of-mouth reputation.”

UPMC encourages faculty to speak at regional, national and international meetings, so they can present their research results. “However, I do not think we would do anything differently if there were not a ranking. We would still be trying to get the word out about new science, new treatments and new cures,” he said.

Schuman said it is human nature to want to know which hospital or doctor is the best.

“Hopefully, in the future, those decisions can be made based on objective quantitative measures,” he said. Schuman is optimistic that he and his colleagues will be able to discuss the proposals outlined in their article with the ranking agencies and have those valid criteria included in future rankings.

Haller is also hopeful that the U.S. News & World Report survey will continue to evolve to reflect all aspects of the ophthalmic practice, “which is very attentive to outcomes.” – by Bob Kronemyer

References:

Olmsted MG, et al. Methodology: U.S. News & World Report Best Hospitals 2014-15. www.usnews.com/pubfiles/BH_2014_Methodology_Report_Final_Jul14.pdf. Published July 14, 2014.
Scherer WJ, et al. Journal of Academic Ophthalmology. 2013;6(1):46-56.
Top-ranked hospitals for ophthalmology. U.S. News & World Report. health.usnews.com/best-hospitals/rankings/ophthalmology.

For more information:

Eduardo C. Alfonso, MD, can be reached at Bascom Palmer Eye Institute, University of Miami Health System, 900 NW 17 St., Miami, FL 33136; 305-326-6303; email: ealfonso@med.miami.edu.
Jay S. Duker, MD, can be reached at 800 Washington St., Box 450, Boston, MA 02111; 617-636-4677; email: jduker@tuftsmedicalcenter.org.
Julia A. Haller, MD, can be reached at Wills Eye Hospital, Suite 1510, Philadelphia, PA 19107; 215-928-3073; email: jhaller@willseye.org.
Carmen A. Puliafito, MD, MBA, can be reached at Keck School of Medicine at the University of Southern California, 1975 Zonal Ave., KAM 500, Los Angeles, CA 90089-9034; 323-442-1900; email: cpuliafito@usc.edu.
Joel S. Schuman, MD, FACS, can be reached at UPMC Eye Center, 203 Lothrop St., Suite 816, Pittsburgh, PA 15213; 412-647-2205; email: schumanjs@upmc.edu.

Disclosure: No products or companies that would require financial disclosure are mentioned in this article.

Ideally, what criteria should be included in ranking ophthalmology institutions?

Reputation-only method can be improved

 

U.S. News & World Report has been ranking hospitals since 1990. For 12 of the 16 ranked specialties, publicly reported and rigorously studied hospital quality metrics in four weighted areas (outcomes, structure, process and patient safety) are combined to help patients identify unusually skilled inpatient care. Ophthalmology is one of four specialties in which measures of structure and outcome are lacking because procedures are performed on an outpatient basis and pose a low risk of death.

Applicable metrics for accessing the quality and safety of eye centers and institutes are still years away. In the meantime, the reputation-only ranking can be improved in a way that benefits everyone. The stated goal of the annual rankings is “to identify the best medical centers for the most difficult patients — those whose illnesses pose unusual challenges because of underlying conditions, procedure difficulty or other medical issues that add risk.” Despite hundreds of excellent and eligible eye institutes in the U.S., current survey rules resulted in a ranking for only 12. While these are the best by reputation in the U.S., the public would benefit from a full listing of the top 50.

The new collaboration of U.S. News with Doximity has now increased the potential pool of available board-certified survey respondents. Eye institutes could now be ranked based on reputational scores in designated niche areas such as corneal transplant, uveitis, ocular genetics or pediatric cataract surgery. The number of national and regional U.S. News Top Doctors could also be used as a metric. These data would be valuable for patients and could be summed to arrive at an overall ranking.

Hospital CEOs and medical school deans use U.S. News & World Report rankings as a metric for resources and marketing. When most ranked specialties are listed from 1 to 50, why should the ophthalmology list be limited to 12?

M. Edward Wilson, MD, is an OSN Pediatrics/Strabismus Board Member. Disclosure: Wilson has no relevant financial disclosures.

Data, not reputation alone, should determine ranking

 

Reputation is too subjective and seems to me to be a poor way of measuring anything.

First, the report’s purpose needs to be defined. If the purpose is for patients to find places that offer the best care for patients, then we should look at surgical volume and academic credentials. What are the academic credentials of the physicians? Are they publishing research? Does the institution have residents? Does it have fellows? What kind of equipment does it have? All of those things are going to impact whether that site is up-to-date, experienced and knowledgeable.

It is clear that if you want to rate a program, you ought to use some hard data or metrics, not just reputation alone.

For example, some universities are known by reputation to be “good schools,” and if one of them was mentioned, I might say, “That’s a good school.” And it likely is a good school. But that judgment, besides being subjective, says little objectively about a particular problem or patient and ignores lesser-known institutions we may not be all that familiar with. We tend to repeat the same thing over and over — mentioning the well-known places and perhaps forgetting less well-known places with excellence in particular problems.

So, I think using reputation is a poor way of rating anything. There are just too many sources of bias and uncertain evaluation. For ophthalmology or any of the other programs that are only rated by reputation, it does not mean anything by it, reputation does not tell us enough.

Penny A. Asbell, MD, FACS, MBA, is OSN Contact Lenses Section Editor. Disclosure: Asbell has no relevant financial disclosures.