This article is more than 5 years old. Information may no longer be current.
Costs, benefits of child vision screening depend on predictive criteria of screening tools
Ophthalmologists may not be used to thinking in terms of quality-adjusted life years and other public-health concepts, but these criteria affect the value of vision screening, experts say. Part 3 of a three-part series.
Click Here to Manage Email Alerts
The effectiveness and cost-effectiveness of pediatric vision screening efforts depend on concepts that many ophthalmologists may not be familiar with. But grasping public-health concepts such as quality-adjusted life years and cost-benefit ratios may be crucial to the success and acceptance of universal vision screening, according to experts in the field.
“In public health, vision screening is not a simple issue,” said Maureen G. Maguire, PhD, director of the Center for Preventive Ophthalmology and Biostatistics at the University of Pennsylvania’s Scheie Eye Institute. She said that screening models used in other fields of medicine do not apply directly to pediatric vision screening because “a multitude of factors” complicate the decision-making process.
“Screening for amblyopia is not a life or death matter, so people have very different views on the seriousness of misclassifying a child. People also have varying ideas on what the levels of sensitivity and specificity should be because there are so many socioeconomic and epidemiologic factors that influence people and produce very different perspectives,” Dr. Maguire said.
“No one type of screening would be ideal for each scenario,” she added.
“In the ideal world, every child would receive a complete eye exam every year and no child would be left out,” said Graham E. Quinn, MD, a member of the executive committee for the National Eye Institute’s Vision In Preschoolers Study. “But there is no feasible way to provide that service for the 4 million children born every year, either in terms of manpower or resources or ability to reach children in underserved populations and underserved areas. So we have to search for the best screening tools available and decide how many kids with what kind of eye problems are we willing to miss. It basically becomes a cost issue: a cost to the child, cost to society. It’s a complex interaction that most of us just can’t get our heads around.”
“The screening issue is something that lay people don’t get,” said David B. Granet, MD, a participant in the University of California San Diego’s (UCSD) public screening project. “Every ophthalmologist can get it — they just have to take the time to think through the complexities.”
Familiar with the problems facing public health programs, Kevin D. Frick, PhD, said that a mixture of statistics and science surrounds the screening dilemma.
“It’s a little bit economics, measuring quality of life, understanding statistics and epidemiology. For ophthalmologists, optometrists and other health care professionals who want to engage in the policy process, knowing at least a little bit of these issues is becoming more and more important,” said Dr. Frick, an associate professor at the Johns Hopkins Bloomberg School of Public Health who has focused his research on preventive ophthalmology.
He said that while physicians do not need to fully comprehend the mechanisms behind cost-benefit analysis, it is important for them to interpret the end result of studies to determine how sensitivity, specificity and cost issues affect their practices and patients.
“What this requires is putting together all the different societal factors to ask the ultimate question: ‘How can we make our patient’s lives better?’” Dr. Frick said.
Sensitivity, specificity
In simple terms, sensitivity and specificity are elements that help determine the predictive value of screening.
“Sensitivity refers to the proportion of children who have a condition and are identified by a screening (ie, true positives), and specificity relates to the proportion of children who do not have a targeted condition and are correctly classified as normal by the screening test (ie, true negatives),” Dr. Maguire said. Ideally, she said, both the sensitivity and specificity of a screening tool would be high, 90% or better.
“Both predictors should be high, so that the screening fails most of the children who have vision problems and passes the children who are normal,” Dr. Maguire said.
She gave this example to illustrate sensitivity and specificity. In a screening of 1,000 children looking for a disorder that is present in 15% of the population, using a screening device with a sensitivity of 66%, 100 of 150 children with vision problems will be identified correctly as having a problem. Because 66% sensitivity indicates that about two of every three children with the disorder of interest will be identified, 50 children with the disorder will be missed. In that same group of 1,000 children, if specificity is high at 90% and there are 850 patients who are normal, 10% of children will fail the screening test even though they do not have the disorder. Therefore, 85 children (10%) will be misclassified and referred for an eye examination.
If sensitivity is low, many patients who have vision problems could be missed. These cases are deemed “false negative.” If specificity is low, a lot of patients who do not have vision problems could be referred for an evaluation. They are considered “false positive.”
In vision screening, the implications of false negatives and false positives include the under-referring of children with problems and the over-referring of children without problems.
False negatives, false positives
Some experts are willing to accept false negatives in vision screening because they believe that refractive errors, amblyopia and strabismus will probably not significantly affect a child’s quality of life.
“It’s not as though a child will die if they are not caught,” Dr. Maguire said. She noted that while catching children with vision problems early is beneficial because there is a greater chance of therapeutic success, identifying children a few years later can still provide benefit and may not severely hamper a child’s quality of life.
“If you miss a 3-year-old, that child may be screened again the next year and many years after that,” Dr. Maguire said. Thinking about sensitivity and specificity in vision screening is different than in screening for AIDS or breast cancer, she said.
“If you do an AIDS screening, it’s extremely important to get the answer right. The implications of an incorrect answer are tremendous either way you look at it — both if you tell a patient he has AIDS when he doesn’t and if you tell a patient he doesn’t have AIDS when he really does,” Dr. Maguire explained.
At worst, false positives in vision screening are a burden to budgets, an unnecessary eye examination for the child and a “nuisance” to society, she said, but they do not carry the physical, emotional or financial burden of misdiagnosis of a cancer patient.
“It’s a waste of resources to examine a child who does not have a vision disorder, but it’s not a horrible thing to have happen,” Dr. Maguire said, referring to the cost of eye exams, the time spent to take the child and family members out of school or work, and the possible pay lost if a parent takes a child on a visit and has to miss work.
“If you want to find every last case, or at least as many as we can expect to find, you’re going to have a lot of these false positives,” Dr. Frick said. He noted that a false positive may make parents and patients wary of screening measures.
“It can be very discouraging to a parent when they think there’s a problem with their child’s vision, and they spend a lot of money and time to get their child checked out, and nothing’s wrong,” agreed Sean P. Donahue, MD, PhD, founder of Vanderbilt University Medical Center’s Operation KidSight, a screening program that has screened more than 140,000 children in Tennessee.
The alternative, Dr. Frick noted, is that programs would have to minimize the number of false positives, and then many cases could be missed.
“It’s a real trade-off,” he said.
Low sensitivity, high specificity
Because the levels of specificity and sensitivity can be adjusted in some screening tools, the amount of over-referral and under-referral falls into the hands of individual screening programs. It is up to the administrators or health care professionals who provide the screening to decide what number of failures or passes they are “willing to accept,” Dr. Quinn said.
Dr. Granet agreed. “Society really has to decide how many misses it’s willing to accept. The problem is, it’s easy to say how many children you are willing to miss, until that child is yours,” he said.
For many screening programs, the level of sensitivity or specificity is tailored to the programs’ needs. Factors such as where a program is located (in a rural or urban environment) or if the screeners are experts (physicians or volunteers) help determine the predictive value of screening.
“In certain settings, you want the sensitivity to be as high as possible, but you also worry about the specificity,” Dr. Maguire said. For example, screening programs such as the Alaska Blind Child Discovery (ABCD) in Anchorage prefer a lower sensitivity and higher specificity because the cost of paying for over-referrals is a heavy burden.
“It costs us $1,000 per child for each eye exam,” said Robert W. Arnold, MD, director of the ABCD and vision screen chairman of the American Association for Pediatric Ophthalmology and Strabismus. He said many children screened in Alaska live in rural, isolated areas. ABCD sends volunteer crews to screen these children, but the cost of bringing children into Anchorage for a comprehensive eye exam makes over-referrals prohibitive.
“Amblyopia is present in less than 5% of children. If we send a screener into a village of 100 children and 18% are identified as having problems, we know that they all can’t possibly have amblyopia,” Dr. Arnold said. “But when those 18 children are identified, it’s our practice to pay for them and their parents to fly out to Anchorage, to stay in a hotel overnight and get an eye exam.”
If all those 18 children are seen and only four require glasses, project coordinators are left with a large expense. “We are grateful that the children who needed glasses were caught,” Dr. Arnold said. “But we just spent $14,000 on children who didn’t need eye exams.”
Dr. Donahue, of Vanderbilt, said he also prefers a screening program with low sensitivity and high specificity. “With low sensitivity and high specificity, most children who get referred are going to have a problem,” Dr. Donahue said. He noted that 93% of children at Operation KidSight receive a passing grade, 3% retake their screening due to an unreadable photograph, and 4% fail their screening tests and are referred for an eye exam.
“This is different from most screening programs that want to catch everybody,” Dr. Donahue said.
High sensitivity, low specificity
Not everyone is comfortable with low sensitivity and high specificity for screening. In fact, certain screening programs prefer higher sensitivity and lower specificity rates because the cost of travel is not an issue, as it is in the Alaskan program, and because referring more children improves the chances that children who otherwise would not receive medical attention are seen.
In the Head Start program, a Department of Health and Human Services program for children of low-income families, program coordinators prefer to have a high sensitivity for their screening tests.
“They want to make sure that they can arrange care for children while they have responsibility for them,” Dr. Maguire said. “So in that situation, administrators might ask for high sensitivity because if they have extra kids to examine, if it turns out that they are OK, it doesn’t seem as bad to them as missing a child with a vision problem.”
Children in the Head Start program usually are covered under some type of health insurance program, so an eye exam would not be an out-of-pocket expense for their parents, nor would it be a financial burden on program coordinators.
“A lot depends on who is paying the bills,” Dr. Maguire noted.
At UCSD, Dr. Granet deals with the problem of setting screening criteria. “If you live in an area [where it is] easy to get to a pediatric ophthalmologist, the last thing you want to do is miss someone. You’d rather over-refer to make sure everyone with disease is checked,” he said. “But if you live in an area where you have to go 500 miles to see an eye doctor, you don’t want to spend a lot of money to send a whole bunch of people who don’t require an exam.”
But he pointed out that some children may not get another opportunity to have their eyesight checked, especially if their family thinks they have already been evaluated. “So in my mind, it’s worth it to have a high sensitivity,” Dr. Granet said.
Cost-effectiveness evaluation
Once policy makers, physicians, managed care providers and others in the community begin to understand the complexities of the predictive value of screening, they can begin to evaluate what types of screening devices are desirable, based on sensitivity and specificity. This is where cost becomes a factor, Dr. Frick said.
“After you decide on how sensitive a test is at finding a case and how specific it is at ruling out healthy kids, you need to find out the costs of both screening and follow-up using this criteria,” Dr. Frick said. Policy makers must decide whether they want to find more cases and spend more on false positives or detect fewer cases but save on false positives.
“Every situation is different, all factors interact,” Dr. Frick said. “The cost of screening might be as low as $10, while the cost of screening and follow-up per child identified can be in the hundreds of dollars or higher.”
He gave an example for a population of 100 children. If a screening test costs $10 per child and one in 10 children have the eye health problem being screened for, the actual cost per case identified is $100 because 10 children were screened for every one case found. The total cost of the screening is $1,000. If the screening test used is only 90% sensitive, only nine children in 100 would be found instead of the ideal 10. This raises the cost of identifying one child to $111.
The cost increases again when false positives are included. In his example, Dr. Frick suggested that 11 false positives were identified, bringing the total number of children to be examined to 20. If all 20 children go for an eye exam, at $50 per exam, the cost of the screening and follow-up exams combined is $2,000, “and we’ve only identified nine cases,” Dr. Frick said. “So now we’re up to $222 for each case that we found.”
To this amount, add the cost of treatment. For lifetime amblyopia treatment, Dr. Frick suggested a ballpark figure of $2,000. Treatment for all nine children, combined with screening and the initial exam, is now $20,000.
Gauging quality of life
One way economists gauge the benefit of interventions such as vision screening is through quality-adjusted life years, or QALYs. In public health research, QALYs are the measure of an intervention’s effect on quality of life over time as well as the number of years added to a person’s life by the intervention.
As Dr. Frick said, QALYs are “a combination of saying how good your life is and how long your life is.”
Traditionally, 1 QALY refers to 1 year of life in “perfect” health. An intervention worth 5 QALYs might add 5 years to a person’s life, for example. Public health analysts also use QALYs to calculate economic effects by speaking about dollars per QALY.
“It puts a dollar value on what people are willing to spend on reduced risk,” explained David Ropeik, director of risk communication at the Harvard Center for Risk Analysis.
While amblyopia treatment, for example, may not extend a person’s life, it can improve a person’s quality of life over time, Dr. Frick noted. He said studies have assigned a value of about 0.8 QALY over a person’s life to the treatment of amblyopia.
“What happens with amblyopia treatment is that each year you get a tiny improvement in a patient’s quality of life. Over the course of a lifetime this adds up and eventually comes to 80% of a QALY,” Dr. Frick explained.
While 0.80 QALYs might seem insignificant, Dr. Frick said that it is a substantial improvement in quality of life in comparison to some other common interventions and comparatively inexpensive at the average value of amblyopia treatment.
Experts sometimes use the rule of thumb that a therapy that costs $50,000 or less per QALY is both beneficial to society and cost-effective, he said.
Vision researchers have said that the benefits of amblyopia screenings can be in the range of $2,000 to $3,000 per QALY, but the figures vary from study to study, Dr. Frick said.
“It’s safe to say that the cost of amblyopia therapy is a good buy to society because it falls far below the commonly cited threshold of $50,000 per QALY,” he said.
Dr. Frick added that the same can be said about comprehensive eye exams. He noted that some in the vision care community have called universal comprehensive eye exams too expensive because they are more costly than vision screening. But even at twice the cost of vision screening, they would still be a “good buy,” if the resources were available, because they also fall under $50,000 per QALY, he said.
To learn more
While dollars per QALY and other concepts may not be part of the vocabulary of many physicians and health care workers, vision screening researchers say the important message is that there is no definitive answer when it comes to screening methods and cost issues.
Regarding screening criteria, Dr. Maguire said that the Vision In Preschoolers Study will attempt to provide guidelines on sensitivity and specificity.
“Currently, we don’t have a good idea about what sensitivities and specificities should be, but in collecting our data we will certainly make recommendations about sensitivity and specificity levels and pass-fail criteria,” she said.
As for deciding how to calibrate screening tools for sensitivity and specificity, Dr. Maguire said “the VIP will leave that up to the lawmakers.”
|
Dr. Frick said physicians who are interested in learning more about sensitivity and specificity and the cost-effectiveness of interventions in medicine should look into general public health courses offered at institutions. (See the box above.)
“These courses will enable physicians and other health care workers to understand the concepts and be able to broach the subject with lawmakers and legislators,” Dr. Frick said.
Ultimately, Dr. Quinn said, legislators will make the final choices.
“It’s easy for people to say everyone should have eye exams. But the reality is, even eye exams don’t catch everyone. This is something that President George W. Bush and our Congress will have to grapple with, once society decides what is needed,” he said.
For Your Information:
- Robert W. Arnold, MD, can be reached at Ophthalmic Associates, 542 W. Second Ave., Anchorage, AK 99501; 907-276-1617; fax: 907-278-1705; e-mail: eyedoc@alaska.net.
- Sean P. Donahue, MD, PhD, can be reached at the Department of Ophthalmology, Vanderbilt University Medical Center, 8000 Medical Center East, Nashville, TN, 37232; 617-936-2020; fax: 615-936-1540; e-mail: sean.donahue@vanderbilt.edu.
- Kevin D. Frick, PhD, can be reached at Johns Hopkins Bloomberg School of Public Health, 624 North Broadway, Room 606, Baltimore MD 21205-1901; 410-614-4018; fax: 410-955-0470; e-mail: kfrick@jhsph.edu.
- David B. Granet, MD, can be reached at Shiley Eye Center, 9415 Campus Point Drive, La Jolla, CA 92039; 858-534-2020; fax: 858-534-5695; e-mail: dgranet@ucsd.edu.
- Maureen G. Maguire, PhD, can be reached at the Center for Preventive Ophthalmology and Biostatistics, Scheie Eye Institute, 3535 Market St., Suite 700, Philadelphia, PA 19104; 215-615-1501; fax: 215-615-1531; e-mail: maguirem@mail.med.upenn.edu.
- Graham E. Quinn, MD, can be reached at 1st Floor, Wood Center, The Children’s Hospital of Philadelphia, Philadelphia, PA 19104; 215-590-2791; fax: 267-426-5015; e-mail: quinn@email.chop.edu.
- David Ropeik can be reached at the Harvard Center for Risk Analysis, 718 Huntington Ave., Boston, MA 02115; 617-432-6011; e-mail: dropeik@hsph.harvard.edu.
- Nicole Nader is an OSN Staff Writer who covers all aspects of ophthalmology, specializing in pediatrics/strabismus and neuro-ophthalmology.