ChatGPT accurately, yet sometimes inconsistently, debunks many allergy myths
Click Here to Manage Email Alerts
Key takeaways:
- Allergists rated 70% of ChatGPT’s responses to questions posed about allergy myths as accurate and 21% as somewhat accurate.
- Almost half of allergists planned to use chatbots in the future for patient education.
ANAHEIM, Calif. — ChatGPT answered accurately or somewhat accurately 91% of the time when asked about 10 allergy myths, according to a study presented at the American College of Allergy, Asthma & Immunology Annual Scientific Meeting.
However, the AI chatbot also appeared inconsistent, sometimes answering differently when repeatedly asked the same question.
Although chatbots such as ChatGPT are in the preliminary stages, especially in terms of their health care capabilities, many patients may be using these platforms to seek medical advice, Ricardo J. Estrada-Mendizabal, MD, of Tecnologico de Monterrey, School of Medicine and Health Science, told Healio.
“The impact of these chatbots in allergy/asthma care in the future will depend on their accuracy and consistency when answering medical questions,” he said. “This process will require a huge effort from [artificial intelligence (AI)] developers, researchers and health care workers.”
To further understand the accuracy of ChatGPT to answer medical questions, Estrada-Mendizabal and colleagues sought to test its accuracy against allergy myths. Allergy as a field in particular may be more susceptible to misinformation, Estrada-Mendizabal said.
“In our study, we made a database search in PubMed and Google Scholar encompassing the last 23 years, and we found a total of 49 common myths and misconceptions,” he told Healio. “We recommend patients seek information from a board-certified allergist.”
From that list, the researchers selected 10 of these myths to pose to ChatGPT as a true-or-false question, including:
- Hypoallergenic pets are better for people with asthma and allergies.
- Anaphylaxis can be prevented by antihistamines and/or steroids.
- Food allergies can be prevented by delaying introduction of allergenic foods.
- Individuals with egg allergy cannot receive the influenza vaccine.
- A positive allergy test indicates a clinical allergy.
- IgE tests can identify food triggers in childhood eczema.
- Seafood allergy is a contraindication for contrast media.
- An allergic reaction cannot occur after the first exposure.
- Breastfeeding can help prevent food allergies.
- Oral immunotherapy cures food allergies.
Two sub-investigators entered these myths into ChatGPT, with a third selecting the most complete answer of the two or re-entering the question if there was a discrepancy. This scenario with discrepant responses occurred for three of the myths, including those involving contrast media, breastfeeding and oral immunotherapy.
Next, 24 allergists — 88% of whom were from Mexico, 8% from the U.S. and 4% from Argentina — rated each of the answers using a Likert scale from 1 to 4, from inaccurate (100% not true), to somewhat inaccurate (more than 30% not true), somewhat accurate (less than 10% not true) and accurate (100% true).
In general, the allergists’ ratings revealed that ChatGPT was mostly accurate in its responses, with 70% rated as accurate and 21% as somewhat accurate. Only 5% were rated as somewhat inaccurate and 4% as inaccurate.
Allergists had the most heterogeneous accuracy rating of ChatGPT’s response to the statement about hypoallergenic animals, which read: “Some animals are better suited for people with asthma and allergies than others. While no animal is completely hypoallergenic, certain breeds or species may produce fewer allergens or trigger fewer allergic reactions in sensitive individuals. It’s important to note that allergies can vary from person to person, so what works for one individual may not work for another. ...”
Seventeen percent of the allergists rated that response as inaccurate, whereas 21% found it to be somewhat inaccurate, 41% to be somewhat accurate and 21% as accurate.
With 96% of the allergists rating the response as accurate, ChatGPT appeared most accurate in its answer to whether a positive allergy test means a clinical allergy, saying, in brief, “While a positive test result suggests sensitization to the allergen, it does not confirm the presence of clinical symptoms or an allergic reaction upon exposure to that allergen.”
The overall accuracy of ChatGPT was unexpected, Estrada-Mendizabal said, “especially given the fact that ChatGPT’s ‘training’ ended in September 2021 and that it is unable to provide references.
“However, a significant finding of our study is the inconsistency of ChatGPT because when prompted about the same myth on two separate occasions by two different sub-investigators, it responded with opposing statements,” he added.
Next, the allergists indicated whether they would use natural language models such as ChatGPT for patient education in the future, with 46% answering that this was very likely and 29% answering that it was likely.
It is important to continue to study the impact of AI in medicine, Estrada-Mendizabal said.
“I would certainly like to continue exploring the impact of AI in medicine, especially the impact of chatbots, considering that several versions are being launched constantly by multiple developers,” he said. “For my next study, I would like to focus on common questions that patients might have, and I also want to have a higher and more representative sample of allergists.”