Accuracy of voice assistants for medical information varies
Click Here to Manage Email Alerts
Researchers found distinct differences between popular digital voice assistants when they were used for information about cancer screening.
“There is room for improvement across all assistants,” the researchers wrote in Annals of Family Medicine.
Google is asked more than 1 billion health questions daily, Steven Lin, MD, co-author of the new study and head of technology innovation for the division of primary care and population health at Stanford University School of Medicine, told Healio Primary Care.
“Since nearly one‐half of US adults use a voice assistant, most commonly through their smartphones, we decided to examine how popular voice assistants respond to questions about cancer screenings, one of the most important preventive services that patients are missing due to the pandemic,” he said.
For the study, researchers asked Amazon’s Alexa, Apple’s Siri, Google Assistant and Microsoft Cortana whether they should get screened for 11 different types of cancer.
“When evaluating accuracy, we compared responses to the U.S. Preventive Services Task Force’s (USPSTF) cancer screening guidelines,” Lin and colleagues wrote. “A response was deemed accurate if it did not directly contradict this information and if it provided a starting age for screening consistent with these guidelines.”
According to the researchers, Alexa “consistently” did not know the answer to questions about cancer screening. Although Cortana, Google Assistant and Siri comprehended the questions, each displayed varying levels of accuracy in the answers. Cortana provided correct screening information via a web search 68% of the time and verbally answered correctly 45% of the time; Google Assistant provided correct screening information via a web search 67% of the time and verbally answered correctly 64% of the time; and Siri provided correct screening information via a web search 70% of the time. The CDC or the American Cancer Society were referenced in 47% of all web search responses.
“Given how often patients and the general public are searching for answers to health-related questions online and through their voice assistants, physicians should care about the accuracy of health information being delivered digitally,” Lin said in the interview.
Previous research has similarly revealed discrepancies in voice assistants’ abilities to provide accurate responses to health care-related questions.
In one recent study, researchers from Ohio reported that Siri and Google Assistant interpreted all 14 questions on postpartum depression accurately, while Microsoft Cortana showed 93% accuracy and Alexa showed 79% accuracy. The researchers added that all four devices offered “clinically appropriate advice” for 64% of the 14 questions.
A separate study analyzed responses from Alexa, Siri and Google Assistant to questions about vaccine safety and use. Each device was graded a maximum of six points on each question for quality and accuracy. Researchers reported that Alexa averaged 0.98 point per question, Google Assistant averaged 5.1 points per question and Siri averaged 5.16 points per question.
“Voice assistants have the potential to support health care delivery and information dissemination, both during and after COVID‐19, but they also present a risk of propagating misinformation,” Lin said. “Improving the accuracy and trustworthiness of online health information requires partnership between technology providers, physicians and public health authorities.”
References:
Alagha EC, Helbing RR. BMJ Health Care Inform. 2019;doi:10.1136/bmjhci-2018-100075.
Hong G, et al. Ann Fam Med. 2021;doi:10.1370/afm.2713.
Yang S, et al. JMIR Mhealth Uhealth. 2021;doi:10.2196/24045.