ChatGPT ‘generally accurate’ in answering questions, providing references on IBS
Click Here to Manage Email Alerts
Key takeaways:
- Overall accuracy of ChatGPT 4.0 was 80%, although some details were missed or information was outdated.
- The references provided by the chatbot were suitable for 33% of the answers.
VANCOUVER, British Columbia — ChatGPT 4.0 had an overall accuracy of 80% in its answers to questions about irritable bowel syndrome, although the chatbot still missed some details or provided outdated information, according to research.
“The use of ChatGPT is likely to increase,” Anthony Lembo, MD, study author and director of research at Cleveland Clinic’s Digestive Disease Institute, told Healio. “Patients and clinicians should be aware of the accuracy of information they are receiving.”
Seeking to examine the accuracy of answers and references given by ChatGPT 4.0 to questions about IBS, Lembo and colleagues used Google Trends to identify related searches over the past 5 years and developed 15 questions based on those queries. Each question was then entered into ChatGPT 4.0 and the model was asked to provide references for generated answers.
Three independent gastroenterologists assessed ChatGPT’s answers, which were evaluated by two grading systems: an overall assessment (accurate or inaccurate) and a granular assessment (100% accurate, 100% inaccurate, accurate with missing information or partly inaccurate). The chatbot’s references also were graded as suitable, unsuitable (existent but unrelated to answer) or nonexistent.
According to results presented at ACG Annual Scientific Meeting, ChatGPT 4.0 had an overall accuracy of 80%, with granular grading demonstrating that 53% of the answers were accurate, 33% were partially inaccurate and 13% were accurate but missing information. No answers were completely inaccurate.
“For common IBS patient questions, ChatGPT is generally accurate in providing patient information for IBS,” Lembo said. “ChatGPT did miss some details and provided outdated information, but no fully inaccurate information.”
Further, references provided by ChatGPT were suitable for 33% of answers, unsuitable for 53% and nonexistent for 13%, researchers reported.
Although ChatGPT remains problematic for medical professionals with regard to literature research and referencing, as it improves, “ChatGPT could be an important reference for patients,” Lembo told Healio. “ChatGPT also has the potential to assist physicians in answering patient questions sent through records, like EPIC MyChart.”
Collapse
El Dahdah, et al. ChatGPT-4.0 answers common irritable bowel syndrome patient queries: Accuracy and references validity. Presented at: ACG Annual Scientific Meeting; Oct. 20-25, 2023; Vancouver, British Columbia (hybrid meeting).
Read more about
Click Here to Manage Email Alerts