ChatGPT ‘should be used with caution’ for common pregnancy questions

Key takeaways:

• ChatGPT generated at least one satisfactory response to 93% of common pregnancy questions.
• Of 84 responses, 27% contained misinformation.

ChatGPT 3.0 generated satisfactory responses when asked common pregnancy questions, but responses became inconsistent when asked various iterations of the same question, data show.

In findings presented at The Pregnancy Meeting, researchers evaluated ChatGPT 3.0, one of the rapidly increasing AI platforms available to patients, to identify its ability to answer questions regarding common pregnancy conditions.

“It’s important to study these resources specifically in our population and for patients who may be struggling to access care in a hospital or clinical environment, given that there are roadblocks to care, such as child care,” Leonardo F. Antelo, MD, resident physician at Mount Sinai West Obstetrics and Gynecology in New York, told Healio. “As these technologies keep advancing, they could serve to aid patients in getting access to medical care they need in those settings.”

Antelo and colleagues generated seven questions related to gestational diabetes, preeclampsia, fetal growth restriction and low amniotic fluid and fed them into ChatGPT 3.0 identically three separate times for a total of 84 responses. Each response set was reviewed for appropriateness, consistency and the presence of misinformation by a board-certified maternal-fetal medicine physician. Researchers rated each response as satisfactory, incomplete or inadequate.

Of 28 questions, 47% received consistent satisfactory ratings over the separate iterations. In addition, 93% of questions had at least one satisfactory response and 25% had consistent generated answers rated as satisfactory without containing misinformation.

Overall, 27% of 84 responses contained misinformation and 61% were rated as satisfactory without containing misinformation.

“For patients, these findings showed that ChatGPT is a quick and easy resource to access comprehensive answers on the internet, but these responses that were generated should still be reviewed with their physician,” Antelo told Healio. “For physicians, it shows that although ChatGPT is a great resource, there are small nuances within the responses that may not apply to every clinical scenario and may need to be further reviewed before we wholeheartedly recommend ChatGPT as a resource to our patients.”

In an abstract, the researchers noted that ChatGPT was inconsistent over multiple iterations of the same question and “should be used with caution.” Some of the documented misinformation was related to the diagnosis of certain conditions, and some were related to condition management with minor nuances deviating from standard of care, Antelo said.

“We should incorporate patients into our future studies to see exactly what online resources they’re using and how they’re using these platforms, possibly incorporating surveys to see what questions they are looking to get answered on the internet and how they’re accessing these resources,” Antelo told Healio. “It is also important to expand the reviewers used to validate ChatGPT as a successful resource for patients. This can include incorporating physicians, trainees and medical providers who are in different stages of training, such as residents and fellows, to see how the different groups of medical providers would rate ChatGPT as a good resource for patients.”

For more information:

Leonardo F. Antelo, MD, can be reached at leonardo.antelo@mountsinai.org; Instagram: @msw.obgyn.residency.