Q&A: ChatGPT shows benefits in clinical setting but still carries risks

Key takeaways:

• Experts said that artificial intelligence like ChatGPT may have benefits in areas like billing or scheduling.
• There are challenges that could make ChatGPT implementation difficult in clinical settings in the near future.

Artificial intelligence has become increasingly viewed by experts as technology that could streamline several areas of health care while alleviating administrative burdens that have long plagued physicians.

As such, the abilities of models like ChatGPT — a chatbot from the artificial intelligence firm OpenAI — in the clinical setting are being piloted by experts like David Do, MD, an assistant professor of clinical neurology at the University of Pennsylvania, and Yevgeniy Gitelman, MD, a clinical assistant professor of medicine at the same institution, leading to ideas of how they might best be utilized.

Speaking to Healio, Do and Gitelman discussed the strengths and risks of ChatGPT in practice, the feasibility of its implementation in the short term and where research on the AI model may lead.

Healio: What benefits could ChatGPT offer in clinical settings, and what areas could it particularly perform well in?
Do: We’ve been exploring these use cases a lot ... the first breakdown is clinical care versus administrative parts of health care like billing or scheduling. Within each, there’s potential.

It could be helpful with manual tasks involving lots of clinical notes. For example, one administrative use case is extracting concepts from notes for billing. We’ve also been exploring use cases like one that helps patients schedule an appointment.

Gitelman: In clinical areas, two use cases have a lot of vendor interest, and that’s handling inbox messaging and ambient listening. Ambient listening has been going on since before GPT’s arrival, but GPT might improve what’s under the hood of these products.

Inbox handling is also important because clinicians are struggling with the volume of messages coming in. There's lots of hope that GPT will help ... but none of it is proven at this point.

Healio: What are potential challenges or risks involved with ChatGPT use?

Gitelman: I think explainability is the challenge for clinical scenarios like diagnosis or managing patients. GPT might tell you something incorrect or misdescribe something and then double down on it.

For example, when I asked GPT to write a letter to get a medication approved for prior authorization. I put it in the information, and it created citations that don’t exist. Then I asked for links to those citations, and it created links that looked real.

This is one of the other places where vendors can add value for note transcribing. It is not just taking a conversation and generating a note but also helping pinpoint where points are coming from in the conversation because people want to be able to trace things back. Why did it think what it was thinking? So, the clinical care part is going to be trickier. It might happen at some point, but not in the short term.

Do: Some might jump to the risk of misdiagnosis, but medicine is a very risk-averse field, so we wouldn’t unleash it to go make diagnoses without having a very long vetting process.

Another concern is privacy because this is very sensitive data. We definitely can’t use the public GPT to do clinical work.

Further, if you use clinical data to fine-tune these models, even if you try to de-identify it, it does scare me that enough information will get through, potentially allowing the model to learn about certain patients and their names, and so I think the risk is certainly a loss of privacy.

Healio: Where does research on ChatGPT implementation go from here?

Do: I think the first test is basically at face value. Researchers will pilot GPT for various types of roles and see if it’s helpful. I think the most important research to be done is piloting these with clinicians and asking, “Is this something that helps?”

Gitelman: Research is going in every direction. You can imagine people are trying to say, “Can I use it to abstract charts for summarization? Can I use it to summarize why a patient is here for a visit?” If I’m a radiologist, “Why is this patient getting an imaging study?” That is often hard to figure out.

In some ways, the inbox and ambient listening piece will not look like traditional research — but more like quality improvement, as many are trying to implement it actively.

People are just taking different scenarios, feeding it data and seeing how well it correlates to humans or corresponds to what a human clinician might figure out.

There will probably be some projects along those lines of, “Can it catch the things you may be missing?” Like, here's a note for a patient, and you think they’re having back pain. But did you at least consider that maybe this could be a clot or something else that also fits with this history? That's getting more into the realm of how it might get used clinically to augment decision-making.

Healio: Anything else to add?

Gitelman: It's really fun to play, dream and experiment with, and we’re also probably on that early ascending part of the hype cycle where there’s a lot more promise and talk than reality. People just need to be cautious about adopting and implementing it. I think that's going to be the approach of most health systems and probably bigger vendors at least.

Do: We're almost certainly on the ascending part of the hype curve, and most companies trying to apply this to medicine are going to fail. For many problems that we’re thinking about, there existed AI technologies that might have solved them — and the fact that it never caught on suggests that the absence of technology was only a small part of the problem.

For example, people hate doing prior authorization. As soon as you find a technology that makes it easier to get approved for medications and testing, utilization would increase, and the insurance companies would add another barrier. So, a lot of inefficiencies in medicine are by design.

Reference:

• Opportunities and pitfalls of ChatGPT in health care. https://chibe.upenn.edu/chibeblog/opportunities-and-pitfalls-of-chatgpt-in-health-care/. Published April 3, 2023. Accessed May 3, 2023.