ChatGPT would not pass ophthalmology board exam
Click Here to Manage Email Alerts
SEATTLE — If ChatGPT were to take an ophthalmology board-style exam, it would not pass, even if given feedback and a second chance, according to a poster presented at the Association for Research in Vision and Ophthalmology meeting.
However, when given detailed history and exam elements, it showed the ability to likely formulate the correct diagnosis and differential diagnosis in acute ocular problems.
Belinda Ikpoh, MD, and co-authors selected 15 questions covering 11 subspecialty categories from the American Academy of Ophthalmology question bank. On first attempt, ChatGPT answered 59% of the questions correctly. Feedback was given for incorrect answers. The same question was asked a second time, but ChatGPT showed little ability to learn and answered only 24% of the questions correctly. Combining all attempts, the average accuracy was 69%.
“ChatGPT would not pass the ophthalmology written qualifying exam,” the authors wrote.
In a second part of the investigation, ChatGPT was asked to formulate a primary diagnosis and a differential diagnosis based on data from the electronic medical record of patients presenting with new acute ocular problems at a triage clinic. The ChatGPT diagnosis matched the physician’s diagnosis in 67% of the cases, and the differential diagnosis included the correct diagnosis in 88% of the cases.
“Correct EMR responses are likely dependent on relevant history and exam elements being provided to ChatGPT,” the authors wrote.
Collapse
Ikpoh B, et al. Assessing the accuracy of ChatGPT in diagnosing ocular disease. Presented at: Association for Research in Vision and Ophthalmology meeting; May 5-9, 2024; Seattle.
Read more about
Click Here to Manage Email Alerts