Accuracy of an artificial intelligence chatbot in identifying congenital glaucoma from other ocular etiologies

Purpose To evaluate the accuracy and appropriateness of responses provided by ChatGPT in identifying congenital glaucoma from a series of written and image-based prompts. Methods A series of questions regarding common signs and symptoms of congenital glaucoma were developed and queried to ChatGPT-3.5 and ChatGPT-4.0, and a set of publicly available images of patients with congenital glaucoma were queried to the image search function of ChatGPT-4.0. Outputs were graded by three pediatric ophthalmologists with expertise in congenital glaucoma. Completeness of response, accuracy, potential for harm, and concern for glaucoma were assessed by each reviewer. Results A higher proportion of prompt responses from ChatGPT-4.0 were graded to be acceptable/appropriate than from ChatGPT-3.5 (22/33 vs 9/33 [P = 0.001]) among text-based queries. A higher proportion of ChatGPT-4.0 responses were felt to raise appropriate concern for congenital glaucoma (8/11 vs 2/11 [P = 0.03]) and a lower proportion of responses had incorrect or inappropriate information of major clinical significance (0/33 vs 6/33 [P = 0.02]) than ChatGPT-3.5 responses. There was no significant difference in the proportion of responses from ChatGPT-3.5 and ChatGPT-4.0 that were deemed to have potential likelihood of harm (P = 0.17). Among clinical images queried to ChatGPT-4.0, responses to two of three images were universally felt to be unacceptable with a major amount of incorrect or inappropriate clinical information and high/definitive likelihood of harm. Among readability indices, the SMOG Index score showed more difficult readability scores for ChatGPT-4.0 than for ChatGPT-3.5 (14.8 ±1.2 vs 14.0 ± 1.4 [P = 0.009]). Conclusions Despite superior performance from ChatGPT-4.0 compared with ChatGPT-3.5 in raising concern for congenital glaucoma and appropriateness of responses from text-based prompts, it performed poorly in recognizing clinical images of congenital glaucoma.