Recent years have seen a surge in the use of artificial intelligence (AI) and deep learning (DL) in ophthalmology. ChatGPT is a large language model developed by OpenAI based on the Generative Pre-trained Transformer 3 (GPT-3) series, which produces human-like text. To assess the performance of ChatGPT in ophthalmology, a study was conducted which evaluated the model’s success and shortcomings.
The study evaluated the performance of two versions of ChatGPT, with the updated version including enhanced factuality and mathematical capabilities. Using the American Academy of Ophthalmology’s Basic and Clinical Science Course (BCSC) Self-Assessment Program and OphthoQuestions online question bank, the study’s test questions reached up to 260. Then the authors tested ChatGPT with 20 randomly selected questions from thirteen sections of the Ophthalmic Knowledge Assessment Program (OKAP) exam.
The results showed that while ChatGPT displayed an accuracy of 59.4% based on the BCSC testing set, the model achieved a score of 49.2% in the OphthoQuestions testing set. Humans scored 74% on the BCSC question bank, and a group of ophthalmology residents who completed their training in 2022 scored 63% on OphthoQuestions. ChatGPT’s performance in ophthalmology is promising as it matches the accuracy levels of advanced LLMs in general medical question-answering, which usually rank between 40-50%.
The authors plan to conduct a qualitative analysis of ChatGPT in the future to identify areas requiring improvement. An application programming interface (API) for ChatGPT could help validate the technology and facilitate the process. In addition, it would be beneficial to incorporate specialized foundation models trained with domain-specific sources, such as EyeWiki.
OpenAI is a technology company based in San Francisco that specializes in artificial intelligence research and development. Founded by Elon Musk, Sam Altman, and others in 2020, OpenAI creates revolutionary technologies such as the autonomous driving systems powering Tesla, autonomous robotics, and natural language processing.
Dr. Priyom Bose, Ph.D. is an AI researcher and data scientist with a strong background in biomedical engineering and machine learning. He has published several research papers on AI and deep learning, including the recent Ophthalmology Science study on Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings. In addition, Dr. Bose has also worked on AI-driven medical diagnosis and computer-aided diagnostics.
