A new study published in JAMA Ophthalmology has found that the artificial intelligence system known as ChatGPT scored just 46% on a board certification practice test for ophthalmology. The performance improved to 58% the second time it was tested, one month later.
The examination used a dataset of practice multiple choice questions from a commonly used resource for board certification exam preparation, and the response was at a great convenience since ChatGPT is free and accessible from any internet connection.
The use of artificial intelligence (AI) in medical and exam practice has become increasingly popular since its launch in November, 2022, but many are concerned that the accuracy may not be reliable.
The study was led by Rajeev H. Muni from St. Michael’s Hospital, a site of Unity Health Toronto. He has stressed for the responsible use of such AI systems, and recommended that ChatGPT is not suitable for providing assistance in preparing for board certification at the moment.
The research team analysed 125 text-based multiple-choice questions and provided feedback on the results. When comparing the outcomes with the most popular responses of ophthalmology trainees, the AI system answered 44% of the questions correctly and was least popular 11% of the time in selecting the appropriate answer.
Another researcher, Marko Popovic of the Department of Ophthalmology and Vision Sciences at the University of Toronto, is optimistic on the role of AI in medical education. He believes that with the development of AI, it will eventually be able to provide more exact answers.
The most accurate answers from ChatGPT were regarding general medicine questions which quizzed 79% correctly. The chatbot scored the lowest in niche subspecialties such as oculoplastics (20%) and retina (0%).
Andrew Mihalache, a Western University undergraduate student and lead author of the study, foresees the potential for ChatGPT accuracy to improve over time. As such, it is important to ensure responsible use and ensure that it does not provide any short cut for board certification.
