OpenAI’s Chatbot Fails U.S. Urologists’ Self-Assessment Test

OpenAI’s ChatGPT chatbot has failed a US urologist exam, according to a study. The study, revealed in the journal Urology Practice, indicated that ChatGPT achieved below 30% accuracy rate in answering American Urologist Association’s Self-Assessment Study Program for Urology. It not only has a low rate of correct answers, but it also makes errors that could spread medical misinformation. The study excluded 15 questions containing visual information such as pictures. The chatbot gave accurate answers for less than 30% of SASP questions, whether multiple-choice or open-ended. The researchers suggested that while ChatGPT might perform well in factual recall tests, it fails on questions related to clinical medicine. It requires multiple overlapping facts, situations, and outcomes.