A recent study has found that the AI chatbot ChatGPT nearly passed the radiology board exam. Radiology is a field of medicine that uses imaging technology to diagnose and treat disease. The study was conducted over the course of late February and early March of this year and involved a 150 multiple-choice questionnaire modeled on the Canadian Royal College and American Board of Radiology exams. Evaluation of the AI chatbot’s performance was measured by topic and question type. ChatGPT correctly answered 69% of the questions, only 1% short of the passing threshold of the exam.
The AI chatbot proved to be more proficient on simple lower-order questions, answering 84% correctly, and 60% on the more difficult higher-order questions. It was less efficient in questions involving calculations, classification, and conceptual applications. The ChatGPT also consistently gave confident answers, even if they were wrong.
The study brings to light the potential of language learning models (LLM) like ChatGPT, as well as the limitations that make it unreliable. Another study found that ChatGPT’s predecessor, GPT-4, showed vast improvement compared to the GPT-3.5. GPT-4 was tested on the same 150 questions, and scored 81% (121 questions correctly) compared to the GPT-3.5 which only answered 104 correctly.
The ChatGPT AI chatbot is a product of OpenAI, a research company whose mission is to advance the field of Artificial Intelligence (AI). They build products to help make AI more practical, reliable, and secure. By conducting research and building systems, OpenAI hopes to benefit humanity through AI. Their ChatGPT AI chatbot specifically is a large-scale language model, which can generate language similar to the way humans do.
The study is led by Rajesh Bhayana, MD. He is a medical doctor focused on furthering the development of AI-based applications in the healthcare industry. He has participated in several research studies focusing on artificial intelligence and various aspects of its application to the healthcare field. His current research focuses on ChatGPT’s abilities in the context of the radiology board exam, where he hopes to provide insight into the potential and limitations of language learning models.
