ChatGPT, an AI chatbot designed for pediatric diagnoses, has been found to be highly inaccurate, according to a new study. The research, published in the journal JAMA Pediatrics, revealed that the AI chatbot failed to correctly diagnose 83% of the pediatric cases it examined.
Parents often turn to online resources to check their children’s symptoms, and AI chatbots like ChatGPT are becoming increasingly popular. However, this study shows that relying on such technology for accurate diagnoses may be problematic.
ChatGPT, powered by the OpenAI language model GPT-3.5, was put to the test by running pediatric case challenges through the chatbot and comparing its diagnoses with those made by clinicians. The results were concerning, with ChatGPT providing incorrect diagnoses for 72 out of 100 cases. In 11 cases, the diagnoses were deemed too broad to be considered accurate.
One example highlighted the chatbot’s inaccuracy in diagnosing a teenager with autism who displayed symptoms of a rash and joint stiffness. While the physician diagnosed the teen with scurvy, ChatGPT incorrectly diagnosed immune thrombocytopenic purpura, an autoimmune disorder. Another case involved an infant with a draining abscess, which the original physician attributed to Branchiootorenal (BOR) syndrome, while ChatGPT diagnosed a branchial cleft cyst.
Despite these inaccuracies, there were a few instances where ChatGPT matched the physicians’ diagnoses, such as in the case of a 15-year-old girl with idiopathic intracranial hypertension (IIH), where the chatbot correctly identified Addison’s disease as a possible underlying cause.
The study authors acknowledged that large language models like ChatGPT still have value as administrative tools for physicians, but the chatbot’s diagnostic performance is underwhelming. The researchers highlighted two limitations of ChatGPT: its inability to find connections between medical disorders and a lack of real-time access to medical information.
Moving forward, the researchers suggest that more selective training is necessary to improve AI’s accuracy in diagnosing medical conditions. They also stress the importance of ensuring chatbots stay updated with current research, diagnostic criteria, and health trends.
In conclusion, while AI chatbots have potential in healthcare, this study reveals significant inaccuracies in ChatGPT’s ability to diagnose pediatric cases. It emphasizes the crucial role of clinical experience and the need for ongoing improvements in AI technology to enhance diagnostic accuracy.
