Charité Study Reveals Limits of AI in Precision Medicine, Shows Promise for Future Use in Cancer Treatment
Treating cancer has become increasingly complex, with a wide range of possibilities arising from a better understanding of tumor biology and genetic features. To provide patients with personalized therapies tailored to their disease, extensive analysis and interpretation of various datasets are required. Researchers at Charité – Universitätsmedizin Berlin and Humboldt-Universität zu Berlin have conducted a study to investigate whether generative artificial intelligence (AI) tools, such as ChatGPT, can assist in this process. This study, among many others at Charité, explores the potential of AI in patient care.
In the development of a tumor, an imbalance of growth-inducing and growth-inhibiting factors plays a crucial role. This imbalance can result from changes in oncogenes, which have the potential to cause cancer. Precision oncology, a specialized field of personalized medicine, utilizes this knowledge to target and disable hyperactive oncogenes through specific treatments like low-molecular weight inhibitors and antibodies.
The first step in identifying potential treatment targets among genetic mutations is to analyze the genetic makeup of the tumor tissue. This involves determining the molecular variants of the tumor DNA that are critical for precision diagnosis and treatment. Using this information, doctors craft individual treatment recommendations. In complex cases, expertise from various medical fields is required. This is when the molecular tumor board (MTB) at Charité comes into play. Experts from pathology, molecular pathology, oncology, human genetics, and bioinformatics collaborate to analyze the latest studies and determine the most promising treatments. This meticulous process ultimately leads to personalized treatment recommendations.
In their recent study published in the journal JAMA Network Open, Dr. Damian Rieke, Prof. Ulf Leser, Xing David Wang, and Dr. Manuela Benary investigated whether artificial intelligence could facilitate treatment decisions at this critical stage. They explored the capabilities and limitations of large language models, like ChatGPT, in automatically scanning scientific literature to identify personalized treatment options.
We prompted the models to identify personalized treatment options for fictitious cancer patients and then compared the results with the recommendations made by experts, explains Dr. Rieke. The conclusion drawn from the study was that while AI models could, in principle, identify personalized treatment options, they were far from matching the abilities of human experts.
The researchers created ten fictional molecular tumor profiles for the experiment. A human physician specialist and four large language models were tasked with identifying personalized treatment options. The recommendations were then assessed by the members of the MTB, who were unaware of the source of each recommendation.
Dr. Benary highlights, There were some surprisingly good treatment options identified by AI in isolated cases, but large language models perform much worse than human experts. Additionally, the use of artificial intelligence in real-world patient scenarios poses challenges related to data protection, privacy, and reproducibility.
However, Dr. Rieke remains optimistic about the potential uses of AI in medicine. He notes that the performance of AI models continues to improve as advancements are made. This suggests that AI could offer greater support in complex diagnostic and treatment processes in the future, with final decisions resting in the hands of human experts who verify and approve the AI-generated results.
AI projects at Charité are actively aimed at enhancing patient care. Prof. Felix Balzer, Director of the Institute of Medical Informatics and Chief Medical Information Officer (CMIO) at Charité, emphasizes how medicine stands to benefit from AI. One area of focus is digitalization and the use of automation and artificial intelligence to achieve greater efficiency in patient care.
The Institute of Medical Informatics is currently working on AI models to prevent falls in long-term care, while other research groups at Charité are exploring AI-based prognosis following strokes and facilitating the validation and certification of AI and robotics in medical devices.
The study conducted by Charité and Humboldt-Universität zu Berlin was supported by funding from the German Research Foundation (DFG), Deutsche Krebshilfe, and the German Federal Joint Committee Innovation Fund.
In conclusion, the study reveals the existing limitations of AI models in precision medicine while offering hope for their future potential in cancer treatment. Although AI models currently fall short of human experts, ongoing advancements pave the way for increased support in diagnostic and treatment processes. As AI continues to evolve, human oversight and verification remain crucial to ensure patient safety and the delivery of personalized care.
* Reference: Benary W., Wang XD., Schmidt M. et al. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Network Open 2023 Nov 20. doi:10.1001/jamanetworkopen.2023.43689
