Verified by 
Artificial Intelligence
AI in Precision Medicine: A Lack of Generalizability?
Study shows AI lacks robustness in predicting schizophrenia treatment outcomes.
Posted January 22, 2024 Reviewed by Davia Sills
One of the most promising uses of the predictive capabilities of artificial intelligence (AI) is in digital health care, especially in precision or personalized medical treatments. As AI algorithms gain traction in health care and precision medicine, it is critical to understand not only the strengths of machine learning but also potential weaknesses. A new study led by the Yale School of Medicine shows how AI algorithms used to predict patient outcomes lack generalizability—they only worked well for the specific clinical trial they were developed for, but not for different clinical trials for schizophrenia treatments.
Using AI machine learning to help predict whether or not a particular patient will respond to drug treatment is an important aspect of precision medicine. According to Yale-led researchers, antipsychotic drugs for treating schizophrenia do not elicit enough of a clinical response in over 50 percent of individuals with a relapse and up to 20-30 percent of first-episode individuals, depending on the clinical outcome definition.
In the field of artificial intelligence, a key metric for robustness in machine learning algorithms is generalizability, the ability of the AI model to perform with a high degree of accuracy on new data that has not been seen before by the algorithm. Ideally, an AI algorithm used for predicting precision medicine treatment outcomes is robust. This new research is important because it examines AI algorithms under the hood at the mathematical level.
“The field as a whole—present authors included—hope that machine learning approaches can eventually improve the allocation of treatments in medicine; however, we should a priori remain skeptical of any predictive model findings that lack an independent sample for validation,” wrote corresponding author Adam Chekroud, Ph.D., a Forbes “30 Under 30 2018: Consumer Technology” award recipient, an adjunct assistant professor of psychiatry at Yale, and the Co-Founder of Spring Health, along with his seasoned interdisciplinary team of noteworthy co-authors from the fields of medicine, psychiatry, data science, and neuroscience.
Yale researchers who participated in this study include the Chair of Yale’s Department of Psychiatry, Professor John Krystal, M.D., Associate Professor of Psychiatry Philip Corlett, Ph.D., Professor Harlan Krumholz, M.D., S.M., director of the Yale New Haven Hospital Center for Outcomes Research and Evaluation (CORE), and researchers Matt Hawrilenko, Ph.D., Ralitza Gueorguieva, Ph.D., and Hieronimus Loho, in collaboration with University Augsburg Professor and Chair for Psychiatry and Psychotherapy, Alkomiet Hasan, M.D., King’s College London Professor and Chair in Precision Psychiatry Nikolaos Koutsouleris, University of Cologne and the University Hospital Cologne Professor of Biological Psychiatry Joseph Kambeitz, M.D., Spring Health data scientist Julia Bondar, and the Scientific Director and President at the Laureate Institute for Brain Research Martin Paulus, M.D.
“Predicting treatment outcomes in schizophrenia could be especially advantageous because the clinical response to pharmacological interventions is heterogeneous and depends on many environmental factors such as individual and family-related stress, drug abuse, homelessness, and social isolation,” wrote the researchers.
Schizophrenia is a severe, chronic brain disorder that affects an estimated 24 million people globally, according to the World Health Organization (WHO). The National Institute of Mental Health (NIMH) defines psychosis as “a collection of symptoms that affect the mind, where there has been some loss of contact with reality.” The psychotic symptoms of schizophrenia may include hallucinations, delusions, unusual or illogical thoughts, and abnormal body movements per the NIMH. Other symptoms of schizophrenia may include disorganized speech, lack of emotion, withdrawal from others, inflated self-worth, and paranoia, according to Johns Hopkins Medicine. The treatment of schizophrenia, which does not have a cure, may include antidepressants or mood stabilizers, cognitive therapy, behavioral therapy, training, support groups, and antipsychotic medicines.
In the quest to understand exactly how accurately AI machine learning models predict schizophrenia patient outcomes across independent clinical trials of antipsychotic medication, the team evaluated an AI model performance on its initial training data and on data from independent clinical trials for patients with a current DSM-5 diagnosis of schizophrenia at the start of the trial. Specifically, data from five multisite, randomized, controlled trials from the Yale Open Data Access (YODA) Project, which included over 1,500 patients from over 190 sites across North America, Europe, Africa, and Asia, was used.
The researchers used an elastic net regression algorithm because this type of penalized regression method has been employed successfully in other psychiatry research studies for predicting psychiatric treatment outcomes. In statistics, elastic net regression is the combination of two types of variations of linear regression, ridge (L2 regularization) and LASSO (least absolute shrinkage and selection operator—L1 regularization). In machine learning, elastic net regression is used in order to decrease overfitting and increase AI prediction accuracy. In mathematics and statistics, linear regression models show the relationship between a dependent variable (y), also called a response variable, and one or more independent variables (X), also called predictor variables.
The researchers posit three potential contributing factors to this lack of generalizability: differences in patient population across trials, data quantity and types, and the context-dependency of patient outcomes.
Patient populations with the same diagnostic category may differ between trials. For example, patients with a current DSM-5 diagnosis of schizophrenia may be at different stages of disease progression.
“If key information that differentiates patients is not captured in the data or if the range of that information is more restricted in the dataset used to develop the model compared with the target trial, predictions may be inaccurate data quantity and type, and the dependency of patient outcomes on context,” wrote the researchers.
AI machine learning requires massive amounts of training data in order for the algorithm to “learn” features from the data. The volume of data may impact generalizability. Moreover, the specific types of data collected may impact the prediction quality of the AI algorithm. This study used patient information that includes sociodemographic, biomarker, and clinical data, but not psychosocial and social determinants of health. The researchers cite a different study published in The Lancet Psychiatry in 2016 by Dr. Professor Koutsouleris, M.D. et al., who also worked on the present study, that found psychosocial and social determinants of health useful in AI machine learning for predicting treatment outcomes for first-episode psychosis.
Interestingly, the researchers do not recommend genetic and brain imaging data types to enhance AI accuracy. While the exact cause of schizophrenia is not known, having a family history of schizophrenia is a risk factor, and differences in brain structure and the central nervous system have been shown in brain imaging, according to the Mayo Clinic.
“Some have suggested the use of neuroimaging and genetic data, but there is currently little evidence to suggest that such data would improve predictions; further, collecting these data would pose additional barriers for routine implementation,” the Yale-led researchers wrote.
Ultimately, antipsychotic medication patient outcomes for those with schizophrenia “may be too context-dependent.” For example, there may be trial-level characteristics in the treatment protocols and recruitment and inclusion criteria that impact patient outcomes.
“Our modeling scenarios using antipsychotic treatment outcome prediction in schizophrenia suggest that predictive models are fragile and that excellent performance in one clinical context is not a strong indicator of performance on future patients,” the researchers concluded.
Copyright © 2024 Cami Rosso. All rights reserved.
