Q&A: Simple machine learning model predicts suicide risk well

Key takeaways:

• Compared with more complex models, a simple machine learning model similarly predicted suicide risk.
• The findings support the use of simpler models, which are easier to implement in clinical practice.

A logistic regression model for predicting suicide risk performed similarly well compared with more complex machine learning models, according to findings published in npj Digital Medicine.

Susan M. Shortreed, PhD, and colleagues used data from more than 25,800,888 mental health visits among 3,081,420 people in seven health systems to evaluate different machine learning models’ ability to predict suicide risk. These included logistic regression, original parsimonious, random forest, artificial neural network and full ensemble models.

Healio spoke with Shortreed, first author of the study and a senior biostatistics investigator at Kaiser Permanente Washington Health Research Institute in Seattle, to learn more about the study and the clinical implementation of machine learning models for suicide risk prediction.

Healio: What prompted this study?

Shortreed: For several years, our research team has been developing and evaluating statistical models that use health care data to identify people at high risk for self-harm or suicide. Our first generation of models considered a few hundred predictors and used relatively simple — and understandable — statistical methods. We were interested in whether we could more accurately identify people at high risk with more complex models that considered several thousand predictors and used more complex statistical methods that are often harder to understand or explain.

Healio: What are the differences between the machine learning models you evaluated?

Shortreed: Our original models, which are currently in use in some health systems, considered a few hundred possible predictors and use logistic regression, a relatively simple statistical method. The models we compared to this simpler model considered several thousand predictors, with much more detailed information about timing, and used more complex statistical methods called random forest, artificial neural network and ensemble models. Both random forest and artificial neural networks models are designed to identify and estimate complex relationships between predictors and the risk of self-harm or suicide. In particular, if the relationship is different for subgroups, random forest and artificial neural networks are designed to find and estimate those differences. Ensemble models combine predictions from all of the models (logistic regression, random forest and artificial neural networks).

Healio: Which model(s) worked best?

Shortreed: The different models we examined performed about the same. Results were similar enough that we believe simpler and more understandable models are preferred. Simpler models are easier for health systems to implement and for clinicians to understand.

Healio: What is the take-home message for clinicians?

Shortreed: Clinicians would not usually be choosing which statistical prediction models health systems would use. Our methods are probably more relevant to health systems implementing (or considering implementing) these tools. Since all the models performed similarly in this setting, health systems can feel comfortable implementing the simpler models. It takes more resources to implement the more complex models because they rely on more predictors. This means more code is required to extract the data and run the model, making it more likely to slow down the computational systems, especially if code is run routinely to generate new predictions for individuals as their care and symptoms evolve over time.

Healio: What do the next steps look like in terms of research and practice?

Shortreed: We are exploring other ways to improve the accuracy of these prediction models. One example is using machine learning to find relevant information in the text of clinicians’ notes that is not reflected in diagnostic codes or other simpler types of data.

Healio: Is there anything else you would like to highlight about this topic?

Shortreed: Current health systems use these risk models to alert providers to conduct additional risk assessment, and these models could also be used to inform who might benefit the most from scarcer resources. We found model performance across subgroups (race, Hispanic ethnicity and sex assigned at birth) was similar, but it is important to examine how accurate models are in different patient populations and consider the potential harms and benefits of any intervention across subgroups of the population before implementation.

Reference:

• Shortreed SM, et al. NPJ Digit Med. 2023;doi:10.1038/s41746-023-00772-4.