This article is more than 5 years old. Information may no longer be current.

Gradient boosting model accurately predicts delirium

A gradient boosting machine model performed best among five machine learning models tested for predicting delirium, according to findings recently published in JAMA Network Open.

“Existing clinical delirium risk prediction tools ... often rely on questionnaires administered by health care professionals (eg, Mini-Mental State Examination), nonroutine clinical data (nursing subjective illness severity assessment), or additional calculations (eg, Acute Physiology and Chronic Health Evaluation score), making their integration into routine clinical workflow impractical. ... Furthermore, existing tools recapitulate well-studied delirium risk factors, such as cognitive impairment at baseline, delirium on admission, and severe illness,” Andrew Wong, BA, of the School of Medicine at the University of California, San Francisco, and colleagues wrote.

Researchers retrospectively gathered data from a training set consisting of 14,227 hospital admissions, of which 5,113 patients were aged older than 64 years; 7,335 were female and 687 had delirium. The test set consisted of 3,996 hospital admissions, of which 1,491 patients were older than 64 years; 1,966 were female and 191 had delirium. All patients had at least one Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) screen or Nursing Delirium Screening Scale (Nu-DESC) performed within 30 days of their hospital admission.

Wong and colleagues evaluated five machine learning model algorithms: artificial neural network with a single hidden layer, random forest, gradient boosting machine, linear support vector machine and penalized logistic regression.

They found the gradient boosting machine model performed best, with a positive predictive value of 23.1% (95% CI, 20.5-25.9) and negative predictive value of 97.8% (95% CI, 97.4-98.1) when specificity was set at 90%. This model also had a number needed to screen of 4.8 and an area under the curve of 0.855. The penalized logistic regression model had an area under the curve of 0.854 and random forest models had a similar score of 0.848, which researchers said indicated these tools also did well.

“We recognize that newer predictive models such as [artificial neural networks] have been shown to outperform older models such as [gradient boosting machine, random forest and penalized logistic regression] in prediction accuracy,” Wong and colleagues wrote.

“However, such models require more computational power and larger training data sets and are far more technically challenging to integrate into clinical workflow. With the goal of creating a usable clinical tool in mind, the use of simpler models is more appropriate for many institutions at this time,” they added.

Sherri Rose, PhD, of the department of health care policy at Harvard Medical School wrote in a related editorial that though machine learning models have their strengths, there are weaknesses to the technology that must also be addressed.

“When implementing machine learning for prediction in electronic health data, it is critical to remember that these data are not collected to answer specific research questions, which is a central difficulty in relying on them for these purposes. Machine learning might or might not provide benefits and the data might not be robust enough to be useful to clinical teams,” she wrote.

“[However,] there is reason to be optimistic about the ability of machine learning to transform prediction in an array of medical fields. Machine learning also has demonstrated promise in clinical domains when the goal is to discover clusters in the data, such as imaging analysis for therapeutic selection. ... We are in a discovery phase, and the pervasiveness of electronic health big data across many clinical areas does not ultimately mean machine learning will be equally valuable in each,” Rose concluded. – by Janel Miller

Disclosure: Rose reports receiving NIH grants during the course of the study. None of the other authors reported any other relevant financial disclosures.