Algorithm predicts 30-day mortality after chemotherapy initiation

A machine learning algorithm used medical records to accurately predict 30-day mortality among patients starting chemotherapy, according to results of a retrospective cohort study published in JAMA Network Open.

“Chemotherapy lowers the risk of recurrence in early-stage cancers and can improve survival and symptoms in later-stage disease,” Aymen A. Elfiky, MD, MPH, MSc, instructor of medical oncology at Dana-Farber Cancer Institute, and colleagues wrote. “Balancing these benefits against chemotherapy’s considerable risks is challenging. Increasing evidence suggests that chemotherapy is too often started too late in the cancer disease trajectory, and many patients die soon after initiating treatment.”

Physicians often refer to trial data or population-level data, such as SEER data, for information about patient mortality before initiating chemotherapy, although these sources provide only broad estimates.

“There is considerable enthusiasm for the role of advanced algorithms to improve prediction; just as modern electronic health records pull complex data for clinicians to use in real time, algorithms could pull and process these data in parallel, presenting accurate probability forecasts to clinicians and patients,” the researchers wrote. “However, little evidence suggests that such algorithms can provide meaningful inputs to clinical decision-making in cancer or elsewhere.”

The model relied on data from 2004 to 2011 for the derivation set, and measured performance using nonoverlapping data from 2012 through 2014 for the validation set.

To transform EHR data into variables that can be used in a prediction model, researchers pulled data from the 1-year period ending the day before chemotherapy initiation and aggregated data into 23,641 potential predictors in the categories of demographics, prescribed medications, comorbidities and other grouped ICD-9 diagnoses, procedures, use of health care resources, vital signs, laboratory results, and terms derived from physician notes using natural language processing. After dropping variables missing in more than 99% of the derivation sample, there were 5,390 predictors in the model.

The analysis included 26,946 patients (mean age, 58.7 years; 86.9% white; 61.1% women) beginning a total of 51,744 chemotherapy regimens between 2004 and 2014. Elfiky and colleagues determined patients’ date of death by linking records with Social Security data.

Death within 30 days of starting new systemic chemotherapy regimens served as the primary endpoint. Secondary outcomes included 30-day mortality in patient subgroups and overall 180-day mortality.

Among patients in the validation set (n = 9,114), overall mortality within 30 days of the start of chemotherapy was 2.1% (95% CI, 1.9-2.4). In this cohort, breast cancer was the most common primary cancer (21.1%; 95% CI, 20.2-21.9), followed by colorectal cancer (19.3%; 95% CI, 18.5-20.2) and lung cancer (18%; 95% CI, 17.2-18.8).

The model appeared accurate for all patients, regardless of chemotherapy regimen (area under the curve [AUC], 0.94; 95% CI, 0.93-0.95).

The model’s predictions remained accurate for a subset of patients (46.6%) beginning palliative chemotherapy (AUC, 0.92; 95% CI, 0.91-0.93).

The researchers ranked patients who received palliative chemotherapy into model-predicted mortality risk deciles to illustrate the model’s discrimination. In the highest decile, the observed 30-day mortality was 22.6% (95% CI, 19.6-25.6), whereas no patient in the lowest decile died.

The model demonstrated accuracy across stages, primary cancers and chemotherapies, even in cases where clinical trial regimens emerged for the first time in the years after the model was trained (AUC, 0.94; 95% CI, 0.88-1).

The model maintained a strong performance for the prediction of 180-day mortality for all patients (AUC, 0.87; 95% CI, 0.86-0.87) and for highest- vs. lowest-risk decile patients (74.8%; 95% CI; 72.7-77 vs. 0.2%; 95% CI, 0.01-0.4).

Researchers then compared their model performance with two external sources of mortality estimates — SEER data and randomized clinical trials — focusing on patients with distant-stage disease. Results showed their model predictions (AUC, 0.81; 95% CI, 0.79-0.82) outperformed SEER estimates (AUC, 0.6; 95%CI, 0.58-0.61) for 1-year mortality. Also, researchers found the overall AUC for randomized control trial estimates was 0.55 (95% CI, 0.51-0.59) compared with 0.77 (95% CI, 0.73-0.8) for model-based estimates for the same patients.

“To be useful, predictive models must improve decision making in the real world,” the researchers wrote. “Thus, rigorous evaluation of predictions’ influence on outcomes is the criterion standard test but one that is often neglected in the literature, which focuses primarily on measuring predictive accuracy rather than real outcomes.” – by Andy Polhamus

Disclosures: Elfiky reports no relevant financial disclosures. One author reports personal fees from GNS Healthcare outside of the submitted work.