Researchers develop AI model to accelerate radiation therapy planning for lung cancer
Click Here to Manage Email Alerts
Radiation therapy is the most common treatment for lung cancer, the most frequently diagnosed type of cancer worldwide.
Lung cancer is often deadly but can be treated successfully — even cured in some cases — through the use of targeted radiation therapy, but planning how much tissue to target is often a time-consuming and resource-intensive process, according to Raymond H. Mak, MD, associate professor of radiation oncology at Harvard Medical School and director of clinical innovation, thoracic radiation oncology disease site leader and director of patient safety/quality in the department of radiation oncology at Brigham and Women’s Hospital/Dana-Farber Cancer Institute.
Mak and colleagues developed an artificial intelligence model for segmenting primary non-small cell lung cancer tumors and involved lymph nodes on CT images. Results of an observational study using their model, published in Lancet Digital Health, showed a significant 65% reduction in segmentation time (P < .0001) and a 32% reduction in interobserver variability (P = .013).
“AI algorithms can actually improve human performance in a human‐AI partnership that can result in a direct benefit to patients through greater consistency in segmenting tumors and accelerating times to treatment,” Mak told Healio.
The researchers anticipate their AI model will more easily translate into clinical use because it has been trained by humans informing the AI on how earlier iterations of the algorithms failed.
Mak spoke with Healio about the results of his group’s research and future plans to make the AI model a clinically relevant and useful offering for planning NSCLC treatment.
Healio: What was your group’s rationale for putting together this AI model?
Mak: Nearly half of patients with lung cancer will eventually require some form of radiation therapy, which can be curative. Planning for a course of radiation currently entails manual, time-consuming and resource-intensive work by highly trained physicians to segment — aka target — the cancerous tumors in the lungs and adjacent lymph nodes on three-dimensional CT scan images. Previous studies have shown substantial variation in how expert clinicians delineate these targets, which can negatively impact outcomes. Additionally, there is a projected shortage of skilled medical staff to perform these tasks worldwide as cancer rates increase.
Healio: How is your approach to applying a deep learning algorithm unique?
Mak: Our team developed deep learning algorithms that can automatically target cancer in the lungs and adjacent lymph nodes from CT scans that are used for radiation therapy planning, and they can be deployed in seconds. We trained the AI algorithm using expert-segmented targets from over 700 cases and validated the performance in over 1,300 patients from external data sets — including publicly available data from a national trial — benchmarked its performance against expert clinicians, and then further validated the clinical usefulness of the algorithm in human-AI collaboration experiments that measured accuracy, task speed and end-user satisfaction.
Healio: What results have you seen so far?
Mak: We demonstrated that our AI algorithm could automatically segment/target lung cancer in the chest — both primary lung tumor and lymph nodes — with a high level of accuracy. As a benchmark, we volumetrically compared the AI-generated segmentations against the gold standard expert-delineated targets and found the overlap was within the variation we observe between human clinicians.
Most importantly, through extensive end-user testing and survey work, we demonstrated that these volumetric overlap metrics (eg, Dice coefficient) commonly used for in silico validation of auto-segmentation algorithms did not correlate well with clinical utility (eg, task speed improvement, end-user satisfaction).
In these human-AI collaboration experiments, we asked clinicians to either manually segment/target lung cancer cases de novo, or to edit a segmentation we provided them in a clinical environment. We added a wrinkle to the study by providing the clinicians with either an AI-generated segmentation or a segmentation generated by another clinician, but in a blinded fashion. In this three-way comparison, we found that AI collaboration led to a 65% reduction in segmentation time (decrease from median 15.5 minutes to 5.4 minutes) and 32% reduction in inter-clinician variation. Interestingly, however, the clinicians did not experience a significant time savings when editing another human’s segmentation.
Healio: What benefits can your AI algorithm provide to clinicians?
Mak: One of the biggest translation gaps in AI applications to medicine is the failure to study how to use AI to improve human clinician outcomes, and vice versa. In this study, we took advantage of clinician expertise and intuition with early input into the development and training of the AI algorithm. Essentially, we first had humans teach AI by studying how early iterations of the algorithms were failing and then augmented our training data to improve performance.
We then demonstrated that the final AI algorithms can improve human performance in human-AI partnerships that can result in a direct benefit to patients through greater consistency in segmenting tumors and accelerating time to treatment. Our surveys of the clinicians who partnered with the AI demonstrate that they also experienced substantial benefits in reduced task time, high satisfaction and reduced perception of task difficulty, which is an interesting additional benefit that we had not thought about initially — that AI could reduce cognitive load and possibly reduce physician burnout. Hopefully, as we deploy these algorithms into the clinic, we will also see additional benefits for clinicians with reduced time doing mundane computer work and more time in quality interactions with patients.
Healio: How does your approach overcome known obstacles to the implementation of AI-based applications in medicine?
Mak: For readers evaluating new AI technologies for clinical implementation, we hope our research presents a framework for thoughtful AI development that incorporates clinician input and includes a rigorous testing and validation framework. Our approach was to include performance benchmarking, identify key modes of failure and determine whether an AI algorithm performs as intended in the hands of clinicians before introduction of the algorithm in the clinic. We believe that an evaluation strategy for AI models that emphasizes the importance of human-AI collaboration is especially necessary because computer-modeled validation can give different results than clinical evaluations. As an extension of this work, we are designing and conducting prospective, randomized trials of similar AI auto-segmentation algorithms in the clinic to provide the highest level of evidence.
Healio: What are your plans for this approach?
Mak: Keep an eye out for our upcoming work to convert this AI development, testing and validation framework into an AI label, which we hope — akin to an FDA drug label — will provide radiation oncology researchers and clinicians with a way to quickly reference and understand the core components, performance features and warnings of a given algorithm.
Healio: Can this research help bridge the gap between proof-of-concept studies in small cohorts and larger prospective clinical trials?
Mak: We believe there will be a direct benefit to patients with cancer through thoughtful testing and implementation of human-AI collaboration in radiation therapy planning by providing patients with higher quality tumor segmentation and accelerating times to treatment. Furthermore, our surveys of the clinicians who partnered with the AI demonstrate that they also experienced substantial benefits in reduced task time, high satisfaction and reduced perception of task difficulty. Wouldn’t it be interesting if this trend bears out in wider studies, with AI leading to reductions in clinician cognitive load and stress, and help with physician burnout?
For more information:
Raymond H. Mak, MD, can be reached at Brigham and Women's Hospital, Department of Radiation Oncology, 75 Francis St., Boston, MA 02115; email: rmak@partners.org.
Collapse
Click Here to Manage Email Alerts