This article is more than 5 years old. Information may no longer be current.
Evaluation framework informs decision-making for mental health apps
The American Psychiatric Association developed an evaluation framework that uses a four-stage hierarchical process model to help clinicians evaluate and make informed decisions regarding mental health smartphone apps, according to a column published in Psychiatric Services.
“The literature on health app ratings offers tools to help clinicians and patients pick apps, but so far none of these tools provides a reliable method of evaluating an app’s safety and usefulness,” John Blake Torous, MD, department of psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, and colleagues wrote. “Clinician ratings of individual features of mental health apps also suffer from low interrater reliability. Apps are tools that must be selected on the basis of individual needs, abilities, preferences and many other personal patient factors.”
To offer clinicians guidance on evaluating and making informed decisions regarding the thousands of smartphone apps that target mental health, researchers explained the rationale and testing behind the APA’s smartphone app evaluation framework. The APA’s framework approach employs a four-stage hierarchical process that asks users to consider safety and privacy first, evidence and benefit second, engagement third and interoperability fourth, according to the authors.
“The evidence shows that while appealing, scoring and point-based systems for app evaluation are not useful and can even be misleading,” Torous told Healio Psychiatry. “The framework we present in this article offers a tool to help users make more informed decisions by considering relevant information and weighing the risks and benefits for the unique app and unique clinical case at hand. The app world is fast moving, and our approaches to evaluating must also be agile and flexible.”
After beginning with safety and privacy, evaluation only proceeds to the next stage if that specific app meets the unique clinical needs; if an app does not do that, evaluation should stop there. The framework offers a series of questions to determine which app works best for an individual patient.
The first two stages — privacy and safety, and evidence and benefit — establish basic medical decision-making based on nonmaleficence, according to Torous and colleagues. The third stage, engagement, reflects awareness that patients don’t always stick with apps or find them hard to use, possibly because they aren’t involved in mental health app development. The last stage, data sharing, takes into account the need to ensure that app data are available to help direct care and make treatment decisions.
Because the evaluation framework will develop based on user feedback and assessment, Torous and colleagues performed internal interrater reliability testing with five psychiatrists, where each was presented with three mood tracking apps that are similar to those used in a previous study of app usability among patients with depression.
They asked psychiatrists to rate the app, using the app evaluation framework, in two clinical cases. The first involved a tech-savvy patient in his 20s with moderate depression, but no suicidal ideation, interested in using a mood tracking app while on a selective serotonin reuptake inhibitor; the second involved a less tech-savvy patient in her late 60s with moderate depression interested in utilizing two rarely-used apps to monitor her mood while on a SSRI. The researchers advised each reviewer to use each app for at least 15 minutes before reviewing it and to search for any research studies on the apps. They then analyzed concordance among the reviewers in ratings of each stage of the framework.
After using Kendall's coefficient to show agreement among the raters, the researchers observed concordance of 0.93 for the privacy and safety stage (P .01), 0.95 for evidence and benefit (P < .01), 0.67 for engagement (P .01) and 0.77 for interoperability (P < .01).
“An evaluation framework for informed decision-making is a useful solution to the current challenges involved in ratings of apps. Although these initial efforts were developed and tested with psychiatrists, efforts are under way to incorporate diverse stakeholder input into this framework, including the voices of patients and family members,” the authors wrote. “We hope the APA efforts presented here will stimulate discussion and encourage informed decision-making around using apps in clinical care.” – by Savannah Demko
Disclosures: Torous reports no relevant financial disclosures. Please see the full study for other authors’ relevant financial disclosures.