Mental health organizations are increasingly turning to technology to ease the burden on their clinicians amid rising demand for mental health services. However, these organizations can’t introduce any innovations that might compromise their clinical standards. How can mental health organizations have confidence that technology can be trusted in an industry as important as theirs?

A new study, available in Volume 7 of the Journal of Medical Internet Research, offers proof that technology can be trusted by demonstrating the performance of Eleos Health’s augmented intelligence capabilities in real-world therapy settings. The study looked at the CareOps Automation platform’s capability to accurately evaluate over 2.83 million micro-dialogues. (A micro dialogue is a brief exchange between a therapist and client, usually containing a few sentences related to the same topic.)

The Question

Augmented intelligence technology is becoming more common in behavioral health and serves many purposes. The industry has an obligation to make sure clinical decision support is accurate, since unreliable tools can result in incorrect diagnoses, ineffective treatments, or even harm to patients.

It’s also crucial to ensure that research studies informing treatments are based on data reflecting the actual habits and practices of therapists in their daily work. Researcher Josef I. Ruzek, PhD put it this way:

“In our field of psychotherapy, relatively little is known about what happens in routine care. That is, when you go into a private practice, or you go into a healthcare system and you see a psychologist or social worker or some other counselor, we don’t know too much about what goes on in those sessions. We do know a great deal about what we think is good care. There’s an entire research literature that evaluates different kinds of psychotherapies for different kinds of mental health problems. But there’s thought to be a huge gap- called the scientist/practitioner gap- between what the researchers or what the specialty trained elite therapists may do and what is happening in routine care… So If there’s a way we can begin to shine a light on what happens in the therapy process, that’s potentially quite helpful. It’s helpful for research but it’s also helpful for the training of young therapists and potentially it’s very helpful for the therapists themselves to know a little bit more about what’s happening in their own sessions. So this study, it’s really a proof of concept.”

To test the Eleos Health augmented intelligence model’s evaluative accuracy, researchers put it to the task of determining how frequently a therapist assigned homework to a client during a session. Even though homework has been found to improve clinical outcomes, until now, there hasn’t been a convenient way to determine how often therapists assign at-home work.

The Setup

This study tested Eleos’ capabilities against a team of four psychology and social work experts with graduate degrees.

The data set evaluated by the software included:

30,000+ audio-recorded outpatient treatment sessions, which included

2.83 million therapy micro dialogue exchanges between

8 behavioral healthcare programs involving

6,236 clients and 675 therapists.

A data set of 2.83 million micro dialogues may seem huge, and according to researcher Dr. Shiri Sadeh-Sharvit, it is actually unprecedented in size. 

“The 2.83 million therapist-client micro-dialogues constitute an unprecedented dataset,” she shared, “And it reflects the diversity and complexity of actual clinical practice, providing a wealth of valuable information and insights that cannot be obtained from other sources.  Unlike previous research in this area that relied on self-reports or data from controlled academic studies, we based this research on how therapy is being delivered in the real world. “

The software was tasked with determining for each session:

(a) whether the session included a homework assignment or not, and 

(b) which type of homework was assigned.

The team of experts did not listen to every session, but instead evaluated a randomly selected representative sample of 100 sessions.

The Software

Eleos’ software is capable of what’s known as “machine learning”. Machine learning is defined as “the use and development of computer systems that are able to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze and draw inferences from patterns in data.” In other words, machine learning is artificial intelligence that lets software learn, adapt, and recognize patterns.

In order to “tell” the software what to look for, researchers worked alongside the Eleos software engineers and the team of experts to develop parameters. 

First, the team of experts categorized the homework types they identified in their sample set. The homework-type categories were:

1. Practicing,
2. Taking action,
3. Journaling, and 
4. Learning.

Using Eleos’ proprietary base of data from 34,000 sessions, the team of graduate-level therapists with at least two years of experience providing direct care to clients in community-based clinics evaluated 4,000 randomly selected micro dialogues to determine whether homework was assigned and what type. 

Software engineers then used those 4,000 labeled micro dialogues to “teach” the model what to look for when detecting exchanges about homework.

The Results

The results of the study answered two questions:

1. What is the frequency of therapists assigning homework to their clients?
   
   Through their initial sample analysis, researchers found that at least one homework assignment was given in 61% of sessions. This is a positive result as homework is known to play a crucial role in the effectiveness of therapy. According to Dr. Sadeh-Sharvit, “This study marks a crucial turning point in our understanding of how therapeutic homework is employed in real-world clinical settings. Through the innovative use of natural language processing and machine learning techniques, we have uncovered new insights that hold great promise for improving treatment fidelity and enhancing outcomes for clients.”
2. How accurate can AI be in determining the frequency of therapists assigning homework? The machine learning model used in the study demonstrated a high level of agreement with expert raters’ classifications, surpassing established machine learning models that rely on larger treatment datasets.

In terms of further research regarding homework, the researchers identified a need to explore using augmented intelligence to enhance therapy homework assignments. Specifically, they suggested that it might be able to assist therapists in determining whether homework is needed, send reminders to include homework assignments as part of the session, simplify the documentation process associated with homework, and prompt both therapists and clients to engage in and review homework assignments.

The Implications

Technology has the potential to radically change the behavioral health industry- and this study proves it. 

The Eleos software answered a clinically significant question with high accuracy, providing insight that may eventually help craft more effective therapeutic interventions, all without relying on tedious self-reporting mechanisms or interfering with clinical workflows.

With ongoing advancements in augmented intelligence- and research like this that proves its legitimacy- we can look forward to a future where technology-assisted therapy becomes an increasingly important part of mental health care, delivering better outcomes and improving the lives of countless individuals.

(You can watch a quick recap by Dr. Shiri Sadeh-Sharvit below.)