“Are your notes done? We can help.”
“The paperwork writes itself.”
“Meet your new AI therapy-note-writing assistant.”
If you’re a therapist, you’ve probably seen plenty of online ads promising to make your paperwork disappear. AI documentation tools that write notes for you sound almost too good to be true—and in some ways, they are.
AI is prone to “hallucinations,” which cause it to generate information that’s off-base or just plain wrong. As a therapist, you hold your clients’ trust—and their privacy—in your hands. A healthy dose of skepticism is a good thing. How can you trust—or vet—a tool you don’t fully understand?
But while AI might seem complicated, therapists are no strangers to learning complex systems. By educating yourself about AI, why it sometimes hallucinates, and how companies can (and should) address those issues, therapists can be smart, critical buyers—fully capable of making informed decisions instead of just being on the receiving end of a sales pitch.
So, let’s break it down.
AI 101: The Basics
Artificial intelligence works by recognizing patterns in the vast amounts of text it has processed. It draws from this huge library of information (basically the entire internet, in the case of ChatGPT) and analyzes how words and phrases are commonly used. That way, it can predict the best response to your questions or prompts.
But AI doesn’t actually understand the information it provides. It doesn’t have real knowledge or context like humans do. It simply guesses the best response, word by word and sentence by sentence, based on the patterns it has learned. This works okay—great, even—for general questions, but AI starts running into trouble when things get more specific or nuanced. And in therapy sessions, nuance is crucial.
What Causes AI Hallucinations?
To make good decisions about behavioral health AI tools, therapists first need to understand the mechanisms behind problems like hallucinations—because without that knowledge, it’s hard to know what to look for or the right questions to ask.
“Hallucinations” happen when AI doesn’t have the context necessary to supply accurate information. But because AI is programmed to deliver a response, it tends to fill in gaps with information that sounds reasonable—but isn’t accurate and factual. For example, if you ask AI a question about a meeting you briefly mentioned—but you don’t provide the full details—the AI might try to guess what was discussed, potentially mixing incorrect (or totally fabricated) information into the answer.
This can be especially risky in behavioral health. Misinformation, even if it’s not intentional, can seriously impact client care and trust. That’s why therapists need to understand what AI can actually do—and, more importantly, what it can’t do in a clinical setting.
Here are some of the most common reasons an AI tool might “hallucinate” in behavioral health:
1. Poor Data Quality
AI is only as good as the data it’s fed. If the data is incomplete or inaccurate, the AI will generate erroneous content. For example, if you’re asking AI to help with a treatment plan for someone with narcissistic personality disorder, but it’s drawing from an internet full of blogs and social media posts that misuse the term “narcissist,” then the AI will probably produce misleading or incorrect suggestions.
“AI doesn’t ‘know’ anything,” explains Amit Spinrad, Eleos Data Scientist. “It’s just predicting the next word based on what it’s seen before.”
In behavioral health, this can lead to the AI offering treatment recommendations based on pop psychology—instead of clinically accurate information.
2. Overly Broad/Unclear Queries
When the question is too vague, AI often struggles to give a precise answer. If a therapist asks for something broad like a “comprehensive treatment plan for depression,” the AI might pull in too much general information and lose the specific details that matter.
“Without clear and specific context, AI might ‘guess’ what the user wants,” Spinrad points out. And guessing increases the likelihood of AI hallucinations.
3. Biased Training Data
AI models are trained on huge datasets, but if that data is biased toward a particular set of characteristics, the AI will reflect this bias in its responses. For example, Spinrad says, “When most sessions are done with adults, the model would start to think in an adult-centric way.” This can be a problem if the AI is asked to interpret sessions involving children.
While therapists should be concerned about all three sources of hallucinations, bias is a particularly important concept to understand—and be wary of. Bias can influence how the AI system interprets sensitive information, sometimes leading it to misrepresent a client’s situation based on false assumptions or stereotypes.
When bias is baked into the data, the AI might miss important details or make inappropriate recommendations. In therapy, this can lead to AI-generated responses that aren’t just inaccurate, but also potentially harmful.
AI Bias and Hallucinations in Behavioral Health
In a therapy setting, AI bias typically happens because the model is trained on data that isn’t specific to behavioral health (i.e., its output is informed by unqualified sources). However, the output can still reflect biases even when the model is trained with therapy sessions and refined by behavioral health experts (as is the case here at Eleos).
Lidor Bahar, Eleos Data Scientist, says this is especially common when the AI favors responses that reflect more common conditions or demographic groups, pointing again to the example of children versus adults.
“Something we saw time and time and again in our models is that whenever there’s a child involved, the model completely biased its output toward the language of adults,” Bahar explains. “So you’ll see super strange stuff, like, ‘the child talked about their spouse.’”
To help address issues like hallucinations and bias, Eleos leverages Retrieval Augmented Generation (RAG). RAG helps AI stay grounded in reality by pulling truly relevant data before generating responses. When AI uses this approach, its output becomes more accurate—after all, in behavioral health, context is everything.
How Eleos Uses RAG and VectorDB to Prevent AI Hallucinations
Without Retrieval Augmented Generation (RAG), AI has millions of sources to choose from but no way of knowing which are the most relevant. It’s like walking into a library and picking a random book to search for the word “depression.” The first reference you find might be in a teen graphic novel—and in the case of AI, the tool would use that reference to create context around the topic. But with RAG, it’s like having an organized library and the Dewey Decimal System to guide you.
Here’s how it works:
- Embedding models act like the catalogers or publishers who classify all the books in the library. They take complex data—such as session transcripts or treatment notes—and convert them into “summaries” in the form of vectors, which are numeric representations of the content. These vectors capture the “theme” of the data, making it easier for the AI to process and compare.
- Once the data has been classified by the embedding models, it is stored in a Vector Database (VectorDB). Think of VectorDB as the library that holds all the books, organized by a framework like the Dewey Decimal System.
- The AI can now search through this library of categorized information and retrieve the most relevant data much more quickly and accurately—rather than sifting through random, unorganized information.
This combination of embedding models (the catalogers) and VectorDB (the organized library) helps the AI pull from the best possible sources—and do it much more quickly.
“You can’t search really, really fast for context if you keep it in a text format…so with RAG, there’s an additional layer of models that convert text into numbers,” Spinrad explains.
Eleos’ Unique RAG Approach
To make sure the most relevant notes are always front and center, Eleos uses a six-tiered system to select which past notes the AI should pull in. It starts with the notes that are closest in context—meaning the same organization, same note type, same provider, and same client—and works down from there.
If three highly relevant notes aren’t available, it broadens the search step-by-step, looking at factors like provider profession, population type, and session type, until it finds enough context to support accurate output. Over time, this tiering system helps the model get more specific, making the responses closer to the individual therapist-client relationship.
Eleos also has strict rules about which past notes the AI can use. The model only pulls previous notes from the same organization and note type, which means the context is in sync with the new note’s needs. Importantly, all notes used by the model are de-identified before they’re added, taking out any information that could reveal personal details. Even if the AI pulls from previous notes, it only draws on general, non-identified data—like common interventions or assessments—so that there’s no accidental cross-referencing of personal health information (PHI) in new notes.
This focus on context and privacy is a big part of why RAG helps cut down on hallucinations in clinical settings. While occasional errors can still happen, Eleos’ approach makes the AI much less likely to fill in gaps with irrelevant or inaccurate details, making the output more trustworthy.
How Therapists Benefit from RAG
The benefits of using RAG in behavioral health AI go beyond identifying the right context. With RAG, Eleos’ AI:
- Becomes more accurate. By selecting notes that are the most relevant to the current session, the model generates more accurate and context-aware content.
- Reduces errors and hallucinations. The RAG model minimizes hallucinations by referencing relevant sessions, so mistakes are less likely.
- Increases consistency. Notes created with RAG follow a more consistent style and tone, as the model draws on past note content that aligns with the new notes.
- Improves efficiency. RAG accelerates the note creation process by selecting contextually similar notes, reducing the need for extensive edits and leveraging a vector database to speed up similarity searches.
In other words, RAG helps Eleos AI keep its output relevant, accurate, and consistent so that the documentation process is both quicker and more reliable.
The Crucial Role of Providers in Preventing AI Errors
AI tools might seem like the answer to every therapist’s documentation woes, but therapists can’t just hand over the reins and hope the software gets everything right. There’s still a need for thoughtful oversight, especially in industries where the stakes are high—like behavioral health.
Here are a few ways therapists can make sure AI doesn’t wreak havoc with hallucinations.
1. Choose AI Tools That Use Advanced Techniques Like RAG
Some AI documentation tools are built on simpler “zero-shot” models, which generate responses without any context from past sessions. Tools based on these models might be easier to build, but they’re less accurate and more prone to hallucinations. Meanwhile, tools like Eleos that use Retrieval Augmented Generation (RAG) produce output that is more context-aware and accurate. Before deciding on an AI tool, clinicians should look into whether the company uses advanced AI techniques like RAG.
2. Review AI Output Carefully
AI may generate notes for you, but you still need to use your own clinical judgment to verify the accuracy of the content. As Bahar puts it, “Models can make mistakes. Therapists are encouraged to go over the suggestions and validate them.” No matter how advanced the system is, it’s not foolproof—and your trained eye is essential in catching errors.
3. Provide Feedback to Improve the AI Model
AI systems are constantly evolving based on user feedback. If something seems off, submitting feedback actually improves the system for everyone. Eleos actively encourages users to report issues and provide feedback so we can continually refine and optimize our AI models.
4. Think of AI as a Tool, Not a Replacement
AI should be viewed as an assistant, not a substitute for your expertise. It can help reduce your workload, sure—but it’s your knowledge and understanding of the client, the session, and the therapeutic process that ultimately determine whether the output is accurate and relevant.
AI can make life easier for therapists, but it comes with its own set of challenges. Hallucinations and bias can compromise the quality of care if therapists aren’t paying careful attention. By understanding how these issues happen—and how tools like RAG can prevent them—therapists can make sure AI works for them, not the other way around.
When therapists stay informed, ask questions, and keep an eye on how these tools are integrated into practice, they help keep the industry safe from AI risks. After all, it’s still the human connection—and the clinical expertise of human providers—that makes therapy effective and meaningful.
And as AI becomes more common in behavioral health, therapists should hold all companies—not just Eleos—to high standards for addressing risks like hallucinations and bias, so the technology can be as safe and reliable as possible for as many people as possible.
Want to learn more about the cutting-edge technology that drives our purpose-built behavioral health AI? Request a demo of the Eleos platform here.
