A few years ago, some dear colleagues of mine built an AI-powered chatbot to support people with eating disorders. It was a closed system with carefully designed, rule-based responses. That way, it would only provide safe, accurate, and compassionate advice—always.
After a while, without consulting the clinicians who built the system, the company allegedly enabled generative AI responses crafted by models that pulled from general data sources.
Suddenly, the chatbot started spitting out standard dieting advice to clients struggling with their body image. I still cringe just thinking about it.
But whenever I talk about bias, I like to tell this story—because it so compellingly proves that:
- AI is only as good as the data it’s trained on, and
- subject-matter experts must be involved in all steps of AI design and implementation.
If that data is “the entire internet,” then we shouldn’t be surprised when the AI does something as foolish as giving dieting advice to someone with an eating disorder. And if professionals who have worked with the target audience their entire lives are not actively involved every step of the way, behavioral health AI can go deeply wrong.
AI can do a lot of good in behavioral health—but only if it doesn’t learn the wrong ideas from the wrong sources.
The Foundational Problem of Bias in AI
Therapists are highly attuned to anything that might negatively impact their clients, and they’re starting to catch on to the problem of bias in AI. Just the other day, I gave a talk on AI in mental health at UC San Francisco. Before I even got to my slides on bias, the therapists in the audience were already bringing it up in their questions. Clearly, awareness around the issue of AI bias is growing—and so is the concern.
Bias isn’t just an AI side effect; it’s “baked into” the way large language models (LLMs) are constructed. This means we have to address it head-on, not as an afterthought.
One significant source of bias lies in the dataset itself. We face an issue with non-diverse datasets that overlook specific communities, especially marginalized ones. Mental health data, by their very nature, are private and rarely shared openly. This limitation means algorithms often end up being trained on data from non-representative sources—like social media posts or public websites—that don’t capture the true clinical realities of behavioral health.
A lack of accurate representation in AI training data can lead to AI outputs that don’t account for the nuances of different communities. As a result, the data tends to misrepresent those who are already the most likely to be misrepresented. For mental health, where marginalized communities already have greater risk and fewer resources, bias is a problem we can’t afford to ignore.
The Need for Transparency in AI Algorithms
You trust your doctor in part because you know they understand the science behind the medicine they prescribe. They know exactly how it works and why it’s recommended for patients with certain diagnoses or symptoms. The same should be true for therapists using AI tools: the tools need to be transparent enough for clinicians to know exactly how they arrive at conclusions.
If an AI tool flags an issue or suggests an intervention, therapists should be able to see the logic behind that recommendation and decide whether it aligns with their clinical judgment. After all, AI tools don’t have licenses to practice therapy—therapists do.
When providers can follow the reasoning behind AI-generated insights, they can spot potential biases embedded in the data or the algorithm itself. Without this, biases may go unnoticed and end up harming clients, especially those from already-marginalized communities.
Clients also deserve to know how AI is being used in their treatment. When therapists can confidently explain how an AI tool works, it builds trust and reinforces the idea that the AI is there to assist, not replace, the human connection at the heart of behavioral healthcare.
During my talk at UCSF, I kept getting questions on how to make sure AI supports therapists without taking over the therapeutic process. This points to why transparency matters so much. Clinicians need to know that the tools they use are allies designed to support their work (rather than override it) before they can trust the tools with the care of their clients. Transparent technology is a safeguard against bias—one that builds trust between therapists and the tools they use.
Diverse Leadership as a Check on AI Bias in Healthcare
Preventing AI bias in behavioral healthcare starts with making sure the right people are in the room when decisions about building, training, using, and fine-tuning AI are made. Leadership teams need to include diverse voices—mental health providers, women, nonbinary individuals, and people of color—because without them, important ethical considerations get missed.
Clinicians play an especially crucial role here. Too many companies in the mental health tech space don’t have providers at the table when key decisions are made, which means their choices overlook the ethical and practical realities therapists face every day.
But it’s not just on tech developers and leadership teams to keep AI bias in check. End users—therapists and clients—have an important part to play, too. When using AI tools, therapists must stay vigilant for signs of bias—and say something when they see something.
Eleos’ Approach to Mitigating Bias in AI
I know firsthand how difficult it can be to address bias when creating AI technology—it’s something we have to keep a constant focus on here at Eleos. Bias isn’t something you fix once and move on; it requires continuous attention and adaptation.
One of the ways we tackle bias at Eleos is by using data that reflect the true scope of diversity found in behavioral healthcare. We draw from a wide range of therapy sessions across many different locations and demographics, integrating over 500,000 sessions to make sure our models are inclusive and representative. That’s how we can be confident that our tools will work for a broad spectrum of clients and providers, not just a narrow subset.
User feedback is also essential. Many of our learnings and improvements have come directly from customer feedback.
We may be an AI company, but we don’t outsource the feedback process to AI; a human being reviews every single product satisfaction rating and user message, and directs them to the appropriate subject-matter expert to resolve any issues raised.
We want to stay connected to the real-world experiences of the providers and clients we serve. Their voices always have been—and always will be—the guiding force behind the development of our technology.
The Flipside: How AI Can Actually Improve Care Equity
I’d be remiss not to mention that AI can also help combat some of the biases that lead to inequitable care. In fact, it already is.
By increasing access to care in underserved areas, providing evidence-based guidance tailored to each individual client’s background, and elevating equitable processes and standards for clinical research, AI can support therapists in delivering more consistent, unbiased treatment to every client. For example, telehealth tools powered by AI are helping bridge care gaps in rural and low-resource areas, so that underserved people have access to care they might not otherwise have. AI can also act as a counterbalance to human bias by giving impartial recommendations rooted in data, which helps all clients receive fair and equitable care.
This is exactly why addressing bias is so important—it’s the only way to unlock AI’s ability to support the diverse needs of behavioral health providers and their clients.
Bias is the built-in fault line in human thinking as well as in AI. To keep it from harming clients and undermining progress, tech companies need humility and a willingness to adapt, and therapists need to stay educated and aware. For companies, addressing bias effectively means employing a diverse team who can ensure we question ourselves, listen to feedback, and make changes when needed. For providers, addressing bias means being mindful of the tools they use, supervising AI carefully, and giving feedback honestly.
The goal isn’t just building AI that works, but building AI that works ethically and inclusively. When we commit to these values, we create a culture where technology serves people, not the other way around—and that’s exactly the kind of future we should be striving for in mental health tech.
To learn more about all the ways Eleos is driving responsible AI development and use, request a demo of our purpose-built platform.
