Stop me if you’ve seen this episode of Care at Home in the City:
An AI pilot launches with real enthusiasm. Clinicians try it. Feedback is collected. And then… it quietly—or not-so-quietly—disappears. No fanfare, no post-mortem. Just a slow fade into darkness. With or without disgruntled clinical staff.
It isn’t just on must-see TV. It’s the reality many home-based care orgs have lived—in the city and the country. Maybe you’ve even been there before.
But we do have some good news. You’re not alone—sand the problem isn’t always the technology, which means we have answers and actionable tips that can really help.
At a recent McKnight’s Home Care AI Forum, Eleos’ Christy Doneff joined Michael P. Johnson—physical therapist, researcher, and executive leader at Bayada Home Health Care—for an honest conversation about why so many AI pilots stall, what the hidden costs of that stalling actually are, and what separates the organizations that successfully scale AI from those that stay stuck in testing mode.
Here’s what they had to say.
“Pilot” Is a Four-Letter Word
Let’s start with the word itself.
When Christy and Michael unpacked why so many pilots fail before they ever become operationally embedded, they both landed on the same diagnosis: The word “pilot” sets the wrong expectations from the start.
“When you have a pilot environment, you’re assuming you can create some level of empirical study,” Christy explained. “But it simultaneously ignores what’s required to effectively bring change across the organization—that unified view, that leadership direction.”
In practice, what happens is a pattern we’ve seen far too many times. Participants know it’s a pilot, so they engage with a kind of passive good faith. They’ll give it some time. They’ll fill out the survey. But they never truly commit—because leadership never truly committed either. The result is what Christy calls a “mirage environment”—one that looks like adoption but isn’t.
There’s the deeper problem with that mirage that Christy identified; pilots are built on a fallacy. They assume you can create a controlled study—a group that has access to the solution and a group that doesn’t. And therein actually lies the problem. A controlled environment, by definition, can’t replicate the unified leadership direction and organizational commitment that a real implementation requires. So what you often end up with is clinicians giving the technology a polite audition rather than a genuine chance—and smile-sheet feedback that tells you very little about whether the solution will actually work at scale.
Michael’s reframe? Stop calling it a pilot. Call it an experiment.
“A pilot means ‘we’re gonna try it and see,'” he said. “An experiment has hypotheses, goals, outcomes—a level of rigor that actually gives you something to learn from and scale.”
That distinction matters more than most organizations realize, and it starts with being honest about context because piloting in your fifteen best offices won’t tell you anything about whether the technology will work in your fifteen hardest ones.
The Hidden Costs of a Pilot
Organizations often treat pilots as the low-risk option. It’s easy to see why. Pilots are usually framed as a low-cost, dip-your-toes-in way to test out new tech. Sometimes they’re even free. Except it turns out they’re often not low cost, and absolutely not free.
The costs accumulate—and they compound. Christy and Michael both pointed to workforce fatigue as the first place it shows up.
“I can’t tell you how many times I’ve been in a training for a new pilot or rollout and there’s just this fatigue,” Christy said. The collective, “Not again,” energy. Clinicians and staff start to feel like they’re being asked to do leadership’s job—evaluating org solutions rather than delivering care. Too many of these cycles, and your staff start looking for the exit signs.
Michael put a sharper point on it:
“Every time you have a poorly designed pilot that doesn’t go well, you’re training your entire workforce to have a lower trust in you the next time around.” There’s actually a clinical term for this—change fatigue—peer-reviewed research links it to increased burnout and, critically, intention to leave.
Michael P. Johnson
In a sector where turnover can hit 30–40% and home health organizations are fighting to retain every nurse, aide, and therapist they have, that’s not an abstract risk. It’s a direct threat to your ability to deliver care.
Michael’s advice is simple: “Stop shoveling.” Stop digging the hole deeper before you can’t get out.
Where Pilot Friction Actually Lives
So what’s the literal breaking point? When pilots sputter and stall, where does the breakdown usually happen?
Both speakers agreed: You feel it first in the operational layer, but the root cause is almost always a leadership issue.
“Frontline clinicians and office managers are really good at identifying when something isn’t working and why,” Michael said. “What oftentimes happens is we don’t build in enough feedback—and the feedback we do get is after the fact.”
That after-the-fact approach is the tell. If you’re asking “How did it go?” at the end of a pilot rather than “How is it going?” throughout, you’ve already missed the window to course correct. By the time issues surface in a retrospective, the clinicians have already updated their understanding: The next one probably won’t matter either.
The fix isn’t complicated, but it does take intention. Build in focus groups. Engage staff across functions and technology skill levels. Create psychological safety so frontline staff can tell a room full of leadership exactly what isn’t working—and mean it when you thank them for it. And bring your technology partner into those conversations, or at least share the recordings. The candid feedback that happens in those rooms is where real product improvements get made.
One mindset shift Christy flagged that stuck out: The best organizations she’s worked with don’t use the word “vendor.” They say “partner.” It’s subtle—but it signals something real about how much accountability is being shared, and how likely the relationship is to survive its first hard conversation.
Ready to read more? Peep the Complete Guide to Behavioral Health AI Implementation or 8 Simple Rules for Rocking Your AI Rollout.
What Success Looks Like
So what does it take to actually cross from experiment to operational deployment?
Christy pointed to two non-negotiables:
1. Leadership alignment that goes beyond buy-in.
It’s not enough for leaders to know the implementation is happening. Org leaders need to be able to articulate the value—to say, clearly, why this investment is being made and what it’s expected to change. That’s a meaningfully higher bar than “leadership approved it.”
2. Minimal workflow disruption.
“Workflows are broken,” Christy acknowledged. “They’re meant to be improved upon. But too much change, and you will shock your system so significantly that you will not see the scale.” The technology that wins long-term is the technology that fits into how people actually work—not the technology that forces people to work around it. Parallel processes (the Excel spreadsheet someone keeps on the side “just in case”) are a dead giveaway that true adoption hasn’t happened.
Michael added a third lens to consider; a rigorous evaluation framework built around four dimensions—clinical validity, workflow integration, data governance, and implementation science.
Clinical validity asks whether the intervention has actually been shown to improve clinical outcomes in patient populations similar to yours—ideally through peer-reviewed evidence or documented real-world results, not just vendor-provided case studies.
Workflow integration is where most implementations either succeed or break down. How does the technology interact with your existing clinical documentation and care coordination? A seamless fit isn’t just a nice-to-have; it’s the primary test of whether adoption will actually happen. When workflows get “moved,” friction follows—and friction kills scale.
Data governance addresses who owns the data, how it’s used, and what the privacy obligations are. In home health and hospice—highly regulated environments handling sensitive clinical and personal information—this is non-negotiable due diligence, not a checkbox.
Implementation science is the one that gets skipped most often, and it’s the one that determines whether the first three actually stick. “If your technology partner gives you a login number and a training manual and says ‘good luck,’ that’s not an implementation plan,” Michael said. There’s an entire body of research on how to operationalize new tools effectively—your partner should be fluent in it.
Early indicators that you’re on the right track:
- Voluntary adoption—clinicians who have it are telling colleagues they want it too
- Inclusive language—your team is calling it “our tool,” not “their product”
- Middle managers raising issues as signals rather than noise
- Real-time feedback loops that celebrate wins and identify opportunities
Want to kick the pilot habit for good? Request a demo and experience Eleos today!
When It Works, the Impact Extends Beyond Efficiency
When organizations get the rollout right—when the technology becomes ubiquitous, woven into the daily workflow rather than layered on top of it—the returns go well beyond time savings.
Before we get to the numbers, it’s worth distinguishing between two types of return: ROI (return on investment) and VOI (value on investment). ROI is what you can model in a spreadsheet—time saved, claims recovered, services added. VOI is harder to quantify but arguably more important: the change in how your providers feel about their work, how long they stay, how present they are with their patients. In lean-margin environments like home health and hospice, the instinct is to lead with ROI. But the organizations seeing the biggest ROI are often the ones that led with VOI first.
Christy shared results from a behavioral health organization in Kentucky that used Eleos to support a move to open access care. The ROI wasn’t the starting assumption; it was a byproduct of doing right by their providers. By reducing documentation burden and supporting real-time quality and compliance review, they freed their clinicians to do more of what they came to do.
The outcomes followed:
- $173,000 in additional billable services
- 25% more services delivered
- 12% more unique clients per clinician
- 15% increase in clinician productivity hours
- 16% increase in client engagement and length of stay
- 5% increase in daily intakes
None of these were the stated KPIs going in. They were what happened when the technology worked as a true amplifier—getting people back to operating at the top of their intrinsic why.
Michael framed it around three domains he tracks at Bayada: patient outcomes, workforce outcomes, and organizational outcomes. When technology reduces the documentation burden that consumes 30–40% of a clinician’s day, everything else has room to improve. “If you give them the space,” he said, “they can do that. I think we get in their way.”
One Thing to Stop. One Thing to Start.
Stop thinking that pilots or out-clauses are your only financial protection. They’re not. They’re a hedge that often ends up costing more—in workforce trust, in implementation momentum, in relational capital—than the risk they were meant to manage.
Start thinking about what you can offer your technology partner in exchange for performance commitments. References, case studies, access to real outcomes data for peer review—these are things a serious partner values deeply. Put them in the contract, gated to results. Now both sides are motivated.
“If you start thinking in the back of your mind, ‘I don’t know if I’d even recommend talking with this leadership team again’—say something,” Christy said. “Those are early indicators that the partnership is no longer performing as a partnership.”
The Critical Technology Implementation Takeaway
If you only gain one nugget from this webinar, it is this.
Technology (including AI) won’t scale in home-based care through better technology alone. It scales through better partnerships, better rollout design, and leaders willing to hold themselves as accountable as they hold their technology partners.
The organizations getting this right aren’t approaching it as a procurement decision. They’re approaching it the way Michael described when he walked into office circles across the country and told four levels of bosses and frontline clinicians the same thing: We’re here because you’re doing really cool stuff, and we all want to hear it so we can help you solve it.
That’s the posture that turns an experiment into something that works. Eleos offers home health orgs the opportunity to experience a system of action that unites the care journey without disrupting workflows. Ready to talk? Request a personalized demo today.
