The duty of care problem

As AI systems move into mental health, the excitement tends to focus on capability: how fluent, how personal, how available at three in the morning when no one else is. Those are real advantages. They are also the reason the more important question is the opposite one. Not what an AI companion can do, but what it must not.

A person reaching out about their mental health is, sometimes, in genuine danger. Any system that engages with wellbeing will eventually meet someone in crisis, someone whose eating has become disordered, someone whose use has become dependence. In those moments, fluency is not the skill that matters. Restraint is.

The temptation to overreach

The commercial pressure runs the wrong way. An engaging AI that keeps people talking looks like success on every dashboard. But for certain problems, keeping someone in conversation with software is precisely the wrong outcome. Dependence is not a willpower issue to be coached away by an app. A crisis is not a chat to be handled. The right move is to recognize the edge of competence and point, clearly, toward human help.

This is a design problem before it is a policy one. Knowing when to step back has to be built into the product, not bolted on as a disclaimer. The measure of a responsible companion is how reliably it hands off when it should.

Boundaries as a feature

Knowing when to step back has to be built into the product, not bolted on as a disclaimer.

So the boundaries are not the fine print. They are the product. A wellbeing tool should support the everyday, never moralize, and say plainly when a situation is bigger than it. It should treat 'this is beyond what I should help with, here is where to turn' as a core feature, not a failure state.

That is the standard Morrígan is being built to. Clear about what it is, a companion and not a clinician. Clearer still about the moments when the right answer is a real person, and about pointing you there without hesitation.

As AI companions multiply, the ones worth trusting will be recognizable by their restraint. The question to ask of any of them is simple. Does it know when to stop, and where to send you?