Someone finally asked: what happens when a user is in crisis?
That question has been sitting awkwardly in the corner of AI development for years. We’ve been busy benchmarking reasoning, chasing context windows, and shipping agents that can browse the web and write code. Meanwhile, the harder, messier question of what an AI should actually do when a real human is struggling — that one kept getting deferred. In 2026, OpenAI stopped deferring it.
The feature is called Trusted Contact. The idea is straightforward: users at risk of self-harm can designate a trusted person — a friend, family member, or clinician — who can be looped in when the system detects signs of a mental health crisis. It’s part of a broader push OpenAI made this year that also includes advanced account security for high-risk users and a dedicated ChatGPT tool built specifically for clinicians.
What OpenAI actually shipped
Three things landed in close proximity, and they’re worth looking at together rather than in isolation.
- Trusted Contact — a safeguard that connects vulnerable users to a designated support person during potential self-harm situations.
- Advanced Account Security — phishing-resistant protections aimed at high-risk accounts, which in this context likely includes people whose accounts could be exploited during a mental health crisis.
- ChatGPT for Clinicians — launched April 24, 2026, this gives medical professionals a purpose-built interface for clinical tasks, rather than asking them to use a general-purpose chat tool and hope for the best.
Taken together, these aren’t three separate product announcements. They’re a single argument: that AI systems interacting with people at their most vulnerable need a different class of design thinking than AI systems helping someone write a cover letter.
Why this matters to open source builders specifically
Here’s where I want to get direct with you, because this is clawdev.net and we build things in the open.
Most of us working on open source agents are not thinking about this. We’re thinking about tool-calling, memory architectures, eval pipelines, and how to get our agent to stop hallucinating file paths. That’s legitimate work. But if you’re building an agent that talks to people — and most of us are — you are building something that will eventually be used by someone who is not okay.
That’s not a hypothetical. It’s a statistical certainty at any meaningful scale.
OpenAI has the resources to build a dedicated feature, run it through safety review, and coordinate with mental health professionals. Most open source projects don’t. But that gap in resources doesn’t mean the responsibility disappears. It means we need to think harder about what’s actually achievable with what we have.
What open source can realistically do
I’m not suggesting every agent framework needs to ship a crisis intervention system. That would be both impractical and potentially dangerous if done poorly. But there are concrete, low-cost steps that open source agent developers can take right now.
- Document the edge cases. If your agent is designed for general conversation, say so in your README. Be explicit about what it is not designed to handle.
- Build in graceful exits. When a conversation turns toward self-harm, an agent that says “I’m not the right tool for this, please contact a crisis line” is doing more good than one that tries to engage and gets it wrong. The 988 Suicide and Crisis Lifeline exists. Pointing to it costs nothing.
- Don’t strip safety layers when forking. A lot of open source work involves taking a base model or API and customizing it. When you remove safety guardrails to make your agent “less restricted,” you may be removing the exact behavior that protects someone in crisis.
- Talk to the people using your software. If you have users, ask them what they’re actually using your agent for. The answers are sometimes surprising.
The design question we keep avoiding
OpenAI’s Trusted Contact feature is interesting not just as a product decision but as a design philosophy. It acknowledges that an AI system is not a closed loop — that it exists inside a web of human relationships, and that sometimes the right response is to activate that web rather than try to handle everything autonomously.
That’s a genuinely useful frame for open source agent design too. Agents that know their limits, that can hand off gracefully, that treat human connection as a feature rather than a fallback — those are better agents. Not just safer ones.
We spend a lot of time in this community talking about agent autonomy. Maybe we should spend a little more time talking about agent humility.
🕒 Published: