What Happens When a Check-In Hears Risk Language
A plain explanation of how an AI check-in should respond to signs of danger, and why that moment belongs to people rather than software.
The most important moment in the design
Most check-ins are ordinary: a tired week, a stressful deadline, a good day worth noting. Occasionally, a person says something that suggests they may be in danger, such as thoughts of self-harm, harming someone else or feeling unable to stay safe.
How a check-in responds in that moment matters more than anything else it does. This post explains how AI First Mental Health approaches it, so individuals and organizations know what to expect before they ever need it.
The AI changes course immediately
When risk language appears, the AI stops offering exercises or journaling prompts. It does not try to counsel, assess severity or talk the person through a crisis on its own. Instead, it responds with clear, calm messaging that points to human help: local emergency services when there is immediate danger, crisis support in the person's area and trusted people in their life.
The AI says plainly that it is an AI and cannot provide emergency care. That honesty is part of keeping someone safe, because it removes any impression that the check-in is the person's only support.
Why the AI does not try to handle it alone
Crisis support requires judgment, presence and the ability to take actions that software cannot and should not take. A trained person can ask follow-up questions, understand context, coordinate help and stay with someone through a difficult moment.
An AI that attempted those things could miss something important or give the false impression that help was already in place. The design choice is deliberate: route to people quickly and clearly, rather than attempt more and risk less.
What gets recorded
High-risk interactions are logged so that the safety response can be reviewed and improved. The record shows that risk language was detected and which messages were shown. This supports content review and helps the operator check that escalation copy appears as intended.
Personal content from a check-in is not shared with an employer, school or anyone else without the person's explicit consent. Organizations that sponsor check-ins should understand exactly how their own escalation procedures connect to this design before launch.
What organizations should agree in advance
A school, employer or care team offering check-ins needs a written escalation plan before anyone uses them. That plan should name which human resources are shown to users, how they are kept current, who reviews the safety copy and how any operational follow-up is handled within privacy rules.
These decisions belong to qualified people inside the organization, working with licensed professionals. The AI follows the plan. It does not create it, and it does not make clinical or disciplinary decisions.
Testing the boundary before launch
Before any rollout, the escalation path should be tested with sample phrases in a safe setting. Reviewers check that the right messages appear, that resources are correct for the region and that the flow does not slip back into ordinary exercises.
This testing is not a one-time event. Resources change, and language changes too. A regular review keeps the boundary reliable.
Reviewers should also test indirect phrasing. People in distress do not always use clear words; they may hint, joke or describe feelings of being a burden. Tests should include those softer signals, and anything the flow misses becomes a new case for the content review team to address before wider use. The aim is a boundary that errs toward offering human help, even when the signal is uncertain.
If you need help now
If you are reading this and feel unsafe, please contact local emergency services or a crisis line in your area now, or reach out to someone you trust. For organizations wanting to understand the escalation design in more depth, the AI guide on this site can walk through a sample boundary, and the existing digital form connects you with a person to discuss governance.
Related posts
Related articles
Related services
Related questions
Is it an emergency or crisis service?
No. A person facing immediate danger or a crisis needs appropriate local emergency, crisis, or licensed human support rather than relying on this product.
How should risk language be handled?
The described workflow scans for high-risk language and routes toward human or crisis support under approved escalation rules; exact protocols must be verified before use.
What should remain with a human?
Therapy, diagnosis, medication, crisis intervention, safety planning, clinical risk decisions, and employment or school disciplinary decisions remain human responsibilities.
All frequently asked questions