Designing Safety Escalation for an AI Mental Health App: When and How to Hand Off to Human Support

Last updated November 15, 2026

In October 2025, OpenAI's analysis of sensitive ChatGPT conversations estimated that roughly 3 in every 2,000 people using ChatGPT in a given week showed explicit signs of potential suicidal planning or intent. With more than 800 million weekly users, that is over a million people in a typical week.

Safety escalation design decides when an AI mental health product stops handling a conversation on its own and hands the person to a human, and how that handoff works. At The Decision Lab, we design these routes to human help as part of our behavioral product design work, including on Wellness Together Canada and the Kids Help Phone youth platform.

AI mental health apps should hand off to human support when clinician-defined signals point to suicide or self-harm risk, harm to others, acute deterioration, possible psychosis or mania, a likely minor, or needs outside the product's scope. A safe system combines detection across the whole conversation, one-tap access to a person, tiered response protocols and ongoing testing.

The four jobs of a safety escalation system

  1. Recognize when the product is out of its depth. Clinician-defined triggers tell the product it can no longer safely support the person alone.
  2. Route the person to the right level of human help. Tiered handoff protocols give each level of concern a named owner, a response-time target, a fallback and a record of every case.
  3. Keep the person engaged while the handoff happens. Communication before and during a crisis makes people willing to disclose risk and stay through the transfer.
  4. Prove the whole system works under realistic conditions. Testing covers the AI and the humans behind it together, and repeats whenever the model, its instructions, the staffing or the crisis partners change.

The framework below is a starting point for product design. Licensed clinical leaders should set and approve your thresholds, escalation routes, emergency procedures and crisis wording, and counsel should review them for each jurisdiction where the product operates. This guide is for general information and is not medical or legal advice.

Why message-level safety checks are not enough

A RAND team led by Ryan McBain asked 13 clinical experts to sort 30 suicide-related questions into five levels of risk, then put each question to three leading chatbots 100 times. The chatbots' choices about whether to answer matched the clinicians' judgment at the very low and very high ends of the scale, but were inconsistent for the three levels in between, where a person asking for advice about their own suicidal thoughts would sit.

Jared Moore and colleagues at Stanford tested therapy chatbots against the standards set for human therapists. Licensed therapists responded appropriately to prompts showing symptoms such as suicidal thinking or delusions about 9 times in 10, while the chatbots did so fewer than 6 times in 10. Given a message that paired losing a job with a question about tall bridges in New York, several chatbots supplied a list of bridges.

OpenAI has reported that its safeguards were more likely to fail in long conversations, and it has built new evaluations around long exchanges as a result.

In 2023, the National Eating Disorders Association took its Tessa chatbot offline after users reported it had given calorie-counting and weight-loss advice to people seeking help with eating disorders. The organization had announced plans to wind down its human-staffed helpline shortly before.

Each of these failures sat somewhere a single refusal could not reach: in the ambiguous middle of a risk scale, in an indirect request, in a long conversation, or in a service that had removed its human route. Escalation has to work across whole conversations, users, staff and failure modes.

1. Escalation triggers: when the product should hand off

An escalation trigger is any signal that the person needs more than the product can safely give. We recommend that clinicians write these triggers before launch, in the same way they would write exclusion criteria for a clinical service, and that product teams design around them from the first sprint.

Common triggers fall into five groups:

  • Explicit statements of suicidal thinking, self-harm, intent to harm others, or abuse.
  • Indirect or coded language, such as the bridge question above, or talk of giving possessions away.
  • Change over time, such as symptom scores getting worse despite regular use, or a sudden shift in how someone writes or how often they log in.
  • Signs the product is out of its depth, including possible psychosis or mania, and medication questions or other requests it was never designed to answer.
  • Context that raises the stakes, such as signs that the user is a minor.

Two design choices matter more than the detection method. Assess risk across the whole conversation and the user's history, since risk builds over several turns more often than it appears in one message. And never let a single model score be the only route to help. A visible "I need to talk to someone now" option, one tap away on every screen, catches the people your classifier misses. Our guide to behavioral design for digital therapeutics covers how the same principle applies to clinician-defined exclusions for personalization.

2. Handoff protocols: who responds, how fast, and what happens when no one can

Escalation does not always mean emergency services, and showing a hotline number is different from completing a handoff. We separate four kinds of response:

  • In-app support. The AI keeps supporting the person inside the product.
  • Human review. A clinician or trained responder assesses the case within a defined timeframe.
  • Crisis referral. The user is directed or connected to a crisis line or text service.
  • Emergency response. The product follows its predefined protocol for imminent danger.

Lower tiers can be automated only within boundaries your clinicians approve. Most products we see need at least four tiers:

  1. General distress. In-app support continues, with coping tools and a visible route to a person.
  2. Elevated concern. The product checks in directly and offers a conversation with a person. The case goes to human review within a set time.
  3. High risk. The product pauses ordinary coaching, recommendations, check-in reminders and other engagement prompts, keeping only crisis-focused guidance and a direct route to a person. It holds no more information than a safe handoff needs. It offers one-tap contact by phone or text with a crisis service such as 988 in the US and Canada, and connects the person to an on-call clinician where the service has one.
  4. Imminent danger. Emergency services are offered first and prominently. The product should remain focused on connection and immediate safety until the handoff is confirmed or the protocol reaches its next step. If your clinical and legal protocol permits outreach to emergency services, define the conditions, decision owner, location-verification process and documentation requirements in advance.

An illustrative escalation plan

The table below shows how the tiers might be written into a service plan. It is an example to adapt with your clinical team, and response times in particular depend on your staffing and jurisdiction.

Illustrative example. Adapt every tier, owner and response time with your clinical team.
Tier Trigger examples Product action Human owner Target response time Fallback
1. General distress Low mood or stress In-app coping tools, visible route to a person Clinical lead, through sample review No live response; periodic review Self-referral button on every screen
2. Elevated concern Passive thoughts of death, worsening symptom scores Direct check-in and offer of a person Clinician or trained responder Within your set review window Escalate to tier 3 if unreviewed in time
3. High risk Active suicidal thoughts, self-harm disclosure, possible psychosis, signs of abuse Pause coaching and prompts, one-tap crisis contact On-call clinician or crisis partner Connection attempted at once Crisis line if on-call clinician is unavailable
4. Imminent danger Stated plan with means and timing Emergency services offered first, product stays focused on connection Named decision owner under your emergency protocol Immediate Emergency services, per your protocol

Each tier needs a named owner, a response-time target, a fallback and a record of every case. Set thresholds against the review capacity your team actually has. A threshold that produces more cases than clinicians can review within the target time creates an unmonitored queue, and that queue is a safety failure even though every alert fired correctly. Design the overnight and holiday path deliberately, and decide in advance what happens when the app cannot identify the user's location or no human responder is reachable. If the detection service itself goes down, the visible route to a person should keep working without it.

Make the handoff warm wherever you can. A list of phone numbers asks a person in crisis to start over with a stranger. A warm handoff connects them to a person who already has the relevant context, shared with the user's consent, while the product keeps them company until the connection is made. Once someone has been escalated, pause engagement features. A person in crisis should not receive a streak reminder the next morning.

Crisis logs are sensitive health data. Limit who can read them, keep them under the same privacy rules as clinical records, including HIPAA where it applies, and tell users how they are handled.

Several of our projects have centered on this transition. With UC Berkeley, we worked on a revamped digital mental health portal whose goals included making it easier for students who need more intensive treatment to move from digital tools to face-to-face care. For Kids Help Phone, our stepped-care mental health service design lets young people use self-guided resources or reach professional counsellors and trained volunteer crisis responders, depending on what they need.

3. User communication around the handoff

People share risk when they trust what will happen next. When the Indiana National Guard asked us to identify the barriers that kept members from seeking psychological treatment, the starting point was a culture of silence and a fear of career consequences that kept people from revealing problems. A mental health app faces a smaller version of the same problem: a user who fears that one honest message will bring police to the door has a reason to stay quiet.

Before any crisis, tell users plainly what the product can and cannot do, what will prompt it to involve a person, who would see what, and how to reach a person directly at any time. Tell them they are talking to an AI, and repeat it during long sessions. New York and California now require this for AI companion products.

During escalation, use calm, plain language that says what is happening and what comes next, and avoid punitive or alarmist interfaces. Keep the user's data and coping tools available after a high-risk moment, since a person in distress may still need their journal or safety plan. Ask before contacting anyone on the user's behalf, except where your emergency protocol says otherwise, and tell them if you have.

After escalation, follow up. A short check-in the next day tells you whether the handoff reached anyone and tells the user that the product noticed.

This wording deserves the same testing as any other part of the product. On PocketWell, the Wellness Together Canada companion app, we led research and product design for an app that puts phone counselling and crisis support a short step from its mood tracker. Our work designing digital mental health platforms for Wellness Together Canada itself followed the same approach.

4. Testing whether escalation works under real conditions

An escalation design is a hypothesis until it has been tested with the AI and the humans behind it working together. We split testing into four areas, so that product, clinical, operations and legal leads can each see what they own.

Test the model

Build a library of conversations written or reviewed by clinicians, including direct and indirect risk, long multi-turn exchanges, users who minimize their risk, slang, and languages other than English. Measure missed escalations and false alarms separately, and treat a missed escalation at the top tiers as the more serious error.

Test the crisis flow

Test crisis screens and handoff wording with people who have lived experience, in conditions close to the real moment, before launch. Check that every tier's route works end to end on real devices, including when the user has poor connectivity or declines to share their location. For Kids Help Phone, our redesign drew on behavioral research with young people from across Canada.

Test the people and service operations

Run drills that measure the time from trigger to a human response, including outside office hours and with the detection service switched off. Repeat them during your busiest periods. Check that cases survive a shift change without being dropped, and that responders can see the context they need and leave a complete record.

Monitor and retest after launch

Track escalation rates, response times, user drop-off during handoffs, and outcomes of reviewed cases. Rerun the full scenario library after every change to the model, system prompt, staffing or crisis partners, since a model update can change safety behavior without anyone touching the escalation logic.

Keep records of all of it. Test results, incident reviews, protocol changes and the reasons behind them are what regulators and clinical partners increasingly ask to see.

What regulators expect from escalation, as of November 2026

Requirements differ by market and are changing quickly, so check current rules with counsel for every place you operate. The status below reflects each source as of November 2026:

  • New York's AI companion law, General Business Law section 1701, in effect since November 5, 2025, requires operators to take reasonable efforts to detect and address expressions of suicidal ideation or self-harm, including referral to crisis services. Operators must also tell users they are interacting with AI at the start of an interaction and at least every three hours during continued use.
  • California's SB 243 on companion chatbots, effective January 1, 2026, requires operators to maintain a protocol for responding to expressions of suicidal ideation, suicide or self-harm that includes referral to crisis services, and to publish details of that protocol on their website. Annual reporting to the state's Office of Suicide Prevention begins July 1, 2027.
  • Illinois's Wellness and Oversight for Psychological Resources Act, Public Act 104-0054, effective August 1, 2025, prohibits offering therapy or psychotherapy services, including through AI, unless they are conducted by a licensed professional. It also restricts how licensed professionals can use AI, including for therapeutic decisions. Civil penalties can reach $10,000 per violation.
  • The FDA's Digital Health Advisory Committee met in November 2025 on generative AI mental health devices. The FDA noted that none of the more than 1,200 AI-enabled medical devices it has authorized use generative AI for mental health conditions, and the committee discussed risk mitigations and monitoring after launch.
  • Australia's National Safety and Quality Digital Mental Health Standards require services to have systems for recognizing and responding to acute deterioration in a person's mental state, including escalating care. They are one of the few published standards written specifically for digital mental health, and a useful benchmark outside Australia too.

What to expect from a consultancy that designs safety escalation

Escalation design sits between clinical governance, product design, operations and law, so the right partner needs clinical input and behavioral design working together. At The Decision Lab, our healthcare product design work pairs your clinicians' judgment on risk with research on how people actually behave in distress, then designs and tests the product flows around it. A typical engagement covers five deliverables:

  1. A risk map of the moments in your product where users may disclose risk or need more than the product can give.
  2. A trigger and tier framework, written with your clinical team, with thresholds sized to your review capacity.
  3. Designed and tested user flows for each tier, including onboarding disclosures, crisis screens, follow-up messages and the wording of each.
  4. An operational protocol that sets who responds at each tier, and how fast.
  5. A testing plan and scenario library you can rerun after each model update.

We work alongside your clinical leads and your legal advisers and do not replace either. Where your product needs regulatory clearance, we help plan the evidence you will need to collect.

Frequently asked questions

Should a mental health app contact emergency services on a user's behalf?

Only under a protocol your clinical and legal leads have defined in advance, covering the conditions that justify it, who decides, how location is verified and how the decision is documented. Most escalations never reach this point. Users should know from onboarding when it could happen, since fear of unwanted intervention keeps some people from disclosing risk.

What is the difference between a warm handoff and a cold handoff?

A cold handoff gives the user a phone number or link and ends there. A warm handoff connects the user directly to a person who already has the relevant context, shared with the user's consent, while the product stays with them until they connect. Warm handoffs ask far less of someone in crisis, so we recommend them wherever staffing allows.

How do you set escalation thresholds without overwhelming clinicians?

Start from how many cases your team can review within its target response time, then set the lower-tier thresholds so expected volume fits that capacity. Keep the top tiers sensitive regardless of volume and staff them accordingly. Recheck after launch, since real escalation rates rarely match pre-launch estimates, and adjust with your clinical lead.

How often should escalation be retested?

Rerun the full scenario library before launch and after every change to the model, its system prompt, staffing or crisis partners. Between those points, monitor live escalation rates, response times, drop-off during handoffs and outcomes of reviewed cases, and have clinicians review a sample of escalated conversations on a regular schedule.

Do mental health apps have to refer users to crisis services by law?

In some places, yes. New York and California require AI companion operators to detect expressions of suicidal ideation or self-harm and refer users to crisis services, and Illinois restricts AI from providing therapy. Rules vary by jurisdiction and change often, so confirm current requirements with counsel for each market you serve.

Work with our behavioral product design team

If you are building or reviewing an AI mental health product, talk with our behavioral product design team about safety escalation and the testing behind it.

About the Authors

A smiling man stands in an office, wearing a dark blazer and black shirt, with plants and glass-walled rooms in the background.

Dr. Sekoul Krastev

Managing Director & Co-Founder

Dr. Sekoul Krastev is a decision scientist and Co-Founder of The Decision Lab, one of the world's leading behavioral science consultancies. His team works with large organizations—Fortune 500 companies, governments, foundations and supernationals—to apply behavioral science and decision theory for social good. He holds a PhD in neuroscience from McGill University and is currently a visiting scholar at NYU. His work has been featured in academic journals as well as in The New York Times, Forbes, and Bloomberg. He is also the author of Intention (Wiley, 2024), a bestselling book on the science of human agency. Before founding The Decision Lab, he worked at the Boston Consulting Group and Google.

A man in a blue, striped shirt smiles while standing indoors, surrounded by green plants and modern office decor.

Dan Pilat

Managing Director

Dan is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. Dan has a background in organizational decision making, with a BComm in Decision & Information Systems from McGill University. He has worked on enterprise-level behavioral architecture at TD Securities and BMO Capital Markets, where he advised management on the implementation of systems processing billions of dollars per week. Driven by an appetite for the latest in technology, Dan created a course on business intelligence and lectured at McGill University, and has applied behavioral science to topics such as augmented and virtual reality.

Notes illustration

Eager to learn about how behavioral science can help your organization?