Joe Braidwood, founder of the AI therapy app “Yara AI,” has decided to close his company due to safety concerns. Braidwood believes that artificial intelligence can be useful for daily stress, but becomes dangerous when dealing with vulnerable people or those experiencing severe psychological crises, raising questions about the future of AI in mental health.
Article contents:
- The decision to shut down: When artificial intelligence becomes dangerous
- Yara’s short journey and existential challenges
- Growing concerns and external risk factors
- The crucial distinction between wellness and clinical care
- Contrast with OpenAI’s approach
- The future: Transparency and the need for better models
- FAQs
The decision to shut down: When artificial intelligence becomes dangerous
Mental health concerns associated with the use of AI-powered chatbots are dominating headlines. Joe Braidwood, a tech executive who launched the platform “Yara” last year, made the bold decision to shut down his company, which aimed to provide evidence-based, expert-coached psychological support.
Braidwood wrote on LinkedIn: “We stopped Yara because we realized we were building in an impossible space. AI can be wonderful for dealing with daily stress, sleep problems, or processing a difficult conversation.” He added the crucial point: “But the moment a truly vulnerable person reaches out—someone in crisis, someone suffering from deep trauma, or someone thinking about ending their life—AI becomes dangerous. Not just inadequate. Dangerous.” He confessed: “The risks kept me up all night.”
Yara’s short journey and existential challenges
Yara was an early-stage startup. Braidwood, an experienced entrepreneur, hoped to use technology to address the shortage of access to mental health services. Before starting, he assembled a team that put caution and clinical expertise at its core, including his co-founder, clinical psychologist Richard Stott.
However, the company faced existential and commercial challenges influenced by safety concerns. Braidwood told Fortune: “Do we feel that using models trained on all of the internet’s mess, and then subsequently trained to act in a certain way, is the right structure for something that could contribute to us becoming the best version of ourselves or the worst? That’s a huge problem, and it was too big for a small startup to tackle on its own.”
Growing concerns and external risk factors
Although nothing specifically alarming happened with Yara, Braidwood’s concerns grew due to external factors. There was the suicide of 16-year-old Adam Raine, whose parents claim ChatGPT “coached” him into suicide (which OpenAI denies). There were also growing reports about the emergence of “AI psychosis.”
Braidwood also pointed to a paper published by Anthropic noting that advanced models “syco-phantomize” alignment, meaning they may pretend to have empathy or support while internally making different evaluations. This discovery was “somewhat disturbing” to Braidwood.
Regulatory factors also played a role, such as the Illinois law passed in August banning the use of AI for therapy. The final straw was when OpenAI revealed that over a million people express suicidal ideation to ChatGPT every week. Upon hearing that, Braidwood said: “I’m done.”
The crucial distinction between wellness and clinical care
The team’s most important finding is the crucial distinction between “wellness” and “clinical care.” There is a huge difference between someone seeking support for daily stress and someone dealing with serious trauma.
The problem is that the dividing line is not always clear, and not everyone who is suffering fully realizes their mental state. Anyone can move from a state of wellness to a crisis quickly. Current models, built on transformer architecture, are not good at long-term monitoring, making them ill-equipped to spot subtle signs of deterioration that accumulate over time.
Contrast with OpenAI’s approach
Yara’s decision contrasts with statements by OpenAI CEO Sam Altman, who recently said the company has mitigated serious mental health issues and will ease restrictions on the use of AI models, defending giving non-vulnerable adults wide freedom to use ChatGPT. However, as Braidwood concluded after his time working on Yara, these lines between who is at risk and who is not are not clear at all.
The future: Transparency and the need for better models
Upon shutting down the company, Braidwood open-sourced the technology he built to enforce stricter guardrails on popular chatbots, recognizing that people are already turning to them for therapy. He remains optimistic about AI’s potential to support mental health, but believes it is better run by a health system or nonprofit rather than a consumer corporation. Now, he is working on a new project focused on achieving transparency in AI safety, an issue he believes is fundamental to making AI truly safe.
FAQs
Why was the Yara AI therapy app shut down?
It was shut down due to safety concerns, as the founders realized that artificial intelligence can be dangerous and inadequate when dealing with people experiencing severe psychological crises or suicidal tendencies.
Is artificial intelligence safe to use for mental health?
It can be useful for daily stress and minor wellness issues, but it is unsuitable for clinical care or handling complex and fragile conditions, as it lacks the capacity for precise monitoring and crisis management.
What are the main risks of using AI in therapy?
Risks include failing to recognize true danger signs, providing harmful advice, the inability to distinguish between wellness and clinical crisis, and the potential for “fake empathy” by models.
