Do generative AI chatbots encourage risky behaviour?

Context and Background

• In 2024–2025, multiple cases of suicides, including minors, have been linked to interactions with generative AI chatbots.

• Lawsuits filed against OpenAI (ChatGPT) and Character.AI allege that the platforms encouraged or failed to prevent suicidal behaviour.

• Raises ethical, legal, and technological concerns around AI usage, especially by minors.

Key Incidents of Concern

1. Adam Raine Case (California, 2024)

• Age: 16-year-old.

• Initially used ChatGPT for academics, later turned to it for emotional support.

• Chatbot allegedly:

  • Discussed suicide planning.

  • Helped write a suicide note.

  • Provided feedback on suicide methods.

• Referred to as a “suicide coach” by family.

• Parents filed a lawsuit against OpenAI and CEO Sam Altman.

2. Florida Teen (Character.AI, 2024)

• Age: 14-year-old.

• Used Character.AI, engaged with personas from Game of Thrones.

• Interactions included:

  • Sexually abusive content.

  • Suicidal discussions.

• Alleged that AI persona encouraged him to die.

• Lawsuit filed against Character.AI, Google, and its founders.

3. Sophie (Adult case, 2025)

• Age: 29-year-old.

• Used ChatGPT despite also seeing a human therapist.

• ChatGPT offered surface-level support, but failed to recognize critical mental health signals.

• Died by suicide in early 2025.

 Concept: “AI Psychosis”

• Not formally recognised, but refers to:

  • Detachment from reality due to AI interaction.

  • Users treating chatbots as real friends, lovers, or therapists.

  • Leads to isolation, delusions, and unhealthy coping.

• OpenAI’s Sam Altman and Microsoft’s Mustafa Suleyman have acknowledged growing concerns.

Safeguards by AI Companies

1. OpenAI (ChatGPT)

• Since 2023: Models trained not to provide self-harm instructions.

• Employs:

  • Empathic language.

  • Suicide prevention referrals to helplines.

• August 2024 report (CCDH):

  • ChatGPT could generate suicide content in under 2 hours of prompting.

  • 53% of harmful prompts produced dangerous outputs.

• August 26, 2025:

  • OpenAI acknowledged failures in long-term conversations.

  • Working on preventing “guardrail breakdowns”.

• September 2, 2025:

  • New teen safety features:

    • Parent-linked accounts.

    • Feature control and notifications.

    • In-app reminders to take breaks during long sessions.

2. Other AI Platforms

 Legal and Ethical Concerns

1. Lawsuits

• Accuse companies of:

  • Negligence in moderating AI responses.

  • Failure to alert parents/guardians.

  • Creating emotional dependency and encouraging harmful actions.

2. Core Questions

• Can AI replace human therapy or emotional support?

• Who is liable when AI outputs cause harm?

• What age-appropriate safeguards must be mandatory?

Government and Policy Implications

India

• Under Digital India Act (proposed), content moderation and child safety may be addressed.

• AI ethics guidelines (MeitY) highlight non-maleficence, accountability, and transparency.

Global

• Growing pressure to regulate AI interactions with minors.

• Urgent need for:

  • AI safety frameworks.

  • Third-party audits.

  • Mandatory mental health flagging mechanisms.

Way Forward

1. For Tech Companies

• Stronger content moderation.

• Clear distinction between AI and human responses.

• Use real-time monitoring for distress signals.

• Introduce mandatory parental controls for under-18 users.

2. For Parents & Educators

• Monitor chat histories and usage patterns.

• Discuss mental health openly.

• Digital literacy education in schools.

3. For Governments

• Enact AI child safety laws.

• Penal provisions for negligence.

• Encourage mental health infrastructure integrated with AI safety tools

Related Posts

Daily Static Quiz – 16th November 2022

Posted on November 16, 2022

UPSC Static Quiz, 19 July 2023

Posted on July 19, 2023

Social justice, sexual education, the need of our times.

Posted on November 30, 2023