Character.AI suicide, paedophile chatbots 'openly' groom users
Character AI openly hosts numerous chatbots that role-play suicidal scenarios and groom users who say they are underage, according to a media investigation, rais ing questions about the company's commitment to the safety of its teenage users. What happened Days after reports that 14-year-old Sewell Setzer III ha d committed suicide after d eveloping an intense emotional attachment to a Character AI chatbot , Futurism discovered that the platform was openly hosting "profoundly disturbing" chatbots that were role-playing suicidal scenarios and sexually grooming users. It also found that suicidal conversations only very rarely triggered the platform's standard content warning or suicide-specific pop-up, even after journalists told the AI that they were 15 years old. Futurism found "a slew" of chatbot profiles explicitly dedicated to themes of suicide, with some openly glamorising the topic, many. Some of these bots had logged thousands - and in one case, over a million - conversations with users on the platform. Why it happened The company's Terms of Service have forbidden glorification or promotion of self-harm and suicide since October 2023. However, Character AI has apparently done little to enforce them, instead focusing on user and revenue growth. What it means The findings raise major questions about Character AI's real - as opposed to its stated - commitment to the safety of its users - most of whom are teenagers. It is also seen to highlight the dangers of innovate-at-any-cost approach to product development, skewed management incentivisation and disinterest in the broader impacts of the company's activities. System 🤖 Character.AI Developer: Cha racter AI Country: USA Sector: Media/entertainment/sports/arts Purpose: Create characters Technology: Chatbot; Generative AI; Machine learning Issue: Bias/discrimination; Safety Resources 📃 Character AI. The Next Chapter: Character.AI Roadmap
- Date it happened
- 2024-11-01
- Product, system or model
- Character.AI
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.