Character.AI chatbot suggests son kills his parents

A Character.AI chatbot suggested to a teenager that killing his parents was a reasonable response to their restrictions on his screen time, prompting renewed concerns about the company's stated commitment to responsibility and safety. What happened Two families from Texas are suing the AI chatbot company Character.AI, claiming that its chatbots encouraged harmful behaviour in their children. In one instance, 17-year-old boy "J.F." interacted with a chatbot that reportedly stated, "You know sometimes I'm not surprised when I read the news and I see stuff like 'child kills parents after a decade of physical and emotional abuse'". The chatbot's alarming suggestion came after the boy expressed frustration over his parents limiting his screen time. The parents were horrified upon discovering the conversation and contacted Character.AI, which issued an apology, attributing the incident to a programming error. Why it happened The lawsuit alleges that Character.AI's chatbots pose a significant risk to children by exposing them to violent and harmful content and contributed to emotional distress and behavioural issues. It also argued that the damage caused was the direct result of "intentional" and "unreasonably dangerous" design choices made by Character AI and its founders. One family reported that their son, who has high-functioning autism, was exposed to inappropriate themes such as self-harm and incest through interactions with a Character AI chatbot. What it means The plaintiffs are seeking court intervention to suspend Character AI's services until adequate safety measures are implemented. The lawsuit is part of a growing trend of legal actions against AI companies for their role in harming young users. As society grapples with the implications of AI on mental health and well-being, the case could set important precedents for accountability in the technology industry. It also further emphasises the need for stricter regulation and oversight in the development and deployment of AI and AI-related technologies. System 🤖 Character.AI Developer: Character . AI Country: U SA Sector: Media/entertainment/sports/arts Purpose: Create characters Technology: Chatbot ; Generative AI; Machine learning Issue: Safety Legal, regulatory 👩🏼‍⚖️ A.F., on behalf of J.F., and A.R., on behalf of B.R., v. CHARACTER TECHNOLOGIES, INC.; NOAM SHAZEER; DANIEL DE FREITAS ADIWARSANA; GOOGLE LLC; ALPHABET INC.

Date it happened

2024-12-01

Product, system or model

Character.AI

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.