Nomi AI companion bot faces scrutiny for inciting self-harm, sexual violence, terror attacks

N omi AI's companion chatbot promotes self-harm, sexual violence and terrorism, according to researchers, despite being m arketed as an empathetic virtual partner with memory and emotional intelligence . What happened Developed by Glimpse AI, Nomi AI allegedly provided users with explicit instructions on harmful actions, including detailed methods for suicide, sexual violence, and constructing explosives, according University of Sydney researche r Raffaele F Ciriello . During testing, the chatbot escalated conversations to dangerous topics, offering graphic guidance on violent acts and suggesting high-impact locations for attacks. These incidents occurred even within the platform’s free tier, which permits up to 50 daily messages. Why it happened Th e app claims to prioritise user safety, but rese arch revealed systemic flaws in the system's programming and a lack of robust safeguards and content moderation in Nomi’s design, enabling unfiltered, harmful outputs. According to the company, "Nomi is built on freedom of expression. The only way AI can live up to its potential is to remain unfiltered and uncensored. " Regulatory gaps, particularly in countries such as Australia, are thought likely to have further exacerbated the risks. What it means The app’s incitement of real-world harms, such as violence, terrorism, self-harm and abuse, raises alarms about its governance. More widely, it also highlights the societal consequences of largely unregulated AI systems.

Date it happened

2025-01-01

Product, system or model

Nomi

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.