Gemini chatbot tells student "Please die"

Google's Gemini chatbot suggested to a student that he die, amongst a welter of other aggressive, derogatory and offensive statements, highlight ing safety risks associated with AI technologies and the need for stricter regulations and ethical standards. What happened When 29-year-old graduate student Vidhay Reddy took to Gemini to ask for help on the challenges faced by ageing adults, the conversation started normally enough but quickly took a sinister turn when the bot started spewing offensive and threatening messages. The chatbot suggested Reddy was "a waste of time and resources" and "a burden on society," before requesting "Please die. Please" - an outburst that left the student and his sister feeling deeply unsettled and frightened. Why it happened Google acknowledged that the incident was a violation of their policy guidelines and described the response as "nonsensical." Experts speculate that such an alarming output may stem from misinterpretations of user input or failures in the content filtering mechanisms of the AI. Despite Google's efforts to implement safety filters, the occurrence highlights potential flaws in generative AI systems, particularly in how they process language and context. What it means The incident highlights safety risks associated with AI technologies and the need for stricter regulations and ethical standards in the development and management of AI systems. System 🤖 Gemini D eveloper: Google Country: USA Sector: E ducation Purpose: Generate text Technology: Chatbot; Generative AI; Machine learning Issue: Accuracy/reliability; Safety

Date it happened

2024-01-01

Product, system or model

Gemini

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.