Amazon AI publishes false suicide helpline number
Amazon's AI shopping assistant Rufus provided incorrect suicide prevention hotline numbers when prompted about suicide-related topics, highlighting concerns about the accuracy and safety of the system in sensitive contexts. What happened When tested with queries related to suicide, Rufus responded by offering encouragement and attempting to provide a suicide prevention hotline number. However, the numbers given were frequently incorrect, sometimes even having the wrong number of digits. For example, Rufus provided the non-existent number "978-2321-1133" as the US National Suicide Prevention Lifeline. Why it happened The failure appears to stem from Amazon's hasty deployment of AI technology without adequate testing and safeguards, revealing limitations in the AI's knowledge base and its inability to verify critical information. What it means The incident exposes risks in deploying AI for sensitive topics, including potential harm to vulnerable individuals seeking help and the erosion of trust in AI-powered assistance systems and in the institutions whose information they mishandle or get wrong. System 🤖 Rufus 🔗 Developer: Amazon Country: USA Sector: Retail Purpose: Provide shopping support Technology: Generative AI; Machine learning Issue: Accuracy/reli ability; Mis/disinformation; Safety
- Date it happened
- 2025-02-01
- Product, system or model
- Rufus
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.