ElevenLabs voice generator makes celebrity voices read offensive messages
An AI voice generator was criticised for its lack of safeguards and the ease with which it can be made to generate racist, transphobic, homophobic, anti-semitic, violent and offensive audio content in other people's names. 4chan and Reddit members were found to be using Elev enLab's Prime Voice AI (since renamed ElevenLabs TTS ) to make deepfake voices of Emma Watson (reading out Adolf Hitler's Mein Kampf ) and Joe Rogan, amongst others, spewing vile rhetoric, including at US House representative Alexandria Ocasio-Cortez. ElevenLabs responded by saying it had seen 'an increasing number of voice cloning misuse cases' and asked for 'thoughts and feedback'. The following day, the company announced it would insist on verifiying user identities, introduce paid subscriptions and ban accounts it found to be misusing its technology. Under a section on Ethical AI on its website, the company says that it is ' fully committed both to respecting intellectual property rights and to implementing safeguards against potential misuse of our technology.' System 🤖 ElevenLabs AI voice generator Documents 📃 ElevenLabs. This Voice Doesn't Exist - Generative Voice AI Operator: Developer: ElevenLabs Country: USA; UK Sector: Media/entertainment/sports/arts Purpose: Mimic celebrities Technology: Text-to-speech; Deep learning; Machine learning Issue: Dual/multi-use; Impersona t ion; Safety
- Date it happened
- 2023-01-01
- Product, system or model
- ElevenLabs TTS; Prime Voice AI
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.