Study: Whisper AI speech recognition creates violent hallucinations

OpenAI's Whisper generates violent and fabricated language, particularly during long pauses in speech, according to researchers, raising concerns about its use in sensitive areas like hiring and medicine. What happened In a study analy s ing 13,000 audio clips, resear chers from Cornell University and the University of Virginia found that 1.4 pe rcent of transcriptions produced by Whisper contained entire sentences that were not present in the original audio, some of which were violent and potentially harmful. On one instance, Whisper accurately transcribed a single, straightforward sentence but subsequently generated five additional sentences containing words such as "terror," "knife," and "killed," none of which were present in the original audio. These so-called hallucinations encompass invented phrases, personal information, and even fictitious websites, some of which could be exploited for malicious purposes. Why it happened The hallucinations appear to stem from Whisper's underlying AI model, which generates outputs based on patterns learned from its training data. These patterns can lead to nonsensical or fabricated phrases, particularly during moments of silence or background noise in recordings. The researchers noted that this issue is exacerbated when transcribing individuals with speech impairments, such as those with aphasia, who may produce more disfluent speech. What it means The implications of these findings are significant, especially given Whisper's widespread use in sensitive contexts like medical transcription. The potential for misinterpretation could lead to serious consequences, such as misdiagnoses or inappropriate treatment recommendations. Experts have called for stricter auditing and testing of AI models like Whisper to ensure their reliability and accuracy before further integration into critical applications. System 🤖 Whisper D eveloper: OpenAI Country: USA Sector: Health Purpose: Recognise speech; Transcribe speech Technology: Chatbot; Generative AI; Machine learning; Speech-to-text; Speech recognition Issue: Accuracy/reliab ility; Mis/disinformation; Safety; Security

Date it happened

2024-01-01

Product, system or model

Whisper

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.