Google hate detection AI mistakes bullying for civility
Google's AI-powered anti-bullying tool faced criticism for misclassifying certain types of online interactions, prompting concerns about its accuracy and effectiveness. Developed by Jigsaw, Perspective uses machine learning to assess the "toxicity" of online comments, categori s ing them from "very toxic" to "very healthy." However, commentators and researchers pointed out that the AI's training data skews its understanding, leading it to overlook harmful phrases while deeming innocuous comments as toxic. For instance, phrases that express overtly discriminatory views can be rated as only slightly toxic, while straightforward profanity receives a much higher toxicity score. This discrepancy was seen to highlight a significant flaw in the tool's design, reflecting the biases of its creators and the cultural push for civility that can inadvertently sanitise harmful discourse. Critics argue that this approach to moderation may perpetuate existing biases and fail to address the complexities of online communication, where politeness can mask harmful sentiments. It also led to concerns that the tool could be used to silence marginalised voices and promote a culture of civility that prioritises politeness over justice. System 🤖 Perspective API Operator: Developer: Google/Jigsaw Country: Global Sector: Media/entertainment/sports/arts Purpose: Detect hate speech Technology: Machine learning Issue: Accuracy/reliability; Safety
- Date it happened
- 2017-02-01
- Organisation involved
- Product, system or model
- Perspective
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.