Google flags medical images of groins as CSAM
Google's automated system to detect abusive images of children has resulted in two fathers being unfairly investigated by the police and having their accounts deactivated across all Google platforms. The two incidents highlight the inaccuracy and unreliability of automatic photo screening and reporting technology, and the broader impacts these errors can have on individuals when things go wrong. According to the New York Times , a man named Mark had sent pictures of his son’s groin to a doctor in San Francisco after realising it was inflamed, only for Google to identify the images as abusive, suspend his account, and report the photos to the US National Center for Missing and Exploited Children’ CyberTipline, which escalated the report to the police. Per the NYT , ' Not only did he lose emails, contact information for friends and former colleagues, and documentation of his son’s first years of life, his Google Fi account shut down, meaning he had to get a new phone number with another carrier. Without access to his old phone number and email address, he couldn’t get the security codes he needed to sign in to other internet accounts, locking him out of much of his digital life.' Similarly, photos of a boy's 'intimal parts' backed up on Google Photos that had been requested by a Houston-based pediatrician to diagnose an infection resulted in the boy's father also having his account suspended. Google, which uses Microsoft’s PhotoDNA tool as part of its efforts to detect child abuse, said it identified 287,368 instances of suspected abuse in the first six months of 2021. System 🤖 Google Child Safety Toolkit 🔗 Documents 📃 Google (2022). How we detect, remove and report child sexual abuse material Microsoft. PhotoDNA Operator: Alphabet/Google Developer: Alphabet/Google Country: USA Sector: Health Purpose: Detect child sexual abuse material Technology: Hash matching; Machine learning Issue: Accountability; Accuracy/reliability; Transparency
- Date it happened
- 2021-01-01
- Organisation involved
- Product, system or model
- PhotoDNA
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.