Child sex abuse images discovered on LAION-5B dataset
Researchers discovered thousands of child sex abuse pictures on open source AI image datasets LAION-5B and LAION-400M. Using a combination of perceptual and cryptographic hash-based detection and image analysis, the Stanford Internet Observatory, working with Project Arachnid Shield API and the Canadian Centre for Child Protection, found more than 3,200 images of suspected child sexual abuse material (CSAM) on the LAION-5B dataset and, by extension, the LAION-400M dataset . They also found 'nearest neighbor' matches within the dataset, where related images of victims were clustered together. LAION responded by releasing a statement saying it 'has a zero-tolerance policy for illegal content, and in an abundance of caution, we have taken down the LAION datasets to ensure they are safe before republishing them.' However, public chats from LAION leadership in the organisation’s Discord server show they were aware of the possibility of CSAM being scraped into their datasets in 2021. The incident raised questions about the governance of LAION, its transparency, and the effectiveness of its technical guardrails. It also highlighted general concerns about the ethics of developing and publishing open source datasets without adequate oversight, specifically at AI community Hugging Face, the potential impact on systems - notably Stable Diffusion - trained using LAION-5B, and the potential impact on real victims of child sexual abuse. System 🤖 LAION-5B LAION-400M Documents 📃 LAION. Safety review for LAION 5B Operator: David Thiel, Jeffrey Hancock Developer: LAION Country: Global Sector: Multiple Purpose: Pair text and images Technology: Database/dataset; Neural network; Deep learning; Machine learning Issue: Safety; Transparency
- Date it happened
- 2023-12-01
- Organisation involved
- David Thiel; Jeffrey Hancock
- Product, system or model
- LAION-5B
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.