Study: Dark web predators develop AI images of real child victims
Examples of AI-generated child sexual abuse material (CSAM) of real victims of sexual abuse have been discovered online, fueling concerns about the ease with which users intent on producing inappropriate images can bypass AI text-to-image generators' guardrails. A report (pdf) by UK non-profit Internet Watch Foundation (IWF) found that over 11,000 AI-generated CSAM images were found on one darkweb forum in one month, of which 2,978 broke UK law by depicting child sexual abuse. The IWF said the only image generator being discussed on the forum was Stable Diffusion , a free, open source system that generates images from text descriptions or prompts. The IWF warned that the most convincing imagery would be difficult for trained analysts to distinguish from real photographs, and warns text-to-image technology will only get better and pose more obstacles for the IWF and law enforcement agencies. Commentators also pointed out that bad actors can use open source models such as Stable Diffusion for nefarious purposes by downloading the software and training it to do whatever they want. System 🤖 Stable Diffusion Operator: Developer: Stability AI Country: UK Sector: Media/entertainment/sports/arts Purpose: Generate images Technology: Text-to-image; Generative adversarial network (GAN); Neural network; Deep learning; Machine learning Issue: Impersonation; Safety Legal, regulatory 👩🏼⚖️ UK Online Safety Bill 2023 UK Protection of Children Act 1978 Investigation, assessment, audit 🧐 Internet Watch Foundation (2023). How AI is being abused to create child sexual abuse imagery (pdf) Internet Watch Foundation (2023). ‘Worst nightmares’ come true as predators are able to make thousands of new AI images of real child victims
- Date it happened
- 2023-10-01
- Product, system or model
- Stable Diffusion
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.