CivitAI accused of generating synthetic 'child pornography' images
Open source image generator Civitai could be used to make images that ‘could be categorized as child pornography,’ or child sexual abuse material (CSAM), according to a media investigation. I nternal communications at text-to-image platform CivitAI cloud computing supplier OctoML shown to 404 Media revealed that it was aware that some CivitAI users had been creating sexually explicit material, including nonconsensual images of real people and pornographic depictions of children. CivitAI had been using OctoML's OctoAI for image generation. OctoML responded to 404 Media 's report by rolling out a filter to block the generation of NSFW content on CivitAI, before cutting ties with the company. 'We have decided to terminate our business relationship with CivitAI. This decision aligns with our commitment to ensuring the safe and responsible use of AI,' the company told 404 Media . CivitAI founder Justin Maier later told Venture Beat that he had been aware that some people were making NSFW content, but that he tolerated it as they were helping to train the company's models. System 🤖 CivitAI 🔗 Operator: CivitAI Developer: OctoML Country: USA Sector: Media/entertainment/sports/arts Purpose: Generate images Technology: Text-to-image; Generative adversarial network (GAN); Neural network; Deep learning; Machine learning Issue: Privacy; Safety; Tr ansparency
- Date it happened
- 2023-12-01
- Organisation involved
- CivitAI
- Product, system or model
- OctoAI
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.