Telegram AI bots create non-consensual deepfake porn

AI-powered bots on encrypted messaging app Telegram is being used to create photo-realistic nude and sexually explicit images of hundreds of thousands of women without their consent , raising serious concerns about the misuse of deepfake technology and Telegram's laissez-faire content moderation. The bots, which appear to be popular in Russia and eastern Europe, only require one photograph to create the images, according to security company Sensity, and ha ve led to over 680,000 ordinary women and girls being demeaned and degraded. Some of these appear to be underage girls. The images use deep-learning software to "strip" clothing from images of women, creating hyper-realistic nude forgeries in under 60 seconds , and are used and manipulated without the knowledge or consent of the women being targeted . By July 2020, an estimated 104,852 women had been targeted with publicly shared "stripped" images, each of whom may be harassed, abused, ext orted, or otherwise harmed. Telegram’s hands-off moderation policy and commitment to total encryption provided a safe harbour for these bots. Despite being alerted to the specific channels by Sensity, Telegram failed to act, citing a commitment to privacy that effectively shielded perpetrators rather than victims. The photographs clearly violate Telegram's terms of service . System 🤖 Unknown Operator: Telegram Developer: Country: Russia; Global Sector: Media/entertainment/sports/arts Purpose: Simulate nude images Technology: Deepfake; Machine learning Issue: Accountability; Authenticity/integrity; Appropriation; Privacy/surveillance; Safety; Transparency

Date it happened

2020-10-01

Organisation involved

Telegram

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.