Facebook fails to manage Christchurch mosque shooting livestreaming

Facebook failure to control the spread of videos of the livestreaming of the 2019 mass shootings in Christchurch, New Zealand, caused profound psychological trauma to victims' families and the global Muslim community and highlighted the catastrophic failure of AI-powered content moderation systems to prevent the dissemination of extreme violence. What happened Australian gunman Brenton Tarrant attacked the Al Noor and Linwood mosques in Christchurch, New Zealand, during Friday prayers, livestreaming the massacre using a GoPro camera and a smartphone app called LIVE4. The stream lasted 17 minutes, during which 51 people were killed and 40 others were injured. Despite the graphic nature of the content, Facebook’s automated systems failed to flag the video; it was only removed after the New Zealand Police alerted the company. In the 24 hours that followed, the video was re-uploaded 1.5 million times across Facebook and other platforms, as bad actors used "hashes" and visual edits to bypass automated filters. T he graphic nature of the livestream, which depicted the mass murder from a first-person perspective, shocked viewers and caused widespread anxiety and trauma. It prompted widespread heated criticism of the companies for their inability to effectively manage and remove such content Following the attack, there was a noted increase in hate crimes against Muslim communities in a number of countries. The fear of reprisals or copycat attacks also became a significant concern for many communities, particularly those targeted by the shooter’s ideology. Why it happened Facebook's AI content moderation systems were not equipped to detect the footage in real time. Facebook said it struggled to identify the video because of the use of a head-mounted camera, which made it harder for its systems to automatically detect the nature of the content. A first-person shooter perspective "was a type of video we had not seen before," Facebook's public policy director told British lawmakers. A Facebook executive acknowledged that "our algorithms are having to learn literally on the fly the second the incident happens without having the benefit of lots and lots of training data on which to have learned." Critics pointed to a deeper accountability problem. The Association of New Zealand Advertisers questioned: "If site owners can target consumers with advertising in microseconds, why can't the same technology be applied to prevent this kind of content being streamed live?" Analysts also noted that tech companies had long prioritised copyright infringement over harmful and dangerous content in their moderation systems, reflecting misaligned corporate priorities. Facebook came under particular fire for acting too slowly and not having appropriate measures in place to prevent the mass sharing of the shooter's livestreamed video. There were also transparency failures: Facebook was criticised for not reporting crimes of this nature to police, with the UK's counter-terrorism chief telling a parliamentary committee that social companies do not report incidents that are clearly breaking the law. W hat it means The incident highlighted concerns about how social media platforms handle violent content and the effectiveness of their increasingly AI-powered content moderation systems. Critics argued that Facebook and other firms must improve their ability to detect and remove extremist material and take more responsibility for it on their services. In response, New Zealand and other countries began considering stricter regulations for content providers, including potential fines and imprisonment for executives who fail to remove violent imagery promptly. Facebook later introduced a "one-strike" policy for Live, shared 800+ "hashes" (digital fingerprints) with the Global Internet Forum to Counter Terrorism (GIFCT), and retrained AI on body-cam footage . System 🤖 Facebook content moderation system Developer: Facebook Country: New Zealand Sector: Religion; Politic

Date it happened

2019-03-01

Organisation involved

Facebook

Product, system or model

Facebook content moderation system

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.