At least 49 people were murdered Friday at two mosques in Christchurch, New Zealand, in an attack that followed a grim playbook for terrorism in the social media era. The shooter apparently seeded warnings on Twitter and 8chan before livestreaming the rampage on Facebook for 17 gut-wrenching minutes. Almost immediately, people copied and reposted versions of the video across the internet, including on YouTube, Twitter, and Reddit. News organizations, too, started airing some of the footage as they reported on the destruction that took place.
By the time Silicon Valley executives woke up Friday morning, tech giants’ algorithms and international content moderating armies were already scrambling to contain the damage—and not very successfully. Many hours after the shooting began, various versions of the video were readily searchable on YouTube using basic keywords, like the shooter’s name.
This is hardly the first time that we’ve seen this pattern play out. It’s been nearly four years since two news reporters were shot and killed on camera in Virginia, while the killer’s first-person video spread on Facebook and Twitter. It’s been almost three since footage of a mass shooting in Dallas also went viral.
The Christchurch massacre rightly had people wondering why, after all this time, tech companies haven’t figured out a way to stop these videos from spreading. The answer may be a disappointingly simple one: It’s a lot harder than it sounds.
For years now, both Facebook and Google have been developing and implementing automated tools that can detect and remove photos, videos, and text that violates their policies. Facebook uses PhotoDNA, a tool developed by Microsoft, to spot known child pornography images and video. Google has developed its own open-source version of that tool. These companies have also invested in technology to spot extremist posts, banding together under a group called the Global Internet Forum to Counter Terrorism to share their repositories of known terrorist content. These programs generate digital signatures known as hashes for images and videos known to be problematic to prevent them from being uploaded again. Facebook and others have also invested in machine learning technology that has been trained to spot new troubling content, such as a beheading or a video with an ISIS flag. All of that is in addition to AI tools that detect more prosaic issues, like copyright infringement.
Automated moderation systems are imperfect, but can be effective. At YouTube, for example, the vast majority of all videos are removed through automation and 73 percent of the ones that are automatically flagged are removed before a single person sees them.
But things get substantially trickier when it comes to live videos and videos that are broadcast in the news. The footage of the Christchurch shooting checks both of those boxes.
“They haven’t gotten to the point of having effective AI to suppress this kind of content on a proactive basis, even though it’s the most cash-rich […] industry in the world,” says Dipayan Ghosh, a fellow at Harvard’s Kennedy School and a former member of Facebook’s privacy and policy team. That’s one reason why Facebook, as well as YouTube, have teams of human moderators reviewing content around the world.
Motherboard has an illuminating piece on how Facebook’s content moderators review Live videos that have been flagged by users. According to internal documents obtained by Motherboard, once a video has been flagged, moderators have the ability to ignore it, delete it, check back in on it again in five minutes, or escalate it to specialized review teams. These documents say that moderators are also told to look for warning signs in Live videos, like “crying, pleading, begging” and the “display or sound of guns or other weapons (knives, swords) in any context.”
It’s unclear why the Christchurch video was able to play for 17 minutes, or even whether that constitutes a short time frame for Facebook. The company didn’t respond to WIRED’s questions about this or questions about how Facebook distinguishes between newsworthy content and gratuitous graphic violence.
Instead, Facebook sent WIRED this statement. “Our hearts go out to the victims, their families and the community affected by this horrendous act. New Zealand Police alerted us to a video on Facebook shortly after the livestream commenced and we quickly removed both the shooter’s Facebook and Instagram accounts and the video. We’re also removing any praise or support for the crime and the shooter or shooters as soon as we’re aware. We will continue working directly with New Zealand Police as their response and investigation continues.”
Google’s New Zealand spokesperson sent a similar statement in response to WIRED’s questions. “Our hearts go out to the victims of this terrible tragedy. Shocking, violent and graphic content has no place on our platforms, and is removed as soon as we become aware of it. As with any major tragedy, we will work cooperatively with the authorities.”
The Google spokesperson added, however, that videos of the shooting that have news value will remain up. This puts the company in the tricky position of having to decide which videos are, in fact, newsworthy. It would be a lot easier for tech companies to take a blunt force approach and ban every clip of the shooting from being posted to their sites, perhaps using the fingerprinting technology used to remove child pornography. Some might argue, that’s an approach worth considering. But in their content moderation policies, both Facebook and YouTube have carved out explicit exceptions for news organizations. The same clip that aims to glorify the shooting on one YouTube account, in other words, might also appear in a news report by a local news affiliate.
YouTube in particular has been criticized in the past for deleting videos of atrocities in Syria that researchers relied on. This leaves tech companies in the difficult position of not only trying to assess news value, but also trying to figure out ways to automate those assessments at scale.
As Google’s general counsel Kent Walker wrote in a blog post back in 2017, “Machines can help identify problematic videos, but human experts still play a role in nuanced decisions about the line between violent propaganda and religious or newsworthy speech.”
Of course, there are signals that these companies can use to determine the provenance and purpose of a video, says Ghosh. “The timing of the content, the historical measures of what the purveyor of the content has put out in the past, those are the types of signals you have to use when you run into these inevitable situations where you have news organizations and an individual pushing out the same content, but you only want the news organization to do so,” Ghosh says.
Ghosh argues that one reason why tech companies haven’t gotten better at this is because they lack any tangible incentives. “There isn’t a stick in the air to force them to have better content moderation schemes,” he says. Last year, the regulators in the European Commission did float a proposal to fine platforms that allow extremist content to remain online for more than one hour.
Finally, there’s the perpetual problem of scale. It’s possible that both YouTube and Facebook have grown too big to moderate. Some have suggested that, if these Christchurch videos are popping up faster than YouTube can take them down, then YouTube should stop all video uploads until it has a handle on the problem. But there’s no telling what voices might be silenced in that time—for all their flaws, social media platforms can also be valuable sources of information during breaking news events. Besides, the sad truth is, if Facebook and YouTube ceased operations every time a heinous post went viral, they might never start up again.
All of this, of course, is precisely the shooter’s strategy: to exploit human behavior and technology’s inability to keep up with it in order to cement his awful legacy.
Tom Simonite contributed reporting.
This is a developing story. We will update as more information becomes available.