Addressing the Challenge of Extremist Content: Jigsaw’s Open Source Tool
In an era where the swift removal of illicit content has become a pressing concern for online platforms, Google’s incubator subsidiary, Jigsaw, has taken a significant step toward combating extremist content. Jigsaw recently open sourced a counterterrorism tool called Altitude, designed to assist smaller companies in addressing the dissemination of extremist material on their platforms.
While major tech companies like Meta, X, and Google themselves have the resources to moderate and eliminate illegal or toxic content effectively, smaller digital players often struggle with this task. The Digital Services Act (DSA) in Europe and other legislation targeting terrorism-specific content have heightened the pressure on companies to promptly remove such content.
Jigsaw, formerly known as Google Ideas, is dedicated to developing products that tackle societal issues like disinformation, censorship, and the spread of extremism online. One of their latest creations is Altitude, an online content moderation tool that helps moderators verify flagged content against third-party databases. This tool streamlines the process by eliminating the need to check each source individually.
Altitude was developed in collaboration with Tech Against Terrorism, an initiative launched by the United Nations’ Counter-Terrorism Committee Executive Directorate (CTED) in 2017. Tech Against Terrorism aims to support the tech industry in developing more effective tools to combat the use of the internet by terrorists for spreading harmful content. Additionally, Altitude was created in partnership with Global Internet Forum to Counter Terrorism (GIFCT), an industry-led organization that boasts major tech companies among its members.
After nine months of development, Jigsaw has officially handed over Altitude to Tech Against Terrorism for further development and maintenance. The code is now available on GitHub under the Apache 2.0 license, allowing companies to self-host the tool and enabling third parties to inspect the code.
So how does Altitude work? The tool relies on trusted data sources from GIFCT and Tech Against Terrorism’s own Terrorist Content Analytics Platform. These sources identify and flag terrorist and violent extremist content (TVEC) using their unique methods. Altitude consolidates this flagged content into a single view, enabling moderators to swiftly view, review, and remove it to comply with their jurisdiction’s obligations. Importantly, no content is automatically removed; the tool serves to flag and provide alerts for human assessment.
Although the specific platforms that have utilized Altitude during its testing period have not been disclosed, it is expected that messaging apps, URL-shortening services, video-hosting sites, online forums, social networks, and pastebins are among the applications most likely to benefit from the tool.
In a blog post, Jigsaw revealed that they conducted interviews with various platform founders, trust and safety leaders, and TVEC content reviewers to assess the tool’s effectiveness. The study found that organizations of different sizes exhibited varying degrees of resourcing and preparedness in managing TVEC. Interestingly, the level of preparedness did not necessarily correlate with the organization’s size or user base. Even mature services sometimes lacked proactive identification of harms on their platforms. However, the study also highlighted that a separate interface dedicated to addressing harms could be a starting point for under-resourced platforms, even if they eventually aim to integrate all tools into an in-house platform.
As an open-source project, Altitude can be integrated into any company’s platforms. Alternatively, companies can collaborate with Tech Against Terrorism to receive customized onboarding support.
Looking ahead, Jigsaw and Tech Against Terrorism plan to expand Altitude beyond English and consider adding specialized databases to their data sources. This continuous development of the tool underscores their commitment to equipping smaller digital players with effective solutions for combating extremist content on their platforms.
