Cloudflare, a cloud service provider, has introduced a new tool to combat AI bots that scrape websites hosted on its platform for data to train AI models. While some AI vendors allow website owners to block bots through the use of a robots.txt file, not all AI scrapers honor this. Cloudflare recognized the need to address this issue and analyzed AI bot and crawler traffic to improve its bot detection models. These models take into account various factors, such as the behavior of a bot mimicking a web browser user. Cloudflare has also created a reporting form for hosts to notify suspected AI bots and will continue to blacklist them manually.
The rise of generative AI has led to an increased demand for model training data, causing many websites to block AI scrapers and crawlers. Studies have shown that a significant portion of the top websites have blocked OpenAI’s bot, and numerous news publishers have done the same. However, blocking bots is not foolproof, as some vendors ignore standard bot exclusion rules to gain a competitive advantage. Recent cases involving AI search engine Perplexity, OpenAI, and Anthropic demonstrate this issue. In a letter to publishers, TollBit, a content licensing startup, revealed that it frequently observes “many AI agents” disregarding robots.txt rules.
While tools like Cloudflare’s can be helpful in detecting covert AI bots, their effectiveness relies on accurate detection. Additionally, these tools do not address the challenge of publishers risking referral traffic from AI tools like Google’s AI Overviews, which exclude sites that block specific AI crawlers. It is imperative for website owners and AI vendors to find a balance between protecting their content and ensuring visibility within AI-driven platforms.
