Why Reddit is Blocking Search Engines (Except Google)
In recent years, Reddit has seen a significant increase in traffic from search engines. This can be attributed to internet users who have adopted the practice of adding “Reddit” to their search queries, hoping to find authentic discussions and opinions from real people. Whether it’s product reviews or travel recommendations, many individuals turn to Reddit’s vast archives of discussions to find what they need.
However, a recent discovery by tech outlet 404 Media reveals that Reddit has started blocking the majority of search engines from crawling and indexing its website. This means that Reddit posts are no longer showing up in search results on platforms like Microsoft’s Bing and DuckDuckGo. The exception to this is Google, which still allows users to search within Reddit.
So, what search engines are affected and why? According to reports from 404 Media, popular search engines like Bing and DuckDuckGo are not delivering recent results from Reddit.com because they weren’t crawled and indexed before Reddit implemented the change. Even a search attempt by Mashable on Bing using “site:reddit.com” over the past week yielded zero results.
Reddit announced a change to its Robots Exclusion Protocol (robots.txt) on June 25, citing an increase in bot activity on its website. However, this change was not expected to impact legitimate organizations and platforms that assist the internet community. Reddit’s protocol states, “Reddit believes in an open Internet, but not the misuse of public content.”
Interestingly, Google is the only search engine that continues to provide recent search results from Reddit. This can be attributed to the $60 million deal between Reddit and Google earlier this year. As part of the agreement, Google has access to Reddit’s content to train its AI models.
While Reddit denies that its recent changes are related to its partnership with Google, it is clear that the platform is blocking crawlers that refuse to commit to not using crawl data for AI training. This aligns with Reddit’s commitment to enforcing its Public Content Policy and updated robots.txt file.
The speculation surrounding Reddit’s motives for blocking search engines, particularly Microsoft, is unprecedented. It highlights the potential dangers of a future where search engines become pay-to-play platforms, even for organic search results. Mashable has reached out to Reddit for comment and will update the story as more information becomes available.
In conclusion, Reddit’s decision to block search engines, except for Google, raises questions about the future of organic search results and the influence of AI in the search engine landscape. As internet users increasingly rely on Reddit for authentic conversations and opinions, the platform’s stance on protecting its content and ensuring responsible use of crawl data is crucial.
