Reddit Blocks Major Search Engines and AI Bots: New Policy
Reddit is intensifying its crackdown on web crawlers, blocking search engines from accessing recent posts and comments unless they pay. According to 404 Media, only Google shows recent Reddit results using the “site:reddit.com” trick, likely due to a $60 million deal for AI training on Reddit content.
Reddit spokesperson Tim Rathschmidt states that the policy is unrelated to the Google partnership. It highlights ongoing discussions with other search engines that haven’t reached agreements. Due to enforceability issues regarding Reddit content use for AI.
People Also Read: Artificial Intelligence and the World We Live in
To enforce its anti-scraping policy, Reddit updated its robots.txt file. It signals to web crawlers without agreements to stop accessing its data. Ben Lee, Reddit’s chief legal officer, explained this move as a measure against unauthorized data access. Microsoft’s Caitlin Roulston confirmed that Bing ceased crawling Reddit on July 1st in compliance with this update.
Recommended for You: Perplexity AI: The Answer Engine with a Lot of Questions Marks
This strategy, though bold, aligns with Reddit’s recent efforts to protect its data, generate revenue, and satisfy investors. After increasing API costs for some developers, Reddit even threatened to cut off Google for using its data for AI training without payment.
As AI-generated content grows, finding genuine human-written content is crucial. Many users add “Reddit” to searches for authentic answers. In short, it’s frustrating that now only Google or its dependent search engines will support this.
Source: Reddit is now blocking major search engines and AI bots — except the ones that pay
