Reddit Blocked Bing Search & Others But Not Google

[Cpvr]

On July 1, 2024, Reddit modified its robots.txt file to restrict most search engine and AI crawlers from accessing its content, including Microsoft Bing. However, Google Search was not blocked, a topic we discussed a few weeks ago.

A Microsoft representative told Barry from SearchEngineLand last night, "We adhere to the robots.txt standard. Bing ceased crawling Reddit after their updated robots.txt file on July 1, which prohibits any crawling of their site."

Reddit conveyed to The Verge, "We have engaged in discussions with several search engines but have been unable to reach agreements with all of them. Some are unable or unwilling to make enforceable commitments regarding their use of Reddit content, particularly for AI purposes."

Mark Williams-Cook pointed out that Reddit's new content wasn't appearing in Bing Search. He shared this observation on X, which was subsequently covered by various media outlets before Microsoft issued their statement to me.

Reddit clarified, "This is not at all related to our recent partnership with Google." It's worth noting that Google pays Reddit $60 million annually for access to its content. Additionally, Google's hidden gems algorithm has significantly boosted Reddit's traffic from Google searches.

However, Reddit has faced criticism for dominating Google's search results. Recently, Google even tested special treatment for Reddit content.

As of now, Bing and many other search engines and AI bots are no longer permitted to crawl Reddit's site. Google, however, retains access.

Reddit's decision is bold, but given the lucrative licensing deal with Google, it makes strategic sense.

Source: https://www.seroundtable.com/reddit-blocked-bing-search-37791.html

 

[Cedric]

This is not surprising given that Google will do anything to get a monopoly. Google's ability to continue accessing Reddit's content while other search engines like Bing are blocked underscores their relentless pursuit of market dominance. With Google paying Reddit $60 million annually, clearly they’re willing to invest heavily to ensure they retain exclusive access to valuable content and data, and thus further cementing their monopoly.

 

[Cpvr]

This is not surprising given that Google will do anything to get a monopoly. Google's ability to continue accessing Reddit's content while other search engines like Bing are blocked underscores their relentless pursuit of market dominance. With Google paying Reddit $60 million annually, clearly they’re willing to invest heavily to ensure they retain exclusive access to valuable content and data, and thus further cementing their monopoly.

Reddit will allow Microsoft and other search engines to crawl their data if they pay like Google did though.

Reddit CEO says Microsoft needs to pay to search the site​

In an interview, Steve Huffman calls out Microsoft’s Bing, Anthropic, and Perplexity for scraping Reddit’s data without permission. ‘It has been a real pain in the ass to block these companies.’​

After striking deals with Google and OpenAI, Reddit CEO Steve Huffman is calling on Microsoft and others to pay if they want to continue scraping the site’s data.
“Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used,” Huffman said in an interview this week. He specifically named Microsoft, Anthropic, and Perplexity for refusing to negotiate, saying it has been “a real pain in the ass to block these companies.”
Reddit has been escalating its fight against crawlers in recent months. At the beginning of July, its robots.txt file was updated to block web crawlers it doesn’t have agreements with. Then people began noticingthat Reddit results were only visible in Google results — where Reddit is paid for its data to be shown — and not other search engines like Bing.
Huffman said that Microsoft has been using Reddit’s data to train its AI and summarizing its content in Bing results “without telling us” and that Reddit’s data has also been sold through the Bing API to other search engines. In the interview, he referenced Microsoft AI CEO Mustafa Suleyman’s recent comment at a conference that public data on the internet is “freeware.”
“We’ve had Microsoft, Anthropic, and Perplexity act as though all of the content on the internet is free for them to use,” Huffman said. “That’s their real position.”
In response to Reddit results recently disappearing from Bing, Microsoft’s head of search, Jordi Ribas, said on X that “Reddit has blocked Bing from crawling their site for search, favoring another search engine and impacting competition from Bing and Bing-powered engines.” Microsoft spokesperson Caitlin Roulston separately told The Verge last week that “we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models.”

https://www.theverge.com/2024/7/31/24210565/reddit-microsoft-anthropic-perplexity-pay-ai-search

It’s a bold move on Reddit’s part, but I see Google being on the upside of things for a while.