Cloudflare and AI Crawlers: Who Gets to Read Your Site?

If your website sits behind Cloudflare, the rules about which automated visitors get in have been shifting underneath you. Cloudflare serves a large share of the web, so its defaults quietly shape what AI companies can read. This note walks through the recent announcements and offers a way for a small business to decide what it actually wants, without hype in either direction.

The background

In a July 1, 2026 post marking one year since it changed its defaults, Cloudflare recalled that it had begun blocking AI training crawlers by default for new domains. The same report says more than half of internet traffic is now non-human, and that as of June 2026 about 52 percent of crawler requests were for AI training, up from 22 percent in spring 2025. It also says mixed-use crawlers, which gather data for both search and AI, make up over a third of crawler activity. These are Cloudflare’s own measurements, and the company has an interest in the topic, so treat them as one informed view rather than neutral fact.

The company’s argument is about fairness. Crawling used to be paid for in referral visits. Cloudflare says people now spend far less of their search time on the open web, and that some heavily crawled categories saw human traffic fall by as much as 40 percent in under a year. It also notes that Google accounts for roughly 88 percent of referral traffic and that Google’s mixed-use crawler prevents owners from joining search without also joining its AI experiences.

What was reported for September 2026

A news summary reported that from September 15, 2026, Cloudflare would block mixed-use crawlers from ad-supported pages by default, applying to new customers and new sites on existing accounts. It also mentioned managed robots.txt controls and a pilot of “pay per crawl,” where publishers could charge AI companies for access. If your Cloudflare account is older, existing sites were not described as changing automatically, but it is worth opening your dashboard and reading the current settings rather than assuming.

The counterpoint: discovery pilots

Blocking is only half the story. On October 15, 2026, Cloudflare and OpenAI shared early results of a pilot that began in July. Using a cohort of about one percent of Cloudflare’s network, Cloudflare’s signals helped OpenAI’s search crawler find new pages, and OpenAI reported discovering and indexing hundreds of millions of them. The signals describe pages that lead nowhere, pages that have changed, and individual pages flagged as harmful. The post says a signal does not grant permission to access a page, and that site owners can see and manage access with Cloudflare’s AI Crawl Control. The research is described as using metadata, not page content for training.

For a small site, one reading is that better signals could mean fewer wasted visits from crawlers to your server, and easier discovery if you want it.

A simple way to decide

Start with your business model

You live on local or niche discovery (a studio, a clinic, a specialist shop). Being found by assistants may help. Blocking everything could make you invisible to a growing way of looking things up.
You sell your writing, data or courses. Your content is the product. Restricting training and mixed-use crawlers is sensible, and licensing talks are not out of the question for larger publishers.
You run ads on content pages. Crawlers that read without sending visitors cost you directly, so the defaults Cloudflare is moving towards may suit you.

Separate the visitors

The report suggests allowing or blocking discovery and training separately, and favouring AI companies that identify their bots and declare what they crawl for. In practice that means treating a search crawler differently from a training crawler instead of using one blanket rule.

What to do this week

Log in to your Cloudflare dashboard and note your current AI-crawler and bot settings.
Check your robots.txt, in case it says something that contradicts the dashboard.
Look at whether you have any AI crawl reports available, and record which URLs are requested most.
Decide which pages are public marketing material you are happy to be read, and which are protected assets.
Write the decision down with the date, so you can review it in three months.

When to skip it

If your site is small, has no ads, and your content is the sort you want everywhere, you may not need to change anything. The only risk is accidentally blocking something you want, such as a search crawler, through an over-eager rule. If your site is not behind Cloudflare, the same questions apply in your host’s or firewall’s settings.

If you would like a second pair of eyes on your bot and caching setup, our services include technical reviews for small business sites.