Cloudflare's OpenAI Discovery Pilot: What Site Owners Should Change
Cloudflare now feeds network signals to OpenAI's search crawler. How the pilot works, what it skips, and the robots.txt and URL fixes worth making now.
Cloudflare's OpenAI Discovery Pilot: What Site Owners Should Change
Quick answer: Cloudflare is feeding network signals to OpenAI's search crawler so it finds new pages, skips wasted URLs and revisits pages only when they change. The 15 October results claim hundreds of millions of newly indexed pages from about 1% of Cloudflare's network. You cannot opt into it yet. What you can do now: allow OAI-SearchBot deliberately, decide on GPTBot separately, and remove the dead URLs and fake freshness that the signals are designed to filter out.
ChatGPT search is getting a discovery layer that does not depend on links or sitemaps alone. Cloudflare and OpenAI published early results from their joint pilot on 15 October 2026, and the mechanism matters more to developer sites than the headline number does.
This is not a new crawler or a new robots.txt directive. It is a signal feed from the network that sits in front of a large share of the web, and it changes which of your pages get crawled, skipped or revisited.
What Cloudflare and OpenAI actually announced
The Cloudflare blog post, written by two Cloudflare and two OpenAI authors, reports results from a research pilot that started in July. Cloudflare's network provides discovery signals to OpenAI's search crawler; OpenAI then evaluates and ranks the pages itself.
- The participating cohort was about 1% of Cloudflare's total network.
- The pilot helped OpenAI discover and index "hundreds of millions" of new pages.
- The work uses metadata about the web for search and discovery, not page content for model training.
- The post states that "a signal about a page doesn't grant permission to access it". Crawler access settings still apply.
The original July 8 press release named three signal types: content freshness, traffic quality and actual page changes. It also noted that Cloudflare's network sits in front of more than 20% of the web.
What the companies did not say
Search Engine Journal's analysis points out the gaps: no crawl-volume or server-load figures, no data on how often newly found pages were cited in ChatGPT, no referral numbers, and no list of participating sites. "Indexed" is not "cited". Treat the pilot as a crawl-efficiency story until someone publishes outcome data.
The mechanism: filter before fetching
The interesting part is what the signals tell the crawler not to do. The post describes four jobs:
| Signal job | What it catches | What it means for you |
|---|---|---|
| Discovery | Pages the crawler has never seen | New pages can surface without inbound links |
| Filtering | Bots requesting nonexistent files, sites redirecting almost everything to the homepage, duplicate content | Soft 404s and catch-all redirects get your URLs skipped |
| Harm flagging | Individual harmful pages, such as a planted fake login form | Flags target pages, not whole domains |
| Change signals | Whether a page changed enough to revisit | Real updates get recrawled; cosmetic ones should not |
That last row connects to a number from Cloudflare's July 1 post: its data suggests more than 50% of crawl traffic from good bots goes to re-fetching pages that have not changed. A crawler that knows what changed can spend that budget on pages that did.
There is a short-term cost. Per Search Engine Journal, the post notes that finding new pages can initially increase crawling while those pages are evaluated. If you run a large catalogue on a small origin, watch your logs.
Which OpenAI bot is which
The pilot only helps you if the search crawler is allowed in. OpenAI's crawler documentation separates the bots clearly:
| User agent | Purpose | robots.txt |
|---|---|---|
OAI-SearchBot | Surfaces sites in ChatGPT search | Respected; opting out removes you from search answers, though navigational links may still appear |
GPTBot | Crawls content that may be used to train foundation models | Respected; disallow to signal no training |
ChatGPT-User | User-initiated visits from ChatGPT and Custom GPTs | May not apply; not used for automatic crawling or search inclusion |
OAI-AdsBot | Checks landing pages submitted as ChatGPT ads | Visits only submitted pages |
Two details from the same page change how you write rules. Each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. And if you allow both, OpenAI may use a single crawl for both purposes. Changes to robots.txt for search can take about 24 hours to apply.
# Stay in ChatGPT search, opt out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Build and check rules like these with a robots.txt generator. If you also publish training preferences in other formats, an ai.txt file can sit alongside, but robots.txt is the control OpenAI documents. Verify that requests claiming to be OAI-SearchBot come from the IP ranges OpenAI publishes at openai.com/searchbot.json; user-agent strings alone are trivially spoofed. Our piece on signed crawler verification covers the stronger option.
Where Cloudflare's own settings fit
If your site is on Cloudflare, its September changes sit on top of this. The mixed-use crawler post introduced domain-level Search, Training and Agent controls and a Disallow AI Training setting that publishes a no-training preference in robots.txt through Bot Preference Sync. Cloudflare lists OpenAI's training-only crawler among those it categorizes as Accountable, so it can be blocked without affecting search.
Settings carry over automatically, but check two things: that the Search control is not set to Block, and that any legacy WAF rule written against "AI bots" does not also catch OAI-SearchBot. The SEJ article points to Cloudflare's AI Crawl Control, available on every plan, for seeing which AI crawlers actually reach you.
What to change on a developer site this week
You cannot join the pilot; Cloudflare says broader access is planned for later in 2026 with no date. But every signal it uses rewards the same hygiene, and that hygiene helps every crawler:
- Return real 404s. Catch-all redirects to the homepage are explicitly listed as a waste pattern. A deleted product or tool page should 404 or 410, or redirect to a genuine replacement.
- Make freshness honest. Change signals look at whether a page actually changed. Bumping
lastmodor a visible "updated" date on every build teaches crawlers to ignore both. Keep your XML sitemap dates tied to content edits. - Collapse duplicates. Parameter and case variants of the same page are the duplicate content the filter skips. Pick one URL per page.
- Write rules per bot. Decide search and training separately, then confirm in logs that the bot you allowed is getting 200s.
- Measure referrals properly. SEJ notes ChatGPT referral tracking depends on UTM tags or referrers, and that some ChatGPT links lacked both earlier in October.
What this does not require is an llms.txt file or any special markup. The pilot runs on network behaviour, not on files you publish. If you already handled Meta's new crawl terms, the per-bot thinking is the same.
FAQ
Can I opt my site into the Cloudflare and OpenAI pilot?
Not today. Neither company has said which sites participate, and Cloudflare says it intends to make the capability broadly available later in 2026 without giving a date.
Does the pilot send my page content to OpenAI for training?
The 15 October post says the research uses metadata about the web for search and discovery, not page content for model training, and that a signal about a page does not grant permission to access it.
Which bot do I need to allow to appear in ChatGPT search?
OAI-SearchBot. OpenAI's documentation says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers. GPTBot controls training and can be disallowed independently.
Will the pilot get my pages cited in ChatGPT?
Not necessarily. OpenAI evaluates and ranks discovered pages independently, and the companies have not published citation or referral data from the pilot, so discovery and indexing are the only reported outcomes.
Does ChatGPT-User respect robots.txt?
Not reliably. OpenAI says ChatGPT-User handles user-initiated actions, so robots.txt rules may not apply, and it is not used for automatic crawling or to decide whether content appears in search.
The pilot rewards sites whose URLs tell the truth: real 404s, real update dates, one address per page. Fix those first.