Cloudflare now lets every site decide — per bot — whether AI crawlers get in free, get blocked at the edge, or pay per crawl. Set the switches and watch what happens.
They look identical in your logs but do very different things:
robots.txt is a polite note on the door — compliant bots read it, others walk past. Managed enforcement at a CDN edge like Cloudflare actually verifies bot identity (signatures, IP ranges, behavior) and drops or challenges the request before it touches your origin. A rule you can't enforce is a preference, not a policy.
The middle switch. Instead of a binary allow/block, the edge returns HTTP 402 Payment Required with a price; crawlers that present payment credentials get the page, and you get paid per fetch. It turns "scrape me for free" into a licensing market — especially relevant for publishers whose traffic is being replaced by AI answers.
Block every AI bot and your content stops leaking — and you also vanish from ChatGPT, Claude, Perplexity and Google AI answers, where a growing share of discovery now happens. The strategic question isn't "block or not," it's which crawl types create value for you and which just extract it. Most sites land on: allow search/RAG, control or charge training.
| User-agent | Operator | Primary purpose | What allowing it gets you |
|---|---|---|---|
GPTBot | OpenAI | Model training corpus | Little directly — your text may inform future GPT models |
OAI-SearchBot | OpenAI | ChatGPT search index | Citations & links inside ChatGPT answers |
ChatGPT-User | OpenAI | Live user-triggered fetch | Your page shown when a user asks about it right now |
ClaudeBot | Anthropic | Training + indexing | Presence in Claude's knowledge and web answers |
Google-Extended | Gemini training opt-out token | Blocking it does not remove you from Google Search | |
PerplexityBot | Perplexity | AI search index | Cited sources with outbound links in Perplexity |
CCBot | Common Crawl | Open web archive | Feeds many research datasets — and many model trainers |
Key nuance: one company can run several bots with different jobs. Blocking GPTBot (training) while allowing OAI-SearchBot (search) is exactly the kind of split policy edge-level controls make possible — robots.txt alone can express it, but only enforcement makes it real.
Sell attention? Charge the bots. Sell products? Let AI search in so assistants can recommend you. Ship developer docs? Allow broadly — being the answer an AI gives is your marketing. There is no universally right switch — only the right switch for how your site makes money.