Robots.txt and Crawl Policy Interpretation for Autonomous Agents

Robots.txt was built for a single kind of visitor: a search indexer that fetched pages, ranked them, and sent traffic back in exchange. That bargain does not hold the same way for the population of agents reading the file now, because those agents arrive with different jobs. Some train a model on what they read. Some answer a live query for a specific user. Some build a search index that will cite the page later, if at all. The old assumption, crawl now, get traffic later, only describes one of these jobs correctly.

The file itself has not moved an inch since it was standardized. It remains a plain-text, voluntary request under RFC 9309, with no authentication layer and no self-enforcing legal weight in US courts. A comparison from the Ziff Davis v. OpenAI litigation put it well: robots.txt functions like a "No Trespassing" sign, a notice that a well-behaved visitor honors and a determined one can walk past.

What has changed sits entirely on the other side of the fence. Dozens of purpose-specific crawlers now operate across six major AI companies, and each one carries a declared job that a site owner can permit or refuse on its own terms. Autonomous agents in 2026 arrive with fundamentally different purposes, training a model, answering a live user query, building a search index, and the deal of crawling in exchange for referral traffic does not hold equally for all of them.

The four functional categories that make crawl policy legible

"AI crawler" stopped being a useful category once the number of distinct jobs an agent might be doing multiplied. At least four functionally separate roles now hide under that one label, and treating them as one thing is where site owners make policy errors that cost them traffic, model presence, or both.

Training crawlers exist to collect content that improves a future model's weights. They cost a site bandwidth and return nothing in the way of referral traffic, now or later. GPTBot, run by OpenAI, ClaudeBot, run by Anthropic, meta-externalagent, run by Meta, and CCBot, the general-purpose crawler behind the Common Crawl archive that other labs draw on for training data, all fall into this bucket. Blocking one of these tokens takes a site out of some future model's training set. It does nothing to the site's current standing in AI search results, because that's a separate system entirely.

Search indexers are the crawlers that live AI products query when they answer a user and cite a source. OAI-SearchBot handles this for OpenAI, Claude-SearchBot for Anthropic, and PerplexityBot for Perplexity. OpenAI's documentation states that a site blocking OAI-SearchBot will not show up in ChatGPT Search answers. Anthropic goes further and insists that ClaudeBot, Claude-SearchBot, and Claude-User each get addressed on their own terms in robots.txt, retiring the older Claude-Web and Anthropic-AI names that used to lump these functions together.

User-triggered fetchers make up the third category, and they behave the least like a crawler in the traditional sense. ChatGPT-User, Claude-User, Perplexity-User, and meta-externalfetcher all retrieve a single page because a person, mid-conversation, asked the assistant to look at it. This is the category where robots.txt starts to strain against its own design. OpenAI states that robots.txt rules may not apply to ChatGPT-User, and Perplexity says the same of Perplexity-User, because a fetch triggered by a specific human request doesn't fit the bulk-crawling model the file was written to govern. Managing this category well means working at the CDN or WAF layer instead of relying on the text file to do a job it was never built for.

The fourth category never generates its own log entries, because usage-control tokens like Google-Extended and Applebot-Extended don't crawl; they modify what another crawler is allowed to do with what it already collected. Google-Extended governs whether the pages Googlebot indexes can also feed Gemini training or certain grounding uses, without touching Google Search inclusion one way or the other. Applebot-Extended does the equivalent job for Apple, controlling whether Applebot's harvest can train Apple's models, while Applebot itself keeps indexing for Siri, Spotlight, and Safari regardless of that setting.

The practical consequence of this taxonomy is that a site that writes a single wildcard Disallow to block "AI bots" simultaneously removes itself from AI search citations, blocks user-triggered fetches that drive high-intent traffic, and may not even reach the training crawlers it most wanted to stop.

The full 2026 agent reference: 14 tokens across six companies

Fourteen user-agent tokens across six companies make up the documented landscape as of mid-to-late 2026, and the decisive insight is that "AI crawler" is not a single category: it describes at least four functionally distinct roles, and conflating them produces policy errors with real business consequences.

OpenAI runs GPTBot as a training crawler that can optionally be blocked, OAI-SearchBot as a search indexer that should be allowed, and ChatGPT-User as a user fetcher that should be allowed. Anthropic mirrors that split with ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user fetches. Perplexity runs a leaner setup: PerplexityBot covers both indexing and retrieval for cited answers in one token, and Perplexity-User handles user-triggered fetches, which Perplexity's own documentation states generally ignore robots.txt regardless of what the file says.

Google's pair works differently from the other vendors. Googlebot is the search crawler, covering search including AI Overviews, while Google-Extended is a usage-control token governing Gemini training and some grounding uses. Meta splits along the training-versus-fetch line: meta-externalagent handles training and AI feature development, and meta-externalfetcher handles the user-triggered case. Apple's pair follows the same logic: Applebot is the search indexer powering Siri, Spotlight, Safari, and AI answers, and Applebot-Extended is the usage-control token gating whether that same data trains Apple's models.

Two of these fourteen tokens never appear as their own line in a server access log. Google-Extended and Applebot-Extended are directives that a site owner scanning logs for evidence that they're "working" is checking the wrong place for. Googlebot deserves its own note of caution: because it powers both ordinary search results and AI Overviews at once, there's no way to block the AI exposure without also blocking the site from conventional Google Search. The two functions ride on one token, and Google-Extended, not Googlebot, is the lever built for the training and grounding opt-out.

How to make the training-versus-search decision for each vendor

Training and search have been separated by design across every major vendor covered here, and that separation is what makes an independent decision on each front possible without sacrificing the other. OpenAI shows this cleanly: blocking GPTBot removes a site's content from future training runs, while allowing OAI-SearchBot keeps that same site eligible for citation in ChatGPT Search, and the two settings don't interact with each other in OpenAI's documented architecture. Anthropic asks for the same discipline across three tokens rather than two. ClaudeBot, Claude-SearchBot, and Claude-User each need their own line, and turning off ClaudeBot has no bearing on whether Claude-SearchBot or Claude-User can still reach the site. Perplexity collapses that choice into a single decision, since no separate training crawler is documented for the company: PerplexityBot handles both indexing and retrieval for cited answers, so a block here affects search eligibility immediately rather than sitting quietly as a future-training-only decision. Google keeps the lines cleanly divided as well. Google-Extended is the correct opt-out for Gemini training and certain grounding uses, Googlebot has to stay allowed to preserve both Search and AI Overviews presence, and turning off Google-Extended carries no measurable effect on Search ranking according to Google's documented behavior. Meta and Apple round out the picture with the same independence: meta-externalagent, the training crawler, can be blocked without touching meta-externalfetcher's ability to serve a live user request, and Applebot-Extended, the training opt-out, operates apart from Applebot's ongoing work indexing for Siri, Spotlight, and Safari.

These goals cover most of what a site owner is actually trying to accomplish, and each maps to a specific configuration rather than a vague sentiment about "being more careful with AI. A site chasing maximum AI search visibility allows every token for a given vendor and accepts training crawler access as the cost of that visibility. A site specifically after ChatGPT visibility without feeding OpenAI's training pipeline allows OAI-SearchBot and ChatGPT-User while blocking GPTBot. A site that wants to keep its place in Google Search and AI Overviews without contributing to Gemini training allows Googlebot and sets Google-Extended to Disallow. And a site pursuing a complete AI opt-out blocks every training crawler and every search indexer at once, accepting as the trade-off that it will not appear in AI search citations from any of these vendors.

Anthropic litigation is a reminder of what happens when that gap between stated preference and actual behavior ends up in front of a judge. The case survived a federal demurrer in September 2026, and the underlying dispute turns in part on whether crawling that ignored a site's stated terms carries consequences beyond the crawl itself. Robots.txt instructions that go unheeded, in other words, are no longer purely a technical footnote; they can become an exhibit.

What robots.txt cannot control

Even a perfectly written robots.txt governs less ground than most site owners assume, and two structural gaps explain why.

The first gap sits with user-triggered fetches. When someone asks an AI assistant to summarize a specific page, providers generally treat the resulting fetch as a user-directed action rather than as crawling, which is a different category in their own architecture and, often, in their own compliance logic. OpenAI says outright that robots.txt rules may not apply to ChatGPT-User, and Perplexity says its Perplexity-User fetches generally ignore robots.txt as well. Neither statement describes rogue behavior. Both describe a fetch triggered by a live human request rather than a bulk indexing run, and the practical response is to manage that traffic at the CDN or WAF layer, through rate-limiting, IP allowlisting, or authentication, rather than expecting a text file to stop it.

The second gap runs deeper, because it concerns identity rather than intent. A user-agent string is a name a crawler chooses for itself, and the protocol has no mechanism to verify that a request labeled GPTBot is actually coming from OpenAI's infrastructure. Cloudflare's August 2025 investigation documented that when PerplexityBot, Perplexity's declared crawler, was blocked by robots.txt directives and WAF rules, undeclared crawlers impersonating Chrome on macOS, rotating through ranges outside Perplexity's published ranges, continued fetching content at scale across tens of thousands of domains. Nothing about that pattern is specific to Perplexity. Any scraper can adopt a trusted name and inherit whatever allowances a site's robots.txt grants that name.

Compliance data adds scale to the concern. The 2025 AI Agent Index, surveying 30 deployed agents, found that only 6 explicitly stated their crawlers respect robots.txt, while 16 offered no clear statement on the matter at all. Compliance depends on voluntary adherence, and that honor system holds for the major, documented agents named throughout this piece. It does not extend nearly as far across the wider population of agents now operating on the open web.

None of this makes robots.txt worthless. It functions as a norm-setting instrument for the agents that choose to respect it, and enforceable access control has to happen a layer further down, at the server or the CDNc45.

CDN-layer enforcement and the Cloudflare default that may already apply to your site

Enforceable policy, as opposed to a polite request, executes at the CDN layer, and that is where a site owner who wants real control needs to look. Cloudflare reported more than a million customers had activated its AI-bot blocking tools by July 2025; Bytespider traffic dropped sharply across its network once one-click blocking went live, and GPTBot turned up as the most-crawled AI bot on the network, reaching a large share of the highest-traffic domains it monitors.

Cloudflare's Content Signals Policy, launched on September 24, 2025, changed the texture of what robots.txt could say. It added plain-language directives that split usage into three separate categories: search, AI answers, and AI training, and rolled that structure out across more than 3.8 million domains on Cloudflare's managed robots.txt, with defaults set to allow search, block AI training, and leave AI answers neutral. Any site with its robots.txt managed by Cloudflare may already carry a Content Signals Policy on the domain, even if the site owner never wrote one.

A newer default raises the stakes further. Starting September 15, 2026, Cloudflare's settings block "mixed-use" crawlers, the ones that blend search, agent use, and training into a single token, from any page carrying ads, unless the site owner actively changes that setting. A site that depends on AI answer engines for referral traffic and also runs advertising on its pages now faces a conflict built into its own CDN configuration by default, not by any choice it made.

Cloudflare's pay-per-crawl option points toward a different kind of resolution: rather than a flat 403 rejection or an open page serve, a verified AI crawler hitting a participating domain receives a 402 Payment Required response, along with pricing and licensing terms. It's a mechanism suited to publishers whose content is distinctive enough that a model builder would rather pay for access than do without it.

Research on Cloudflare's network found a meaningful share of sites accidentally blocking major AI crawlers at the CDN while their robots.txt says allow, and when the two layers disagree, the effective policy is whatever the CDN enforces. The two layers have to agree, or the site's actual policy is whichever layer enforces harder.

The compliance picture: who follows robots.txt and how EU enforcement changes it

Compliance among the major, named agents holds up well. Googlebot, GPTBot, ClaudeBot, and PerplexityBot all document their user-agent strings publicly and state that they follow standard robots.txt directives. That's the reassuring half of the picture, and it covers the agents a site owner is most likely to encounter by name.

The wider landscape looks less orderly: the AI Agent Index findings from the previous section show a documented minority state compliance outright, while a much larger group leaves the question unanswered. Enforcement used to stop there, at the boundary of good faith and server-side blocking. It no longer does. Since August 2, 2026, an AI company that ignores a site's robots.txt risks a fine under the EU AI Act's enforcement regime of up to €15 million or 3 percent of worldwide annual turnover, whichever figure is higher.

A December 2025 US court ruling had gone the other direction, finding that ignoring robots.txt did not amount to illegal circumvention and comparing the file's authority to a sign asking visitors to keep off the grass. The EU's regime sits on top of that same technical reality and changes what disregarding the sign can cost a company operating in that jurisdiction. Robots.txt remains, at its core, a request rather than a lock.