How AI Crawlers Work (And What They Want From Your Site)

The one distinction to understand first

There is no single AI crawler. Bots arrive at your server with three different jobs, and the same robots.txt rule affects each job differently. Knowing which bot does which job is the difference between an access policy you chose and one that happened to you.

The three jobs AI crawlers do

Training crawlers collect content to improve future models. OpenAI's GPTBot and Anthropic's ClaudeBot are the headline examples. They crawl broadly, store nothing you can query, and their work shows up months later as what a model appears to know about your domain. Blocking them does not remove you from today's answers; it removes you from tomorrow's memory.

Search-index crawlers build the live index that answer engines search before they answer. OAI-SearchBot feeds ChatGPT Search; PerplexityBot feeds Perplexity. This is the category most likely to earn you a citation with a link today. These bots re-crawl over time, so keeping them in is what keeps your content current inside answers.

Assistant fetchers act on a user's explicit request. When a ChatGPT user asks the model to read a specific page, ChatGPT-User fetches it in real time — one fetch, one conversation, nothing stored or indexed. Blocking these bots breaks a feature your users asked for; it does not touch your visibility in any index.

The bots that matter

Bot	Operator	Job	If you block it
GPTBot	OpenAI	Training	Future OpenAI models learn less about your domain
OAI-SearchBot	OpenAI	Search index	ChatGPT Search cannot index or cite you
ChatGPT-User	OpenAI	User-triggered	ChatGPT users cannot make it read your pages on request
ClaudeBot	Anthropic	Training	Future Claude models learn less about your domain
PerplexityBot	Perplexity	Search index	Perplexity answers stop citing and linking to you
Google-Extended	Google	Training control	Gemini loses training access; Google Search and AI Overviews are unaffected
Applebot-Extended	Apple	Training control	Apple Intelligence training blocked; Siri and Spotlight unaffected
Bytespider	ByteDance	Training	Little visible loss for most sites — and it may ignore robots.txt anyway

Two entries deserve a footnote. Google-Extended and Applebot-Extended are not conventional crawlers; they are control tokens that decide whether your content feeds model training at those companies. Blocking them changes training access only — the general crawlers keep working. The full crawler directory covers these plus dozens of smaller bots.

How they actually crawl

No JavaScript, almost always. The big search engines render pages before indexing. Most AI crawlers fetch raw HTML and that is it. Content that only exists after client-side rendering is invisible to them. If your key facts are present in the view-source HTML, you are fine; if they arrive via JavaScript, they may as well be password-protected.

Politeness varies. The major operators rate-limit, identify themselves honestly and spread requests over time. Others hit your server in bursts from rotating address ranges. Practical advice: watch your logs for the user agents above, confirm the fetch patterns are sane, and only then trust the label on the request.

They identify themselves — verify it anyway. Every major operator publishes its exact user-agent string and, in most cases, its IP ranges. A fetch claiming to be GPTBot that does not resolve back to an OpenAI hostname is not GPTBot. Match your access log entries against the published rosters and treat anything that fails the reverse-DNS check as an unverified scraper, whatever its user agent claims to be.

robots.txt compliance is the norm, not the law. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended all honor robots.txt directives. Bytespider is the well-known exception — it has been documented crawling sites that explicitly disallow it, so a robots.txt-only block is not reliable enforcement against it; server-level blocking is. Compliance is a promise made by the operator, not a feature of your server, which is why it pays to verify what your robots.txt actually tells each bot instead of assuming.

What happens after the fetch

Fetched pages are rarely treated as single units. A page gets split into passages — chunks of a few sentences to a few paragraphs, usually aligned with headings. Each chunk is converted into a vector, a numeric representation of its meaning. When a user asks a question, the engine embeds the question the same way and retrieves the chunks that sit closest to it in meaning. Only the retrieved chunks — not your whole page — compete to be quoted in the answer.

This is why structure beats length. A heading that states the question, followed by a short paragraph that answers it, forms a chunk that can win retrieval on its own. The same point buried mid-page inside a 2,000-word wall of text competes badly. The content structure guide turns this into concrete formatting rules you can apply to existing pages.

Two practical consequences follow. First, updates propagate at crawl speed: a corrected price or a fixed fact goes live in answers only after the relevant bots re-fetch the page, which can take days to weeks. Second, deleted pages linger — engines keep serving chunks from their last successful fetch until the index refreshes on its own schedule. Changing content or access rules does not flip a switch anywhere; patience is part of the workflow.

Access is step zero

Everything above reduces to one sentence: if the right bots cannot fetch your pages, nothing else you do matters. Not content quality, not authority, not schema markup. A blocked search-index bot means no citations. A blocked training bot means fading out of model memory. And most sites that actually check discover rules they never consciously chose — a leftover disallow from an old redesign, a host firewall rule, or a CDN bot-fight mode that treats every unfamiliar user agent as an attack.

So start there. Audit your robots.txt per bot, write the policy you actually want using the robots.txt for AI crawlers guide, and only then invest in content and structure. Then learn the operators themselves in the crawler directory — knowing the bot is knowing the buyer.