The AI Crawler Directory
36 crawlers and robots.txt control tokens from 21 companies. Grouped by who runs them, because that is how the decision gets made, and tagged by what each one does, because blocking a training crawler and blocking a retrieval crawler have very different consequences.
- Feeds answers today
- Live retrieval. Builds the index assistants answer from. Blocking this is what costs you citations right now.
- Fetches on request
- User-triggered fetch. Fetches a page because a person asked for it. Blocking fails a visitor who wanted you.
- Trains future models
- Model training. Collects pages to train future models. Blocking changes nothing about today’s answers.
OpenAI
3 crawlersRuns three separate crawlers with three different jobs. One wildcard rule blocks all of them, which is almost never what anyone means.
- GPTBot
Collects pages to train future OpenAI models. Does not affect whether ChatGPT can cite you today.
Trains future models - OAI-SearchBot
Builds the index ChatGPT search looks through when it needs a source to cite.
Feeds answers today - ChatGPT-User
Fetches a page when a ChatGPT user pastes the link or asks about that URL.
Fetches on request
Anthropic
4 crawlersSeparates training, search indexing and user-triggered fetches into distinct tokens.
- ClaudeBot
Collects pages to train future Claude models. Separate from the tokens that govern citation.
Trains future models - Claude-SearchBot
Builds the index Claude searches when it needs a source to cite.
Feeds answers today - Claude-User
Fetches a page when a Claude user shares the link or asks about that URL.
Fetches on request - anthropic-ai
A legacy Anthropic training token. Current crawling identifies itself as ClaudeBot.
Trains future models
Google-Extended is a setting rather than a crawler. Blocking it does not remove you from Search or AI Overviews.
- Google-Extended
A robots.txt setting, not a crawler. Controls Gemini training and grounding only.
Trains future modelsSetting, not a crawler - Googlebot
Builds the Google index that feeds both Search results and AI Overviews.
Feeds answers today - Google-CloudVertexBot
Fetches sites that Vertex AI customers ask Google to index for their own grounding.
Trains future models
Perplexity
2 crawlersMicrosoft
1 crawlerMeta
3 crawlersApple
2 crawlersApplebot-Extended governs Apple Intelligence training only; Applebot keeps crawling regardless.
Amazon
1 crawlerByteDance
1 crawlerWidely reported to crawl aggressively and honour robots.txt inconsistently.
Common Crawl
1 crawlerNot an AI company, but its archive is an input to a large number of models.
Mistral AI
1 crawlerDuckDuckGo
1 crawlerCohere
2 crawlersYou.com
1 crawlerAllen Institute for AI
2 crawlersHuawei
2 crawlersDiffbot
1 crawlerImageSift
1 crawlerTimpi
1 crawlerWebz.io
2 crawlersZyte (open source)
1 crawlerWhich of these
can reach your site?
Check your own domain against every crawler in this directory. Reads your robots.txt, names the exact line behind each verdict, and sends a real request as each crawler whose user-agent its operator publishes.