CCBot: what it is, what it does with your content, and what blocking it costs

What CCBot is

CCBot collects pages to train future models. It does not decide whether Common Crawl can cite you in an answer today — that is a different crawler with a different token. Blocking CCBot is a decision about your content being learned from, not about your visibility.

Operator: Common Crawl
robots.txt token: CCBot
User-agent on the wire: CCBot/2.0 (https://commoncrawl.org/faq/)
Category: Model training

What blocking it actually costs

The highest-leverage single block for training opt-out: Common Crawl is not an AI company, but its archive is an input to a large number of models. Blocking it is retroactively useless — past crawls are already published.

How to block CCBot

Add this to your robots.txt:

User-agent: CCBot
Disallow: /

The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.

How to allow CCBot

User-agent: CCBot
Allow: /

An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.

robots.txt is not the only thing that can block it

A permissive robots.txt does not mean CCBot can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes CCBot and whose edge returns 403 to it is blocked in every way that matters — and nothing in robots.txt will tell you so.

This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.

Official documentation

Common Crawl documents CCBot at https://commoncrawl.org/ccbot.