Claude-User: what it is, what it does with your content, and what blocking it costs

What Claude-User is

Claude-User fetches a page because a person asked for it — they pasted your URL, or followed a link, and asked the assistant to read it. There is no crawl schedule and no training corpus involved. Blocking it means failing a visitor who explicitly asked for your page.

Operator: Anthropic
robots.txt token: Claude-User
User-agent on the wire: not published by the operator
Category: User-triggered fetch

What blocking it actually costs

Claude cannot fetch your page when a user asks it to read a specific URL.

How to block Claude-User

Add this to your robots.txt:

User-agent: Claude-User
Disallow: /

The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.

How to allow Claude-User

User-agent: Claude-User
Allow: /

An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.

robots.txt is not the only thing that can block it

A permissive robots.txt does not mean Claude-User can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes Claude-User and whose edge returns 403 to it is blocked in every way that matters — and nothing in robots.txt will tell you so.

This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.

Official documentation

Anthropic documents Claude-User at https://support.anthropic.com/en/articles/8896518.