Claude-User: what it is, what it does with your content, and what blocking it costs

What Claude-User is

Claude-User fetches a page because a person asked for it. They pasted your URL, or followed a link, and asked the assistant to read it. There is no crawl schedule and no training corpus involved. Blocking it means failing a visitor who explicitly asked for your page.

Operator: Anthropic
robots.txt token: Claude-User
User-agent on the wire: not published by the operator
Category: User-triggered fetch

What blocking it actually costs

Claude cannot fetch your page when a user asks it to read a specific URL.

How to block Claude-User

Add this to your robots.txt:

User-agent: Claude-User
Disallow: /

The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.

How to allow Claude-User

User-agent: Claude-User
Allow: /

An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.

robots.txt is not the only thing that can block it

A permissive robots.txt does not mean Claude-User can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes Claude-User and whose edge returns 403 to it is blocked in every way that matters, and nothing in robots.txt will tell you so.

This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.

Official documentation

Anthropic documents Claude-User at https://support.anthropic.com/en/articles/8896518.