Ai2Bot-Dolma: what it is, what it does with your content, and what blocking it costs
What Ai2Bot-Dolma is
Ai2Bot-Dolma collects pages to train future models. It does not decide whether Allen Institute for AI can cite you in an answer today — that is a different crawler with a different token. Blocking Ai2Bot-Dolma is a decision about your content being learned from, not about your visibility.
Operator: Allen Institute for AI
robots.txt token: Ai2Bot-Dolma
User-agent on the wire: not published by the operator
Category: Model training
What blocking it actually costs
The crawler behind the Dolma open training corpus.
How to block Ai2Bot-Dolma
Add this to your robots.txt:
User-agent: Ai2Bot-Dolma
Disallow: /
The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.
How to allow Ai2Bot-Dolma
User-agent: Ai2Bot-Dolma
Allow: /
An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.
robots.txt is not the only thing that can block it
A permissive robots.txt does not mean Ai2Bot-Dolma can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes Ai2Bot-Dolma and whose edge returns 403 to it is blocked in every way that matters — and nothing in robots.txt will tell you so.
This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.
Official documentation
Allen Institute for AI documents Ai2Bot-Dolma at https://allenai.org/crawler.
Common questions
- What user-agent does Ai2Bot-Dolma send?
Allen Institute for AI does not publish an exact user-agent string for Ai2Bot-Dolma. Match on the
Ai2Bot-Dolmatoken in robots.txt rather than on a full string you found elsewhere.- How do I block Ai2Bot-Dolma?
Add a group to robots.txt naming the token exactly:
User-agent: Ai2Bot-Dolma Disallow: /The group must name
Ai2Bot-Dolmaon its own line. AUser-agent: *group does not combine with a named one — under RFC 9309 the most specific matching group replaces the wildcard entirely, it does not inherit from it.- What does blocking Ai2Bot-Dolma cost me?
The crawler behind the Dolma open training corpus.
- Can I tell a real Ai2Bot-Dolma request from a fake one?
Not reliably. Allen Institute for AI does not publish verified IP ranges for Ai2Bot-Dolma, so the user-agent string is the only signal, and anyone can send it. Treat robots.txt as a statement of policy rather than as enforcement.
Is Ai2Bot-Dolma blocked on your site right now?
Reading your own robots.txt only answers half of it — a firewall rule can block Ai2Bot-Dolma while robots.txt says it is welcome. The checker tests both.
Check your site free