Ai2Bot-Dolma: what it is, what it does with your content, and what blocking it costs
What Ai2Bot-Dolma is
Ai2Bot-Dolma collects pages to train future models. It does not decide whether Allen Institute for AI can cite you in an answer today. That is a different crawler with a different token. Blocking Ai2Bot-Dolma is a decision about your content being learned from, not about your visibility.
Operator: Allen Institute for AI
robots.txt token: Ai2Bot-Dolma
User-agent on the wire: not published by the operator
Category: Model training
What blocking it actually costs
The crawler behind the Dolma open training corpus.
How to block Ai2Bot-Dolma
Add this to your robots.txt:
User-agent: Ai2Bot-Dolma
Disallow: /
The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.
How to allow Ai2Bot-Dolma
User-agent: Ai2Bot-Dolma
Allow: /
An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.
robots.txt is not the only thing that can block it
A permissive robots.txt does not mean Ai2Bot-Dolma can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes Ai2Bot-Dolma and whose edge returns 403 to it is blocked in every way that matters, and nothing in robots.txt will tell you so.
This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.
Official documentation
Allen Institute for AI documents Ai2Bot-Dolma at https://allenai.org/crawler.
Common questions
- What user-agent does Ai2Bot-Dolma send?
Allen Institute for AI does not publish an exact user-agent string for Ai2Bot-Dolma. Match on the
Ai2Bot-Dolmatoken in robots.txt rather than on a full string you found elsewhere.- How do I block Ai2Bot-Dolma?
Add a group to robots.txt naming the token exactly:
User-agent: Ai2Bot-Dolma Disallow: /The group must name
Ai2Bot-Dolmaon its own line. AUser-agent: *group does not combine with a named one. Under RFC 9309, the most specific matching group replaces the wildcard entirely, it does not inherit from it.- What does blocking Ai2Bot-Dolma cost me?
The crawler behind the Dolma open training corpus.
- Can I tell a real Ai2Bot-Dolma request from a fake one?
Not reliably. Allen Institute for AI does not publish verified IP ranges for Ai2Bot-Dolma, so the user-agent string is the only signal, and anyone can send it. Treat robots.txt as a statement of policy rather than as enforcement.
Is Ai2Bot-Dolma blocked on your site right now?
Reading your own robots.txt only answers half of it, because a firewall rule can block Ai2Bot-Dolma while robots.txt says it is welcome. The checker tests both.
Check your site free