# Free Tools

Seerly publishes two free diagnostic tools plus a reference directory of AI crawlers. They answer the technical questions that decide whether an AI assistant can read and quote a site at all — questions that come before any question about whether it chooses to.

Running a check requires a free Seerly account. There is no card and no limit on the number of checks. Results are held in a short-lived cache so a site checked repeatedly is only crawled once, and are not stored beyond that.

## AI Crawler Access Checker

URL: https://seerly.app/tools/ai-crawler-checker

Checks a domain against 36 AI crawlers and robots.txt control tokens, and reports for each one whether it is allowed or blocked — naming the exact robots.txt line and line number that decided it.

It then goes further than a robots.txt parser. For six major crawlers it sends a real HTTP request carrying that crawler's user-agent and compares the response to a control request made as an ordinary browser. This catches firewall, WAF and CDN rules that robots.txt says nothing about: a site whose robots.txt welcomes GPTBot while its edge returns 403 to it is blocked in every way that matters, and only a real request reveals that.

Findings from the live request are reported only where the browser control succeeded and the crawler request did not, which isolates refusals caused by the user-agent itself. Because the requests originate from Seerly's own addresses rather than the operator's published IP ranges, findings for operators that publish such ranges are marked medium confidence.

### The three kinds of crawler, and why the distinction matters

Results are grouped by what each crawler actually does, because the three have very different consequences and most robots.txt files treat them identically by accident:

- **Live retrieval** crawlers build the index an assistant answers from. Blocking these is what costs citations today.
- **User-triggered** fetchers retrieve a page because a person pasted its URL and asked about it. Blocking these fails a visitor who explicitly asked for the page.
- **Training** crawlers collect pages to train future models. Blocking these changes nothing about today's answers.

A single `User-agent: *` `Disallow: /` rule blocks all three. Most sites that have one intended only the third.

## AI Search Readiness Check

URL: https://seerly.app/tools/ai-readiness-check

Grades a single page against 58 checks covering what an assistant needs in order to quote it: whether content exists in the served HTML without JavaScript, heading structure, extractable summaries, structured data, freshness signals, and the conventional technical SEO foundations.

The most important check is whether the page renders without JavaScript. GPTBot, ClaudeBot and PerplexityBot fetch HTML and do not execute JavaScript, so a page whose content appears only after client-side rendering presents them an empty shell — no headings, no schema, nothing to extract. No other improvement matters until that is fixed.

Two scores are reported separately rather than averaged: AI readiness and technical SEO. A page can score well on conventional SEO and still be invisible to AI crawlers, and a single blended number would hide exactly that.

Checks that cannot apply to the page are excluded rather than counted as failures — Product schema on a pricing page, LocalBusiness on a company with no premises, hreflang on a single-language site. Checks that require comparing pages against each other, such as duplicate titles, broken links and orphan pages, need a full-site crawl and are named in the result rather than silently omitted.

## AI Crawler Directory

URL: https://seerly.app/tools/ai-crawler-checker/bots

A reference page for each of the 36 crawlers and control tokens, grouped by the company that operates them. Each entry documents the user-agent string, the robots.txt token, the operator, what the crawler does with content, the exact rules to allow or block it, and what blocking it actually costs.

Two entries are commonly misunderstood and are documented explicitly:

- **Google-Extended** is a robots.txt control token with no crawler behind it. Google crawls with Googlebot and consults the Google-Extended group only to decide whether content may be used for Gemini training and grounding. Blocking it does not remove a site from Google Search or from AI Overviews.
- **Applebot-Extended** works the same way for Apple Intelligence. Applebot continues crawling for Siri and Spotlight regardless.

## Relationship to the Seerly platform

The free tools answer whether AI crawlers can reach and parse a site. They do not answer whether assistants actually cite it. That requires running real prompts against ChatGPT, Perplexity, Google AI Overviews and others and tracking when a brand is named — which is what the Seerly platform does, across every page of a site rather than one URL at a time.