Technical13 September 2026 · 4 min read
Which AI crawlers to allow in robots.txt
Free tool
If AI assistants can't reach your site, they recommend your competitors instead. Get a 0 to 100 score in seconds: we test your robots.txt and your firewall against every major AI crawler, look for llms.txt and a sitemap, and read your homepage the way a crawler that doesn't run JavaScript does.
| Group | Weight | What it covers |
|---|---|---|
| Crawler access | 35% | Homepage responds, robots.txt rules, firewall test, noindex, content signals |
| Content | 25% | Server-rendered text, one H1, sections under H2s |
| Understanding | 20% | Title, description, language, structured data, sameAs links |
| Discovery | 20% | Sitemap, freshness, canonical URL, llms.txt |
If crawlers can't get in at all, nothing else matters, so a failed access check caps the score at 40. 85 and above is strong, 65 to 84 is good, 45 to 64 needs work and anything lower means assistants are struggling to see you.
Each assistant relies on its own crawlers, so a site can be open to one and closed to another. The report shows ChatGPT, Claude, Perplexity, Gemini, Google's AI Overviews and Copilot side by side, with the crawlers each one needs.
| Kind | Examples | If you block it |
|---|---|---|
| Search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | That assistant can drop you from its answers and stop citing you |
| Fetch on request | ChatGPT-User, Claude-User, Perplexity-User | The assistant can't open your page when someone asks about it |
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended | Your pages stay out of future training. Citations are unaffected, except that Google-Extended also controls grounding in Gemini Apps |
Blocking only training crawlers is a reasonable choice, and the checker scores it as a pass. The full list, with what each company says about its own crawlers, is in which AI crawlers to allow in robots.txt.
robots.txt is only half the story. Bot protection, web application firewalls and CDN settings can turn AI crawlers away before robots.txt is ever read. Cloudflare has asked new domains whether to block AI crawlers since July 2025, and from 15 September 2026 its default for new domains blocks training and agent bots on pages that show ads.
So we ask for your homepage a second time under the names of OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot and ClaudeBot. If a normal request gets your page and those get a 403 or a challenge, something on your server is refusing them. Some firewalls only refuse a crawler's name when it comes from an address the crawler doesn't use, so check your logs before changing anything.
Google renders JavaScript, but many AI crawlers fetch the raw HTML and move on. If your content only appears after scripts run, those crawlers see an almost empty page. The word count check shows you what they get.
Two kinds of tool call themselves AI visibility checkers. This one checks readiness: whether assistants can reach and understand your site. Others ask ChatGPT or Gemini a handful of questions and report whether your brand came up. Both are useful, but they answer different questions.
| Question | Readiness checker (this tool) | Citation tracker |
|---|---|---|
| Can assistants reach my pages? | Yes, crawler by crawler | No |
| Can they read my content without JavaScript? | Yes | No |
| Do they mention or cite me? | No | Yes, for the prompts it asks |
| How reliable is one run? | Deterministic: same site, same result | Answers vary run to run, so it needs repeated prompts over weeks |
A single answer from ChatGPT is too random to mean much. Measuring real citations takes the same prompts asked repeatedly over time, which is what AutomatedSEO's citation tracking does. See how to measure citations across assistants.
Check how accessible your site is to AI crawlers
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
3-day trial · Cancel any time
Still have a question? Email us.
From check to done
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
llms.txt generator
Generate an llms.txt file from your sitemap in seconds, or validate the one you have. We read your pages' titles and descriptions, group them into sections, and check the file against the llms.txt format.
Read moreSchema checker
Check the structured data on any page, or paste your own code. Find missing required properties, rich results you're eligible for, invalid dates, relative URLs, broken JSON and weak author markup.
Read moreGoogle rank checker
Check where any website ranks in Google for a keyword, in 31 countries. See the exact position in the top 50 organic results, the page that ranks, and the top 10 you're up against.
Read moreBacklink checker
Check any website's backlinks for free: referring domains, total links, dofollow share, an authority score and the strongest links pointing at it, with their anchor text.
Read more