Technical13 September 2026 · 4 min read
Which AI crawlers to allow in robots.txt
Free tool
Small technical misses quietly cost rankings. Run up to 25 checks on any page, from whether Google is allowed to index it to how fast your server answers, then get Core Web Vitals measured by Google. Every problem comes with the evidence and the fix.
| Group | Checks |
|---|---|
| Crawling & indexing | HTTP status, redirects, noindex in meta tags and headers, nofollow, robots.txt access for Googlebot, canonical tag, sitemap declaration |
| On-page | Title length, meta description, one H1, heading order, words in the HTML, image alt text, page language |
| Structured data & social | Valid JSON-LD, Open Graph title, description and image, X card, favicon |
| Technical | Mobile viewport, character encoding, hreflang codes and self-reference, mixed content, HSTS, server response time, HTML size |
| Core Web Vitals | LCP, INP and CLS from Chrome users where Google has enough traffic, plus a Lighthouse mobile lab test |
Each check is critical, important or minor, worth 10, 5 or 2 points. A pass earns the full points, a warning half, a failure none, and the score is the share of points earned. A page that fails a critical check, such as noindex or a robots.txt block, is capped below 50, however tidy the rest is, because Google can't rank a page it can't index.
No single title tag decides a ranking. But technical problems compound: a slow server delays crawling, a missing canonical splits signals across duplicates, and thin server-rendered HTML hides content from crawlers that don't run JavaScript, including most AI crawlers. Fixing the basics lets the rest of your work count.
Core Web Vitals are Google's measures of loading (Largest Contentful Paint), responsiveness (Interaction to Next Paint) and visual stability (Cumulative Layout Shift). The audit asks Google PageSpeed Insights for real-user data from the Chrome UX Report. If the page doesn't have enough traffic, it falls back to data for the whole site, and always adds a lab test on a simulated phone.
Check how accessible your site is to AI crawlers
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
3-day trial · Cancel any time
Still have a question? Email us.
From check to done
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
Sitemap validator
Validate an XML sitemap or sitemap index: XML syntax, URL counts, invalid and duplicate URLs, lastmod dates, redirects and whether robots.txt declares it. Paste XML or enter any site.
Read moreCMS detector
Find out what any website is built with: CMS, website builder, ecommerce platform, headless CMS, framework, hosting and analytics, with the evidence for each detection.
Read moreAI visibility checker
Score how ready your site is for ChatGPT, Claude, Perplexity, Gemini and AI Overviews: AI crawler access in robots.txt and at your firewall, llms.txt, sitemap, schema and server-rendered content.
Read morellms.txt generator
Generate an llms.txt file from your sitemap in seconds, or validate the one you have. We read your pages' titles and descriptions, group them into sections, and check the file against the llms.txt format.
Read more