Technical13 September 2026 · 4 min read
Which AI crawlers to allow in robots.txt
Free tool
Make sure Google can find every page you publish. Enter your site and we find its sitemaps through robots.txt, follow every sitemap index, and check each file against the protocol, from broken XML to future lastmod dates. Or paste XML to check it before it ships.
A sitemap lists the URLs you want search engines to know about, with the date each last changed. It helps new and deep pages get discovered, and an accurate lastmod helps Google decide what to recrawl. It doesn't make pages rank, and Google may still skip URLs it considers low value or duplicates.
| Problem | Fix |
|---|---|
| Not declared in robots.txt | Add a line: Sitemap: https://example.com/sitemap.xml |
| Unescaped ampersand | Write & as & inside <loc>, usually in URLs with query strings |
| http:// URLs on an https site | List the https:// version your pages redirect to |
| Every lastmod the same | Output the date each page last changed, not the build time |
| Redirecting or 404 URLs | List only live, canonical URLs that return a 200 |
| Nested sitemap indexes | List every child sitemap directly in one top-level index |
Once your sitemap is clean, check that AI assistants can reach the same pages with the AI visibility checker, or turn the sitemap into an llms.txt with the llms.txt generator.
Check how accessible your site is to AI crawlers
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
3-day trial · Cancel any time
Still have a question? Email us.
From check to done
Check your site's robots rules, homepage signals and llms.txt. These checks do not measure citations in AI answers.
Technical SEO audit
Audit any page for free: indexability, robots.txt, canonical, title and meta description, headings, structured data, Open Graph, hreflang, mixed content and response time, plus Core Web Vitals from Google.
Read moreCMS detector
Find out what any website is built with: CMS, website builder, ecommerce platform, headless CMS, framework, hosting and analytics, with the evidence for each detection.
Read moreAI visibility checker
Score how ready your site is for ChatGPT, Claude, Perplexity, Gemini and AI Overviews: AI crawler access in robots.txt and at your firewall, llms.txt, sitemap, schema and server-rendered content.
Read morellms.txt generator
Generate an llms.txt file from your sitemap in seconds, or validate the one you have. We read your pages' titles and descriptions, group them into sections, and check the file against the llms.txt format.
Read more