Get started

By · 13 September 2026 · 1 min read · Technical

Which AI crawlers to allow in robots.txt

If AI crawlers can't fetch your pages, assistants can't cite them. Here are the user agents that matter and what each one is for.

Technical

To appear in AI answers, the crawlers behind those answers need to reach your pages. Check robots.txt for rules that block them, and check that your firewall or bot protection isn't blocking them either. Each AI company runs separate crawlers for training and for search, so you can choose which to allow.

The user agents that matter

  • GPTBot (OpenAI): collects content that may be used to train OpenAI's models.
  • OAI-SearchBot (OpenAI): finds pages to show in ChatGPT search results.
  • ChatGPT-User (OpenAI): fetches a page when a ChatGPT user's request needs it.
  • ClaudeBot (Anthropic): collects content for Anthropic's models.
  • Claude-SearchBot and Claude-User (Anthropic): search indexing and fetches on behalf of Claude users.
  • PerplexityBot (Perplexity): indexes pages for Perplexity's answers.
  • Google-Extended (Google): controls use of your content for Gemini. It doesn't affect Google Search.
  • Googlebot (Google): Search, including AI Overviews.
  • Bingbot (Microsoft): Bing, which also grounds Copilot.
  • Applebot-Extended (Apple): controls use of your content for Apple's AI features.

Search crawlers versus training crawlers

If you want to be cited but don't want your content used for training, allow the search and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) and disallow the training ones (GPTBot, ClaudeBot, Google-Extended). If you want the widest visibility, allow them all.

Blocked without knowing it

  • A blanket Disallow: / for User-agent: * left over from a staging site.
  • Bot protection or a web application firewall that challenges anything that isn't a browser.
  • A CMS or SEO plugin setting that blocks AI crawlers by default.
  • Content that only appears after JavaScript runs, which many AI crawlers never execute.

Robots.txt is a request, not a lock. Well-behaved crawlers follow it; it won't stop scrapers that ignore it.

Written by

Founder and SEO specialist

Milod is the founder of AutomatedSEO and an SEO specialist who has built and grown several products to large volumes of organic search clicks.

Want this done for you every week?

AutomatedSEO researches, writes and publishes 30 articles a month to your CMS, and interlinks every one.

Start your 3-day trial
All posts

Keep reading