Is ChatGPT a web crawler?

ChatGPT itself isn't, but OpenAI runs several: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for fetching pages when a user's request needs them.

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. GPTBot only collects training data; OAI-SearchBot decides whether you can appear in ChatGPT's search answers. Block one and allow the other if you want citations without training.

Can AI crawlers read JavaScript?

Don't rely on it. Many AI crawlers fetch the HTML without running scripts, so content that appears only after JavaScript runs may be invisible to them. Serve the main text in the initial HTML.

How long do robots.txt changes take to apply?

It varies by company. OpenAI says about 24 hours for its search systems. Others don't publish a figure, so check your server logs after a change.

Which AI crawlers to allow in robots.txt

To appear in AI answers, the crawlers behind those answers need to reach your pages. Check robots.txt for rules that block them, and check that your firewall or bot protection isn't blocking them either. Each AI company runs separate crawlers for training and for search, so you can choose which to allow.

The user agents that matter

User agent	Company	What it's for	If you block it
GPTBot	OpenAI	Collects content that may be used to train OpenAI's models	Your pages are left out of future training
OAI-SearchBot	OpenAI	Finds pages to show in ChatGPT search	You can drop out of ChatGPT's search answers
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user's request needs it	OpenAI says robots.txt rules may not apply to it
ClaudeBot	Anthropic	Collects content that may be used to train Anthropic's models	Your pages are left out of future training
Claude-SearchBot	Anthropic	Indexes pages to improve Claude's search results	You may appear less in Claude's search answers
Claude-User	Anthropic	Fetches pages when a Claude user asks	Claude can't retrieve your pages for its users
PerplexityBot	Perplexity	Indexes pages for Perplexity's answers; not used for training	You can drop out of Perplexity's answers
Perplexity-User	Perplexity	Fetches a page when a user asks	Perplexity says it generally ignores robots.txt
Google-Extended	Google	Controls Gemini training and grounding in Gemini Apps; it doesn't crawl	Gemini Apps can't ground answers in your pages. Google Search is unaffected
Googlebot	Google	Google Search, including AI Overviews and AI Mode	You leave Google Search
Bingbot	Microsoft	Bing, which also grounds Copilot	You leave Bing and Copilot's answers
Applebot-Extended	Apple	Controls whether Apple trains its models on your content; it doesn't crawl	Your pages stay in Apple's search results

From each company's own documentation, checked on 27 September 2026. These names change, so check again before copying a rule.

Search crawlers versus training crawlers

If you want to be cited but don't want your content used for training, allow the search and user agents (Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) and disallow the training ones (GPTBot, ClaudeBot, Applebot-Extended). If you want the widest visibility, allow them all.

Google-Extended is the hard call. Blocking it keeps your content out of Gemini training, but Google says it also stops Gemini Apps using your pages to ground answers. It has no effect on Google Search or its AI Overviews.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Cited but not trained: block the training crawlers, allow everyone else. Add Google-Extended to the list if you accept losing Gemini Apps grounding.

The widest-visibility version is a single group for every user agent that allows all your public pages, which is what this site ships. OpenAI says a change can take about 24 hours to reach its search systems; the others don't publish a figure.

Blocked without knowing it

A blanket Disallow: / for User-agent: * left over from a staging site.
Bot protection or a web application firewall that challenges anything that isn't a browser.
Your CDN's defaults. Cloudflare has asked new domains whether to block AI crawlers since July 2025, and from 15 September 2026 its default for new domains blocks training and agent bots on pages that show ads, while search crawlers stay allowed.
A CMS or SEO plugin setting that blocks AI crawlers by default.
Content that only appears after JavaScript runs, which many AI crawlers never execute.

Our free AI visibility checker tests your robots.txt against each major AI crawler and reads your homepage the way a crawler that doesn't run JavaScript does.

Robots.txt is a request, not a lock. The standard itself, RFC 9309, says it's no substitute for real security. Well-behaved crawlers follow it; it won't stop scrapers that ignore it.

Access is the first step of generative engine optimisation: an assistant can't cite a page its crawler never fetched. Once crawlers can get in, llms.txt is an optional extra, and measuring citations tells you whether any of it is working. Our AI visibility page covers what we check for you.

Once the door is open, crawlers need something worth fetching. A site that hasn't published in months gives an assistant nothing new to cite when your buyers ask this week's questions. AutomatedSEO keeps them fed. It publishes 30 articles a month, written answer-first in your voice with sources linked, straight to WordPress, Webflow, Shopify, Ghost, Framer or your own stack, on a steady schedule.

Now give the crawlers something worth fetching

AutomatedSEO researches, writes and publishes 30 articles a month to your connected CMS.

Start your 3-day trial See pricing

3-day trial · Cancel any time

Sources

Checked on 27 September 2026.

Written by

Mielad Niekzad

Founder and SEO enthusiast

Mielad Niekzad is the founder of AutomatedSEO and an SEO enthusiast.

Questions people ask

The questions searchers ask next, answered.

Still have a question? Email us.

Keep reading

All guides