AskEO is an AI-powered SEO and AEO audit tool that checks whether a site's robots.txt file actually allows the crawlers behind ChatGPT, Claude, and Perplexity answers. Most sites get this half right: someone reads a guide, adds one blanket rule, and assumes every AI crawler is now covered. It is not that simple. OpenAI, Anthropic, and Perplexity each run more than one bot under separate user agents, and the bots do different jobs.
An AI crawler user agent is the identifying string a bot sends in its request header so a server, and a robots.txt file, can tell which company's system is asking for the page. Robots.txt rules apply per user agent, not per company, so one rule almost never covers a whole vendor. Block the wrong agent and you lose a citation channel. Allow the wrong one and you may hand over training rights you never meant to grant.
What Does OpenAI's GPTBot Actually Crawl For?
GPTBot collects web content for OpenAI's model training and follows robots.txt when a site publishes a rule for it. OpenAI's crawler documentation lists four separate agents, not one. GPTBot handles training, OAI-SearchBot crawls to surface pages inside ChatGPT's search feature, OAI-AdsBot only visits pages submitted as ads on ChatGPT, and ChatGPT-User is not scheduled crawling at all: it is a real-time fetch that fires when a person inside ChatGPT asks it to open a specific page.
The split matters in practice. OpenAI's own documentation notes that robots.txt rules for ChatGPT-User may not apply, since the request is triggered by a person, not a crawl job. Blocking GPTBot stops your content from training future models; it does not stop ChatGPT from opening a link a user pastes into a chat.
What Does Anthropic's ClaudeBot Actually Crawl For?
Anthropic runs a similar three-agent split under different names, confirmed in Anthropic's support documentation. ClaudeBot collects data that could contribute to Claude's training. Claude-SearchBot crawls to improve the relevance of Claude's search results. Claude-User is the on-demand agent that retrieves a page when someone asks Claude a direct question. All three are controlled independently in robots.txt, and ClaudeBot alone supports an optional Crawl-delay directive, so a site can slow it down instead of blocking it outright.
What Does Perplexity's PerplexityBot Actually Crawl For?
Perplexity runs two agents instead of three. Per Perplexity's crawler documentation, PerplexityBot is "designed to surface and link websites in search results" and is explicitly not used to train Perplexity's underlying models. It respects robots.txt. Perplexity-User is the second agent, and it behaves like OpenAI's and Anthropic's user-triggered fetchers: it visits a page in real time when someone asks Perplexity a question, and Perplexity states this fetcher "generally ignores robots.txt rules" because it acts on a specific person's request rather than crawling at scale.
How Do All Nine Crawlers Compare Side by Side?
| Company | User agent | Job | Honors robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Trains models | Yes |
| OpenAI | OAI-SearchBot | Indexes for ChatGPT search | Yes |
| OpenAI | OAI-AdsBot | Checks ad landing pages | Yes |
| OpenAI | ChatGPT-User | Fetches a page a user opens | Not guaranteed |
| Anthropic | ClaudeBot | Trains models | Yes, supports Crawl-delay |
| Anthropic | Claude-SearchBot | Indexes for Claude search | Yes |
| Anthropic | Claude-User | Fetches a page a user asks about | Yes |
| Perplexity | PerplexityBot | Indexes for Perplexity answers | Yes |
| Perplexity | Perplexity-User | Fetches a page a user asks about | Generally no |
How Do You Write robots.txt Rules for Each One?
- List every user agent you actually care about by name instead of relying on a single wildcard block: GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
- Decide per agent, not per company. Most sites want the three search-indexing bots, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, allowed, since those are what get a page cited in an AI answer.
- Add an explicit Disallow rule for a training bot, GPTBot or ClaudeBot, if you do not want the page used as training data. Allowing indexing does not require allowing training.
- Leave the three user-triggered agents alone unless you have a specific reason to block them, since two of the three may not honor the rule regardless.
- Re-check the published robots.txt after every edit. A typo in a user agent string silently does nothing, and there is no error message when a rule fails to match.
Checking nine user agents by hand against a live robots.txt file, one at a time, is exactly the kind of task that is easy to get right once and never revisit. AskEO's audit reads a site's robots.txt and reports which of these AI crawlers are explicitly allowed, explicitly blocked, or left ambiguous, as part of the full SEO and AEO score. See what each AskEO plan checks.
Should You Block Any of These Crawlers?
The obvious objection is: why not block the training bots and be done with it? For most content sites, that trade costs more than it protects. Blocking GPTBot and ClaudeBot only stops those two specific agents from using page content in future model training; it does nothing to the search-indexing bots or the user-triggered fetchers, so an answer engine can still cite the page. If the real goal is limiting AI citation of the page, blocking the search-indexing bots, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, does that directly. Blocking the training bots does not touch AI search visibility at all. Decide which outcome you actually want before writing the rule, and check AskEO's other AEO guides for the indexing side of this.
Check Your robots.txt Against All Nine Crawlers
Open the site's robots.txt and search it for each of the nine user agent strings above. If a name is missing, the default applies: search-indexing bots are allowed by default unless a rule blocks them, so an empty robots.txt already permits OAI-SearchBot, Claude-SearchBot, and PerplexityBot to crawl. Create a free AskEO account and run a homepage scan. It checks AI crawler access alongside the rest of the AEO score, and the report is ready in about 30 seconds.
Frequently Asked Questions
What is the difference between GPTBot and ChatGPT-User?
GPTBot is an automated crawler that collects web content for OpenAI's model training and follows robots.txt rules. ChatGPT-User is a different agent that fires in real time only when a person inside ChatGPT asks it to open a specific page, and OpenAI's documentation notes robots.txt rules may not apply to that user-triggered request.
Does blocking ClaudeBot stop Claude from citing my site?
No. Blocking ClaudeBot only stops Anthropic from using page content to train future models. Claude-SearchBot, the separate agent that indexes pages for Claude's search results, and Claude-User, the agent that fetches pages when a person asks a direct question, are controlled independently and keep working unless you block them too.
Do I need a separate robots.txt rule for every AI crawler?
Yes, if different agents should get different treatment. Robots.txt rules apply per user agent string, so a single wildcard rule treats every bot the same way. Listing GPTBot, ClaudeBot, PerplexityBot, and the other six agents by name is the only way to allow indexing while blocking training, or the reverse.
Which AI crawlers ignore robots.txt rules?
OpenAI's ChatGPT-User and Perplexity's Perplexity-User are both described by their own vendors as user-triggered fetchers rather than automated crawls, and Perplexity states directly that Perplexity-User generally ignores robots.txt. Anthropic's Claude-User, by contrast, is documented as controllable through a standard robots.txt rule.
How often should I recheck robots.txt for AI crawlers?
Recheck it whenever a vendor adds a new agent or after any change to the file itself, since a typo in a user agent string fails silently with no error message. Running a fresh AI-crawler check monthly, or right after any robots.txt edit, catches both.