How to Allow AI Crawlers Like GPTBot in robots.txt

You published a solid page, added FAQ schema, and even wrote an llms.txt file. But when you ask ChatGPT or Perplexity a question your product should answer, your site never shows up. The most common reason has nothing to do with your content: your robots.txt file is quietly telling AI crawlers to stay out.

Robots.txt is a plain-text file at the root of your domain that tells automated crawlers which parts of your site they may fetch. Search engines have honored it for decades, and the same file now controls whether AI bots like GPTBot, ClaudeBot, and PerplexityBot can read your pages at all. If a rule blocks them, no amount of good writing, schema, or backlinks will get you cited in an AI answer, because the crawler never reaches the page in the first place.

What Is robots.txt, and Why Does It Matter for AI Search?

Robots.txt is the first file any well-behaved crawler checks before it fetches anything else on your domain. It lives at a fixed address, such as https://yoursite.com/robots.txt, and it lists rules per user-agent: which paths a given bot may or may not request.

For classic SEO, this file mostly matters for crawl budget and keeping staging pages out of Google. For AI search, it matters more directly: AI answer engines run a separate crawler from the one that indexes search results, and each one identifies itself with its own user-agent name. A rule that allows Googlebot does nothing for GPTBot. Every AI crawler needs its own explicit permission.

The convention itself is old. Webmasters agreed on a shared robots.txt syntax as far back as 1994, and it stayed an informal but widely honored agreement for almost three decades before it became an official internet standard, RFC 9309, in 2022. AI companies adopted the same syntax rather than inventing a new one, which is why fixing access for GPTBot, ClaudeBot, and the others is a matter of adding a few lines, not rebuilding anything.

Which AI Crawlers Should You Know About?

Five user-agents cover most of the AI search traffic worth caring about right now. Each belongs to a different company and a different downstream product.

Note that Google-Extended is not the same permission as the Googlebot rule most sites already have. You can allow Googlebot for search indexing while still blocking Google-Extended, and many sites do this without realizing it, which quietly opts them out of Gemini and AI Overviews.

How Do You Check If AI Crawlers Are Already Blocked?

Before you write a single new rule, find out what your site currently allows. This takes about five minutes.

1. Open yoursite.com/robots.txt directly in a browser and read every line.
2. Search the file for a User-agent: line matching each of the five bot names above.
3. For each match, check whether the block underneath says Disallow: /, which blocks the entire site.
4. Look for a catch-all rule, User-agent: * followed by Disallow: /, which blocks every bot that is not named separately, AI crawlers included.
5. Confirm with a direct request, for example curl -A "GPTBot" https://yoursite.com/, since a firewall or bot-management tool can still block a crawler even when robots.txt allows it.

That last step matters more than people expect. Robots.txt is a request, not an enforcement mechanism, and it is common to fix the file, still get blocked, and only later discover a CDN-level bot rule doing the blocking upstream.

How Do You Write an Allow Rule for Each Crawler?

Once you know which bots are blocked, the fix is a short, explicit block for each one. A minimal set that opens your entire site to all five looks like this:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

If you want AI crawlers to skip specific sections, such as an internal admin path or a paywalled area, add a narrower Disallow line inside that same bot's block rather than blocking the bot entirely. Each user-agent block is read independently, so a rule under User-agent: * will not protect a path from GPTBot unless GPTBot's own block repeats it.

Checking five user-agent blocks by hand across every path on a growing site is where most teams stop paying attention and rules drift out of date. This is exactly what AskEO's audit checks automatically: it fetches your live robots.txt, tests it against all five AI crawler user-agents alongside your meta tags, JSON-LD, and heading structure, and flags any block by name. See what the Pro plan's AI recommendations cover if you want the fix written out for you instead of writing it by hand.

What If a Plugin or CMS Generates Your robots.txt?

Many sites do not have a hand-edited robots.txt file at all. WordPress generates one automatically unless a plugin overrides it, and SEO plugins such as Yoast SEO and Rank Math each add their own robots.txt editor screen inside the WordPress admin rather than expecting you to upload a file over FTP. If your site runs WordPress, check that screen first, since a rule added there overrides the default.

Static site generators and headless setups work differently: the file has to exist as a plain text file in the public or static output folder, such as public/robots.txt in a Next.js or Hugo project, and it ships with the rest of the build. There is no admin screen to check. If you cannot find a robots.txt editor anywhere in your CMS and a direct fetch of yoursite.com/robots.txt returns a 404, that is the actual problem: no file means most crawlers fall back to treating the entire site as allowed, but some AI bots default to caution and skip a site with no file at all, so it is still worth publishing one explicitly.

How Often Should You Recheck AI Crawler Access?

A robots.txt rule does not usually break on its own, but the file gets touched more often than people expect: a new SEO plugin, a migration to a new host, or a security tool added later can silently reintroduce a blanket block. Recheck it after any change that touches hosting, your CMS, or your firewall or bot-management settings, and at minimum once a month, since a regression here costs you visibility in every AI answer engine at once, not just one page.

Does Allowing AI Crawlers Hurt Your SEO or Create a Security Risk?

No, and this is the objection that stops people from making the change. Allowing a crawler in robots.txt does not expose anything a browser could not already reach; it is a request for permission to fetch public pages, the same pages a human visitor can already open. It carries no login credentials and grants no write access.

The real trade-off is different: once GPTBot or ClaudeBot can fetch a page, that content may be used to generate an AI answer or, depending on the bot, included in model training data. If that is a genuine concern for specific pages, such as pricing tiers you update often or internal documentation, disallow those exact paths in each bot's block and leave the rest of the site open. Blocking every AI crawler outright to avoid this only trades a small, specific risk for total invisibility in AI search.

Check Your AI Crawler Access Now

You do not need to guess whether your rules are correct. Paste your URL into a free AskEO audit and it will fetch your live robots.txt, test it against GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended, and tell you by name which ones are blocked. Create a free account and run your first audit — it takes about 30 seconds and needs no credit card.