Robots.txt for AI: controlling which crawlers access your site

Your robots.txt file is a text file that sits in your website's root directory. It tells crawlers which pages they can access and which ones are off-limits. For traditional search engines, robots.txt is optional. For AI crawlers, it is the primary tool you use to control whether your content gets indexed or blocked.

The difference matters. Google respects robots.txt but also crawls pages you block if other sites link to them. AI crawlers respect robots.txt more strictly. Block GPTBot in your robots.txt and ChatGPT will not cite your content. Allow it and your content becomes available for AI citations. This chapter covers what each major AI crawler is called, how to block specific bots, and what strategy makes sense for your brand.

How robots.txt works for AI crawlers

When a crawler visits your site, the very first thing it does is check for a robots.txt file. It reads the file line by line looking for rules that apply to it. If your robots.txt says "disallow: /", no crawler can access any page. If it says "disallow: /admin/", crawlers can access everything except the admin folder.

AI crawlers follow these rules more consistently than Googlebot does. If you block ClaudeBot in robots.txt, Claude will not index your content. Period. Googlebot might still crawl pages you block if the content is linked from other sites, but AI crawlers stop at the robots.txt directive.

The format is simple. You write rules that specify which user agents (crawler names) apply to which rules, then list what those crawlers can and cannot access. The file is plain text. No markup. No special formatting. Just rules.

The major AI crawlers and their user agent names

Every crawler has a user agent name. When it visits your site, it identifies itself by that name. You use that name in your robots.txt file to create rules specific to that crawler.

Here are the major AI crawlers you should know about:

GPTBot (OpenAI)

GPTBot is OpenAI's crawler. It indexes content for ChatGPT. When a user enables web search in ChatGPT, the system pulls from pages GPTBot has indexed. GPTBot crawls aggressively but respects robots.txt rules.

To allow GPTBot, do nothing. It crawls by default. To block it, add these lines to your robots.txt:

User-agent: GPTBot
Disallow: /

That rule says "GPTBot, you cannot access any page on this site." If you only want to block specific sections, you can be more specific. Block the /admin/ folder but allow the rest:

User-agent: GPTBot
Disallow: /admin/

ClaudeBot (Anthropic)

ClaudeBot is Anthropic's crawler. It indexes content for Claude's web search. ClaudeBot is more aggressive than GPTBot. It makes more requests and crawls more frequently. If you allow ClaudeBot, monitor your server resources.

ClaudeBot identifies itself with this user agent name. Use it in your robots.txt the same way you would for GPTBot:

User-agent: ClaudeBot
Disallow: /

PerplexityBot (Perplexity)

PerplexityBot crawls the web for Perplexity. It is one of the more frequent crawlers. It updates content multiple times per day because Perplexity prioritizes freshness. If you block PerplexityBot, your content will not appear in Perplexity answers.

User-agent: PerplexityBot
Disallow: /

Googlebot-Extended (Google AI)

Google has two crawlers. Googlebot crawls for traditional search. Googlebot-Extended crawls for Google's AI systems like AI Overviews and Gemini. You can block Googlebot-Extended while allowing Googlebot.

User-agent: Googlebot-Extended
Disallow: /

Other crawlers

Yandex (YandexBot), Microsoft Bing (Bingbot), Naver (Naver), and other search engines also have crawlers. The principle is the same. Identify the user agent name and create a rule.

The strategy: allow or block?

Most brands should allow AI crawlers to access their content. AI citations drive high-quality traffic. Users who find your content through ChatGPT or Perplexity have already decided your answer is valuable. They click through and convert at higher rates than traditional search visitors.

There are exceptions. If you have proprietary information, trade secrets, or content you want to keep exclusive, blocking AI crawlers makes sense. If you have paywalled content, you might block certain crawlers to prevent free access through AI summaries. If you have content that is embarrassing or outdated, you might block AI crawlers until you update it.

But for most content, the calculus is simple. AI visibility creates more opportunity than risk. Allow the crawlers. Get cited. Drive traffic.

Blocking specific crawlers while allowing others

You do not have to choose all or nothing. You can allow some AI crawlers and block others.

Allow Perplexity and GPTBot but block ClaudeBot because ClaudeBot's crawl frequency strains your server:

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow:

User-agent: PerplexityBot
Disallow:

The blank "Disallow:" means no restriction. That crawler can access everything.

You can also block AI crawlers from specific sections. Allow GPTBot everywhere except your admin panel:

User-agent: GPTBot
Disallow: /admin/

Allow it to crawl your blog but not your products:

User-agent: GPTBot
Disallow: /products/

The wildcard rule and default behavior

You can create rules for specific crawlers and a default rule for everyone else. The asterisk (*) is the wildcard that applies to all crawlers not specifically mentioned.

This rule blocks all crawlers from /private/ but allows everything else:

User-agent: *
Disallow: /private/

This rule blocks all crawlers from accessing anything:

User-agent: *
Disallow: /

The crawl-delay directive

You can tell crawlers to slow down. The Crawl-delay directive tells a crawler to wait a certain number of seconds between requests. If your server is struggling with ClaudeBot's aggressive crawling, you can tell it to slow down:

User-agent: ClaudeBot
Crawl-delay: 5

That means ClaudeBot must wait 5 seconds between each request. It slows down the crawl but reduces server load.

How robots.txt relates to your sitemap

Your robots.txt file can reference your sitemap. Add this line at the end of your robots.txt file:

Sitemap: https://yoursite.com/sitemap.xml

Crawlers check your robots.txt and see the sitemap reference. They know where to find your content map. This speeds up discovery. It does not change what crawlers can access. It just tells them where to look.

Testing your robots.txt

You can test your robots.txt file before deploying it. Google Search Console includes a robots.txt tester. You enter a URL and a user agent name and it tells you whether that crawler can access that page according to your robots.txt rules.

Most content platforms (WordPress, WEMASY, Webflow) generate a robots.txt file automatically. Check your site settings. If you can edit it, you can add custom rules. If you cannot, contact your hosting provider.

Frequently asked questions