Products Solutions Templates Pricing Resources

How robots.txt and meta tags control your ChatGPT visibility (and what strategy to choose)

Home / Everything About / Everything About GEO / How robots.txt and meta tags control your ChatGPT visibility (and what strategy to choose)

Controlling whether ChatGPT can find and cite your content involves three separate mechanisms: robots.txt, meta tags, and HTTP headers. Most people know about one. Understanding all three lets you make a deliberate choice about your visibility strategy.

This is not just about blocking or allowing. It is about being intentional. Different directives achieve different outcomes. The wrong combination can backfire.

The Three Control Layers

You have three ways to communicate with ChatGPT's crawlers about your content.

Layer 1: robots.txt

This file in your domain's root tells crawlers what they can and cannot access on your entire site or specific sections. It controls discovery and crawling.

Layer 2: Meta robots tags

These HTML meta tags on individual pages tell crawlers what to do after they have found and read the page. They control indexing and usage of that specific page.

Layer 3: X-Robots-Tag HTTP headers

This header in your server's HTTP response serves the same purpose as meta robots tags but can be applied site-wide or to specific content types. This is useful for non-HTML content like PDFs.

Most people focus on robots.txt and forget the other two. All three work together.

Layer 1: Understanding robots.txt and OAI-SearchBot

Your robots.txt file communicates with OAI-SearchBot (ChatGPT's search crawler). ChatGPT respects standard robots.txt directives.

To allow OAI-SearchBot to access your content

User-agent: OAI-SearchBot
Allow: /

This is the default. OAI-SearchBot will crawl your entire site and your content becomes visible in ChatGPT search.

To block OAI-SearchBot entirely

User-agent: OAI-SearchBot
Disallow: /

This prevents ChatGPT search from finding your content. Your pages will not appear in ChatGPT responses.

To allow OAI-SearchBot but control crawl rate

User-agent: OAI-SearchBot
Allow: /
Crawl-delay: 1

The Crawl-delay directive tells the bot to wait 1 second between requests. This helps manage server load if ChatGPT's crawler is hitting your server too frequently.

To block specific sections

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /private/
Allow: /

This blocks crawling of your /admin/ and /private/ directories while allowing access to everything else.

The key principle: robots.txt controls discovery and crawling. If you block a crawler here, it cannot access the page at all, and therefore cannot read any meta tags on that page.

Layer 2: Meta Robots Tags — Control After Discovery

After OAI-SearchBot has crawled your page (meaning it was not blocked in robots.txt), meta robots tags tell it what to do with the content.

The meta robots tag structure

Key directives for AI crawlers

noindex — Do not index this page or include it in summaries. The crawler sees the page but does not use it.

nofollow — Do not follow links from this page. The crawler can see the page but will not crawl links within it.

noai — Do not use this content for training AI models. The page can appear in search results, but OpenAI cannot use it to train language models.

noimageai — Do not use images on this page for image generation model training.

Example

This tells all crawlers (Google, Bing, ChatGPT, etc.) not to index this page. ChatGPT will not cite it, Google will not show it, and Bing will not include it.

Specific to OAI-SearchBot

This tells OAI-SearchBot specifically not to use the content for training. The page can still be cited in ChatGPT search results, but OpenAI cannot use it for model training.

Critical Rule: Do Not Combine robots.txt and noindex

This is the most important technical rule. Never do this:

User-agent: OAI-SearchBot
Disallow: /article/

And also put this on the article page:

Why: If you block OAI-SearchBot via robots.txt from accessing /article/, the crawler never fetches that page. It never reads the meta tag. The noindex directive has no effect because the crawler never sees it. Furthermore, if your page is linked to from external sites, some crawlers might index it anyway because they discovered it through links, even though your robots.txt said no.

The rule: Use either robots.txt OR meta tags, not both for the same directive.

Best practice: Use robots.txt to control site-wide crawling (allow/disallow entire sections). Use meta tags for page-level control (individual pages that have special rules).

Layer 3: X-Robots-Tag HTTP Headers

If you want to control crawler behavior for non-HTML content (PDFs, images, etc.), or apply rules across multiple pages without editing each page, use the X-Robots-Tag HTTP header.

Example:

X-Robots-Tag: noindex, nofollow

This is applied at the server level and tells all crawlers not to index and not to follow links on any page served with this header.

You can apply this selectively:

X-Robots-Tag: OAI-SearchBot: noai

This tells only OAI-SearchBot not to use your content for training, while allowing it in search.

Strategic Choices: What Do You Actually Want

Your choice depends on your goal.

Goal 1: Maximize ChatGPT citations

Allow robots.txt, no noindex meta tags. Let OAI-SearchBot find and cite you.

User-agent: OAI-SearchBot
Allow: /

No restrictive meta tags.

Goal 2: Appear in ChatGPT search but prevent model training

Allow robots.txt, use noai meta tag.

User-agent: OAI-SearchBot
Allow: /

Goal 3: Hide from ChatGPT but allow other search engines

Block OAI-SearchBot, allow others.

User-agent: OAI-SearchBot
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Goal 4: Hide from all AI crawlers

Block multiple bots.

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Allow: /

The Compliance Problem: Not All AI Crawlers Respect These Rules

Here is the uncomfortable truth: Different AI crawlers have different compliance levels.

Testing shows ChatGPT respects robots.txt and some meta tags. Perplexity respects both consistently. Other AI systems have varying compliance. Some older AI scrapers do not respect robots.txt at all.

You can control OpenAI's crawlers effectively. You cannot fully control every AI system that might scrape your content.

How WEMASY Helps You Manage Crawler Access

WEMASY's website builder includes a robots.txt editor where you can configure OAI-SearchBot permissions without editing raw files. You can add meta robots tags to individual pages through the SEO settings panel. The builder pre-configures best-practice settings — allowing OAI-SearchBot for search visibility while letting you add noai if you do not want model training. Monitor crawl logs to see which crawlers are accessing your site and adjust permissions accordingly. See WEMASY's crawler management tools.