What is robots.txt and how does it control search engine crawling?

Search engines crawl your site constantly, following links and indexing pages. But not every page deserves to be indexed. You have private pages, admin areas, test pages, and duplicate pages that waste crawl budget. Robots.txt is a simple text file that tells search engines which pages they can and cannot crawl. It is the gatekeeper between your site and search engine bots.

Robots.txt is a plain text file that sits in your website's root directory (yoursite.com/robots.txt). It contains instructions for search engine crawlers on which pages and sections of your site they are allowed to crawl. You do not need a login to access robots.txt. It is public. Anyone can visit yoursite.com/robots.txt and see your instructions.

Search engines are supposed to follow robots.txt rules. Most do. Some do not. Well-behaved crawlers (Google, Bing, reputable search engines) respect robots.txt. Bad actors and malicious bots often ignore it.

What does robots.txt look like?

A simple robots.txt file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /blog/
Sitemap: https://www.example.com/sitemap.xml

This tells all bots (User-agent: *) that they cannot crawl /admin/ or /private/ sections. It explicitly allows /blog/ (which is useful if a parent folder is blocked). It also points bots to your sitemap.

Here is another example with more specific rules:

User-agent: Googlebot
Disallow: /temp/

User-agent: *
Disallow: /test/
Disallow: /*.pdf
Crawl-delay: 5

This blocks only Google's crawler from /temp/. All other bots are blocked from /test/ and all PDF files. The crawl-delay tells bots to wait 5 seconds between requests.

How robots.txt helps SEO

Robots.txt helps SEO in one main way: it preserves crawl budget. Search engines allocate a limited amount of crawl budget to each site. For small sites, crawl budget is rarely a problem. For large sites with thousands of pages, crawl budget matters.

If your site has 10,000 pages but only 500 are important, you want bots crawling the 500 important pages multiple times rather than spreading their crawl budget across all 10,000 pages. Robots.txt blocks bots from wasting time on unimportant pages.

Common pages to block: admin areas, login pages, thank you pages, test pages, duplicate content, session pages, and temporary content. Blocking these pages tells bots to focus on content that matters.

What not to block in robots.txt

Do not block pages you want to rank. Do not block your blog section or main content areas. Do not block CSS, JavaScript, or image files needed to render pages properly (unless you specifically want to prevent indexing).

Google recommends not blocking CSS, JavaScript, and image files even if you do not want them indexed. Blocking these resources can prevent Google from properly rendering and understanding your pages.

Do not use robots.txt to hide sensitive information. Robots.txt is public. Anyone can read yoursite.com/robots.txt. If you want to truly hide something, use password protection or server authentication, not robots.txt.

How to create a robots.txt file

Creating robots.txt is simple. It is a plain text file. Most website builders and CMS platforms generate one automatically.

If your CMS did not create one, create it manually. Open a text editor. Write your rules. Save it as "robots.txt" (not "robots.txt.txt"). Upload it to your site root directory (yoursite.com/robots.txt).

Test your robots.txt file. Go to Google Search Console, select your property, go to the Robots.txt Tester tool, and paste in your rules. Google will show you what the file blocks and allows. This confirms your rules work as intended.

Common robots.txt rules and examples

Here is the most common rule: block all bots from a specific directory.

User-agent: *
Disallow: /admin/

Here is how to block a specific file type (like PDFs):

User-agent: *
Disallow: /*.pdf

Here is how to block all bots except Google:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Here is how to add a crawl delay (wait X seconds between requests):

User-agent: *
Crawl-delay: 10

Here is how to point bots to your sitemap:

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap2.xml

Important robots.txt limitations

Robots.txt is not a security tool. It is a courtesy system. Bad bots, hackers, and malicious crawlers ignore robots.txt completely. Do not use it to hide sensitive data or protect important pages. Use password protection for those.

Robots.txt does not guarantee exclusion from search results. Google sometimes indexes pages listed in robots.txt if they are linked from other indexed pages. The block prevents crawling, but not indexing if the page is linked elsewhere.

Robots.txt is not a noindex. If you want to prevent indexing completely, use a robots meta tag or X-Robots-Tag header, not robots.txt. Robots.txt only controls crawling, not indexing.

Frequently asked questions