Robots.txt: what it is, how it works and what to block

Robots.txt is the shortest file on your website and one of the few that can pull the whole site out of Google with a single line. It is a scene that repeats itself in migrations: the staging environment has Disallow: / so nobody finds it, the new site goes live with that same file and two weeks later someone asks why organic visits are falling off a cliff.

That is why I treat the robots.txt file with affection and respect. My take is that it should be decided by looking at what bots actually crawl in your logs, and that today it has to answer two different questions: what Google should not waste time crawling, and which AI systems can use your content to train or to cite you. In this guide I explain what it is, how it works, its directives with examples and what I would block on a large site.

What is robots.txt?

Robots.txt is a plain text file that lives in the root directory of a domain (for example, yoursite.com/robots.txt) and tells crawlers which URLs they can visit and which they cannot. It is also public: anyone can type that address into a browser and read it, your competitors included (quite entertaining, by the way).

The best analogy I know for the robots.txt file is the "staff only" sign on a storeroom door. It has no lock. Polite people respect it and whoever came to steal walks in anyway.

The format comes from the Robots Exclusion Protocol, proposed by Martijn Koster in 1994 and only formalized as a standard by the IETF in 2022 (RFC 9309). Thirty years later, the robots.txt file is still a text with a handful of rules.

A detail that often gets forgotten: each host on your website has its own file. The robots.txt file at www.yoursite.com does not apply to blog.yoursite.com, so every subdomain needs its own in its own root directory.

What is the robots.txt file for?

The robots.txt file is mainly used to manage crawling. According to Google's documentation, its main use is to keep crawlers from overloading your server with requests. In practice, search engines use it to know which parts of the website they can go through.

The most common uses:

• Keeping bots out of areas with no search value: admin panels, carts, login pages.
• Protecting crawl budget on websites with hundreds of thousands of pages, so the search engine does not spend its time on filters and parameters.
• Pointing to where the sitemap is.
• Deciding which artificial intelligence bots can come in and for what.

Now, an honest clarification: if your website has 300 pages and Google crawls new ones without delay, robots.txt will almost never be your bottleneck. I would not spend three months on it (we discuss this more calmly in the article on crawl budget). Where it really matters is ecommerce, portals and media sites with millions of URL combinations.

How does robots.txt work?

Before crawling a website, a well-behaved crawler requests the robots.txt file, reads it and looks for the group of rules that applies to its name. If nothing blocks it, it crawls.

The whole robots.txt system is voluntary. Search engines like Google and Bing respect it, as do the automated bots that the big AI companies declare. Malicious bots ignore it, and some even read it to see what you are hiding.

When two rules in the robots.txt file clash, Google applies the most specific one, measured by the length of the path. If one rule blocks /store/ and another allows /store/deals/, the second one wins for that folder. If they tie, it picks the least restrictive.

Here comes the dry part, but it is short and it has saved more than one team's job. According to Google's specification:

• The robots.txt file is cached for up to 24 hours, so a change does not apply instantly.
• Only the first 500 KiB are read; anything after that is ignored.
• If the file returns a 4xx error (except 429), the search engine assumes there are no restrictions.
• If it returns a 5xx error, it stops crawling the site for 12 hours and then uses the last saved version for up to 30 days.

In other words, a firewall that returns a 403 when robots.txt is requested protects nothing, and a server that is down can stall crawling of the entire website.

Which directives does robots.txt use?

There are few robots.txt directives. Google supports four, and most search engines read them the same way:

• User-agent: which bot the rules below apply to. User-agent: * means all of them (with a curious exception: AdsBot, the ads crawler, has to be named separately).
• Disallow: the path that bot should not crawl. It always starts with /.
• Allow: an exception inside a blocked path.
• Sitemap: the full URL of the XML file listing your pages. You can add several lines if you have more than one (and if it looks like nobody reads it, check why Google is not using your sitemap).

A fifth one you will see often is Crawl-delay, which asks for a few seconds between requests. Bing respects crawl-delay; Google does not support it. And watch out for advice that still circulates: in January 2024 the Search Console tool to limit crawl rate was retired, so that knob no longer exists.

Another robots.txt detail that causes headaches: paths are case sensitive. Disallow: /Deals/ does not block /deals/.

How do the * and $ wildcards work?

There are two wildcards. The asterisk * means "any sequence of characters" and the $ sign marks the end of the URL. With a well-placed wildcard you can block whole patterns instead of writing path after path:

User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$

The first rule blocks any address with the sort parameter. The second, every file ending in .pdf.

Robots.txt examples

Let's start with the most boring one, which is this website's. I checked it today and it has three lines:

User-agent: *
Allow: /

Sitemap: https://www.anafernandezseo.com/sitemap.xml

For a small site, that robots.txt file is perfect. There are no filters, no internal search, nothing to hide.

The one that blocks everything, useful only in a staging environment (and dangerous if it travels to production, as in the opening scene):

User-agent: *
Disallow: /

One that closes a directory to a specific bot and leaves the rest of the website open (you already saw the pattern for blocking PDFs above):

User-agent: Bingbot
Disallow: /drafts/

And a more realistic one for a large ecommerce website:

User-agent: *
Disallow: /search
Disallow: /*?color=*&size=
Allow: /search/help

Sitemap: https://www.yourstore.com/sitemap-index.xml

What should a large site block?

Guides usually give the same list: admin area, internal search, carts and confirmation pages, filters and sort parameters. As a starting point it is fine. My problem is that the list gets copied from one site to another without looking at the data.

Think of a black T-shirt in size M. With color, size, brand and price filters, that same T-shirt can end up with 80 different URLs. The crawler can spend the day going through "black-tshirt-size-m-sorted-by-price" while the new products, the ones that actually sell, wait their turn. It also creates duplicate content at scale.

That is why, when we audit an enterprise site, we start with the server logs and not with a robots.txt template:

1. We take 30 days of logs and filter the real Googlebot visits.
2. We group URLs by template type: product pages, categories, filters, parameters, pagination and internal search.
3. We calculate what share of crawling goes to each group.

An illustrative example: a site with 1,000,000 crawl hits a month, where 38% ends up in filters and parameters nobody wants indexed. That number changes the conversation, because the problem becomes what you are letting bots crawl.

Before blocking, a warning: some filters have searches of their own ("women's running shoes", for example) and those URLs deserve to be indexable pages. Blocking every parameter in one go can leave you without the landing pages that bring in sales. First you separate the filters with demand from the ones that only create duplicate content, and then you write the rules in the robots.txt file.

Can robots.txt keep a page out of Google?

No, and it is probably the most expensive misunderstanding on the topic. Google says so in its own documentation: robots.txt is not a mechanism for keeping a page out of its search results. If a blocked URL gets links from other websites, the search engine can still show it, just without a description.

To keep a page out you have three options:

• The meta robots tag with noindex in the page's <head>.
• The X-Robots-Tag HTTP header, useful for PDFs and other non-HTML files.
• Password protection, which is the only thing that truly keeps it out of everyone's reach.

And here is the trap: if you block a page in robots.txt and also add noindex, Google will never see that noindex, because it cannot crawl the page to read it. The right order is to let the bot in, see the tag and drop the page from the index; only then, if you want to save crawling, do you block it. If you want to go deeper into indexing statuses, how to index a page on Google covers each one.

How is robots.txt used with AI bots?

Until recently, configuring crawlers was almost purely an SEO decision: who we let in so they can find our pages. Now there are bots that look for content to cite it, bots that use it to train models and agents that visit it on a user's behalf. And each company separates them in its own way.

Some clarifications with sources. Google-Extended is a token: it controls whether your content is used to train Gemini, but it has no crawler of its own, and Google's documentation does not mention AI Overviews or AI Mode, so I would not assume blocking it removes you from them.

GPTBot is OpenAI's training bot, and according to its crawler documentation each setting is independent: you can block GPTBot and let OAI-SearchBot through to keep appearing in ChatGPT search. OpenAI also clarifies that, because ChatGPT-User responds to a user's question, robots.txt rules may not apply.

My take is that there are two separate business decisions here. One is whether you want your content to train models. The other is whether you want to be found and cited when someone asks an AI about your category. Blocking everything with "AI" in the name mixes the two (and it usually costs you on the second one).

Also check what happens outside robots.txt. Since September 15, 2026, Cloudflare blocks training and agent bots by default on ad-supported pages of new domains. And in its announcement it explains that multi-purpose crawlers like Googlebot, Bingbot and Applebot are blocked for customers who chose to block training and did not adjust that option. If someone on your team ticked that box, it is worth reviewing today.

What about llms.txt? llms.txt is a proposal to describe a site's important pages to language models, closer to an annotated sitemap than to a permissions file. No bot is required to read it, and a study by SE Ranking did not find that having one is associated with more citations in AI search. Permissions still live in robots.txt.

How do you create and test your robots.txt?

To create the robots.txt file:

1. Create a text file named robots.txt, in lowercase and UTF-8 encoded (a word processor can add characters that break it).
2. Write one group per bot, with its User-agent and its rules below.
3. Upload it to the host's root directory. If you have a subdomain, repeat the process there.
4. If you use a CMS like WordPress or Shopify, the file usually comes pre-generated and is edited from the dashboard or with a plugin.

To test it:

1. Open the robots.txt report in Search Console (under Settings). It shows the files found for your top 20 hosts, when they were read and whether they have errors. From there you can request a recrawl of the file in an emergency. The old tester no longer exists, even though many guides still mention it.

2. Use the inspection tool with your 10 best-selling pages, before and after publishing the change.

3. Check the "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" statuses in the pages report.

In a company, I would add one step: the robots.txt file should live in version control, with someone from SEO reviewing every change, because a single line can affect the whole site.

Common robots.txt mistakes

The ones I see most in audits:

• The Disallow: / inherited from staging. The classic of every migration (we also covered it in the article on vibe coding).
• Blocking CSS and JavaScript. Google needs those files to render the page; if you block them, it understands less of what is on it. Make sure no CSS file is left out.
• Using it to hide secrets. The robots.txt file is public, so listing /secret-admin/ tells everyone where to look.
• Blocking and adding noindex at the same time. They cancel each other out, as we saw above.
• Mixing up case. /Products/ and /products/ are different paths.
• Putting it in a subfolder. Outside the root directory, no search engine bot will find it.

What I would check this week

If you have half an hour, open your website's robots.txt (yoursite.com/robots.txt) and read it from top to bottom. Look for a stray Disallow: /, rules that touch CSS or pages that sell, and if there are AI bot blocks, ask which of the two decisions was made (training or citing). Then look at the bot settings in your CDN, because who gets in is decided there too. It is a free check, and finding the problem through a traffic drop is a lot less fun.