Robots.txt Generator

What This Does, And Who It Is For

The generator builds a robots.txt file: user-agent groups, disallow and allow rules, and the sitemap line. The file goes at the root of your domain, at https://yoursite.com/robots.txt, and nowhere else. A robots.txt in a subdirectory is a text file that nothing reads.

It is for anyone who needs to write or repair this file once and then leave it alone: a site owner cleaning up after a plugin, a developer setting up a new deploy, or someone who has read that they should be blocking AI crawlers and wants to know what that actually does before doing it.

The generator is the easy part. The rest of this page is the part worth reading, because almost every robots.txt problem falls into one of two categories: a rule that looks correct and has no effect, or a rule that works exactly as written and blocks something you needed.

The One Thing To Get Straight First

robots.txt controls crawling. It does not control indexing. These are different operations and conflating them causes most of the damage done with this file.

Google's own introduction to robots.txt, last updated 10 December 2025, opens by saying so: the file "is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." Further down it is more specific about what happens when you try:

A page that's disallowed in robots.txt can still be indexed if linked to from other sites. While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.

And the result you actually see: "If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description."

So a disallowed page can appear in Google as a bare URL with no snippet, indexed on the strength of other people's anchor text. That is a worse outcome than either leaving it crawlable or properly excluding it.

There is a trap inside the trap, and it is the single most expensive robots.txt mistake there is. The correct way to keep a page out of the index is noindex, delivered as a meta tag or an HTTP header. But noindex is only visible to a crawler that fetches the page. Disallow the URL and the crawler never fetches it, never sees the noindex, and the page can sit in the index indefinitely. The two rules cancel each other out. Google flags this pattern directly, cautioning that "combining multiple crawling and indexing rules might cause some rules to counteract other rules."

The order that works: allow crawling, serve noindex, wait for it to drop out, and only then disallow if you want to save the crawl requests. Or skip robots.txt entirely and password-protect it, which is the only option that actually keeps content private, since compliance with this file is voluntary.

Worked Example One: Scoring The Google Crawler Tokens

The old version of this page suggested blocking AI crawlers with a three-token group and left it there. Here is what those tokens do when you check each against Google's list of common crawlers, last updated 14 July 2026. Seven Google tokens are documented. They do not do what their names suggest.

Two of the seven are self-inflicted wounds. Disallow Google-InspectionTool and you break URL Inspection on your own property, which is the tool you would reach for to work out why the page is not indexed. Disallow Google-CloudVertexBot and you block the crawl you asked for when pointing a Vertex agent at your own content. Both appear in copied-and-pasted blocklists precisely because they have Google in the name.

Then there is Google-Extended, which deserves its own paragraph because it is the most misunderstood token in circulation.

Google-Extended is not a crawler. The documentation is explicit: it "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." What it controls is whether content Google has already crawled may be used for training future Gemini models, and for grounding, which the same page defines as "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy."

And then the sentence almost nobody adding this token has read: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Which means, plainly: disallowing Google-Extended does not take you out of AI Overviews or any other AI feature inside Google Search. Those are generated from the ordinary Search index, reached by Googlebot. If opting out of AI answers in Search was your goal, this token was never the lever, and the only thing that would achieve it is blocking Googlebot, which costs you Search itself. That is the actual trade, and it is worth knowing it is the trade before choosing.

If you do want to opt out of Gemini training and grounding, Google-Extended is exactly the right token and it is cheap, since it costs you nothing in Search. Just understand which of those two things you are buying. Grounding is the mechanism behind retrieval-augmented answering generally, and if you want to understand what you are opting out of, the practical guide to building a retrieval system covers the same mechanism from the builder's side.

Non-Google tokens are a separate matter and change more often than a static page can track. GPTBot and CCBot are real tokens belonging to OpenAI and Common Crawl respectively, and both organisations publish their own crawler documentation, which is the only source worth trusting for their current behaviour. Checking the vendor's own page before adding a token is the whole discipline here.

Worked Example Two: Auditing A Real File

Here is the entire robots.txt from this site, fetched on 22 August 2026:

User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://bilalvirk.com/sitemap.xml

Four meaningful lines, and one of them does nothing. Allow: / under User-agent: * is redundant, because allowed is the default state of everything not disallowed. It is harmless, it is in a large share of the robots.txt files on the web, and it exists because people feel a file should say yes to something. It can go.

Disallow: /admin blocks the admin login and, because there is no trailing slash, anything starting with that string. That is deliberate here. Note what it does not do: it does not secure the admin area. Anyone can read this file, and it is arguably an invitation. The admin area is protected by requiring a login, which is the only protection that means anything. The disallow line is there to keep the page out of crawl reports, not to hide it.

The Sitemap: line is the one line in a robots.txt that adds capability rather than removing it, and it is the line most often missing. It is independent of any user-agent group, so it applies globally regardless of where in the file it sits. If you have not built the file it should point to, the XML sitemap generator produces it.

One rule about that pairing, and it is the one that quietly wastes crawl on prerendered sites: every URL in your sitemap should resolve to a page that actually exists as HTML. A sitemap entry is an invitation. If the invitation leads to an empty shell, you have asked a crawler to spend a request discovering nothing.

Common Mistakes

Blocking the resources the page needs to render. The old version of this page advised blocking /api/. On a client-rendered site that fetches its content from /api/, that instruction prevents the page from rendering at all for anything obeying the file. Google's guidance is conditional rather than prohibitive: you can block resource files "if you think that pages loaded without these resources won't be significantly affected by the loss," but "if the absence of these resources make the page harder for Google's crawler to understand the page, don't block them, or else Google won't do a good job of analyzing pages that depend on those resources." Work out which side of that line your /api/ sits on before blocking it. On most modern sites it is the wrong side.

Using disallow to fix duplicate content. Internal search result pages, faceted URLs and print variants are indexing problems, and disallow is a crawling instruction. The tools that address duplication are noindex, canonical tags and not generating the URLs in the first place. Disallowing them just makes the duplicates invisible to you while leaving them eligible to appear as bare URLs.

Blocking a staging environment with robots.txt. If the staging site is reachable, the file is a list of what to look at. Use HTTP authentication. This is the case where "voluntary compliance" stops being an abstraction.

Assuming one token covers a family. Some crawlers respond to more than one token, and Google's documentation notes that where a crawler has several, "you need to match only one crawler token for a rule to apply." So a rule addressed to Googlebot is honoured by Googlebot-Image and Googlebot-News as well, unless a more specific group exists for them. Adding a specific group can therefore loosen a rule you thought you were tightening, because the more specific group wins and yours no longer applies.

Editing it without checking it. A single misplaced character in this file can deindex a site, and nothing will tell you. Paste the result into the robots.txt tester and check a real URL against it before deploying, then check the live file again after deploying, because some hosts and CMS platforms generate their own and will overwrite yours.

Leaving it to a plugin. Many CMS platforms produce this file dynamically and will not show you what they produced. Fetch your own /robots.txt in a browser after every significant change. It takes five seconds and it is the only way to know what is actually being served.

Frequently Asked Questions

Do I even need a robots.txt file?

No. A site with no robots.txt is fully crawlable, which is usually what you want. A missing file returns 404 and crawlers proceed normally. The reasons to have one are a specific area you want left alone, a genuine crawl-volume problem, or the sitemap line. If none of those apply, an empty file is honest and a missing file is fine.

Will blocking AI crawlers keep my content out of AI answers?

Partly at best, and not in the way most people assume. Blocking Google-Extended stops Gemini training and grounding but explicitly does not remove you from Google Search or its AI features. Blocking vendor-specific tokens like GPTBot affects those vendors' own crawling. None of it is retroactive, none of it binds anything that ignores the file, and none of it touches content already gathered. Treat it as a preference you are registering, not a fence.

What is the difference between disallow and noindex?

Disallow says do not fetch this. Noindex says do not list this. Disallow leaves a page eligible to appear as a bare URL with no description. Noindex removes it from results properly, and requires the page to be crawlable so the instruction can be read. If you want something out of search results, noindex is the answer and robots.txt is the thing that will stop it working.

Does robots.txt affect my rankings?

Not directly. It affects what gets crawled, which affects what can rank at all. The failure mode is dramatic rather than gradual: a stray Disallow: / removes a site from search results, and it happens most often when a staging file gets deployed to production.

Is a redundant Allow: / line a problem?

No, just noise. Allow rules earn their place when you need to carve an exception out of a broader disallow, as in disallowing a directory but allowing one file inside it. A blanket allow at the top of a file with no disallows above it is doing nothing.

Where do I put the sitemap line?

Anywhere in the file. It is not scoped to a user-agent group, so it works at the top, the bottom, or between groups. Convention is at the end. Use the full absolute URL, and list multiple sitemap lines if you have more than one.

When A Generator Is Not Enough

A generator gives you a correct file for the site as it is today. The problem is that this file rots, and it rots silently. Routes get added, a directory gets renamed, a framework starts serving content from a path the file blocks, and nothing surfaces the contradiction because no error is ever thrown. You find out from a traffic graph.

The fix is not a better generator, it is making the file a build output rather than a document. On this site the sitemap and the list of routes to prerender are generated in one pass from the same source, so a URL cannot end up advertised without also being built, and the robots rules are reviewed against that same list. That is a twenty-line script and it removes a whole category of silent failure.

That kind of work is most of what I do on the build side: taking the parts of a site that depend on someone remembering, and making them fall out of the build instead. Sometimes that is sitemap and prerender generation, sometimes it is a rough prototype that works on the developer's machine and needs to survive a deploy, sometimes it is a scenario in n8n or Make that has grown past what a visual editor can hold.

For the neighbouring checks, the robots.txt tester verifies a specific URL against a specific file, and the on-page SEO auditor covers what the crawler finds once you have let it in.

If your traffic dropped and you suspect this file, send me the file and one URL that stopped appearing. That is usually enough to find it.