Robots.txt Tester

What this tool does

Paste the contents of a robots.txt file and a list of paths or URLs. For each one you get a verdict — allowed or blocked — the exact rule that decided it, and the user-agent group that rule came from.

The matcher implements the rules Google actually applies rather than a naive first-match scan: group selection by most specific user-agent, * and $ wildcard expansion, longest-path-wins precedence, and least-restrictive on an exact tie. It also flags directives that are parsed but ignored, and syntax that will be discarded.

A worked example

Given:

User-agent: *
Disallow: /admin/
Allow: /admin/public/
Disallow: /*.pdf$

Test /admin/public/report.pdf. A first-match reading says blocked by /admin/. The real answer is that /admin/public/ is nine characters longer than /admin/, so it wins the path contest — but /*.pdf$ matches too and is longer still, so the file is blocked after all, by the PDF rule rather than the admin rule.

Test /admin/public/index.html and it is allowed. That distinction is invisible by eye and is exactly the sort of thing that quietly removes a section from the index.

Directives and how they are treated

The authoritative reference is Google's robots.txt specification, which also documents the 500 KiB parsing limit and how Google behaves when the file is unreachable.

Where people go wrong

Assuming blocked means deindexed. It does not. A URL disallowed in robots.txt can still appear in results if other pages link to it — Google simply cannot see the content, so it shows the URL with no snippet. To remove a page you need noindex on the page itself, which means the page must be crawlable. Blocking it in robots.txt prevents Google from ever seeing the noindex.

Blocking CSS and JavaScript. Google renders pages. Block the assets and it renders a broken page, then judges the page on what it sees.

Trusting first-match order. Rules are not evaluated in file order. Reordering the file changes nothing.

Forgetting robots.txt is per origin. https://example.com/robots.txt does not govern https://blog.example.com/, and the http and https versions are separate origins.

Blocking images that social platforms need. A blocked og:image can render as an empty card. Check with the Open Graph preview if a share looks wrong.

Using robots.txt for security. The file is public and lists exactly what you would rather people did not visit.

FAQ

Does this fetch my live robots.txt?

No. Browsers cannot fetch arbitrary cross-origin URLs, so paste the contents instead. Open yourdomain.com/robots.txt, select all, paste.

Which user-agents can I test?

Any string. Enter Googlebot, Googlebot-Image, Bingbot, * or a bespoke agent, and the tool picks the matching group the way a crawler would — the most specific group only, not a merge of every group that matches.

Why is my noindex in robots.txt not working?

Because it never has been a supported directive there. Google confirmed it was unsupported in 2019 and stopped honouring it entirely. Use a noindex meta tag or an X-Robots-Tag header.

Does Disallow: with nothing after it block everything?

The opposite. An empty disallow value allows everything. Disallow: / blocks everything. One character between the two outcomes.

Are the rules case-sensitive?

Directive names are not. Paths are — /Admin/ and /admin/ are different rules.

Next steps

To write the file rather than test it, use the Robots.txt Generator, then make sure the URLs you do want crawled are listed with the Sitemap Generator. When you are retiring URLs rather than blocking them, the redirect generator produces the rules.

If crawl issues are a symptom of a sprawling manual publishing process, What Is Workflow Automation is a sensible place to start.

The matching rules are specified in RFC 9309, which also sets the 500 KiB point at which a crawler is allowed to stop reading the file — worth knowing if your robots.txt has quietly grown into a wall of legacy disallows.

Losing indexed pages and not sure why? Book a discovery call.