Robots.txt Generator
Generate a valid robots.txt file and control exactly what crawlers can access
User-agent: * Disallow: /admin Allow: / Sitemap: https://bilalvirk.com/sitemap.xml
Create a robots.txt file that tells search engine crawlers exactly where they can and can't go. Block sensitive admin areas, allow your important pages, and point crawlers to your sitemap — all with a visual builder.
What Is robots.txt?
The robots.txt file sits at the root of your domain (https://yoursite.com/robots.txt) and gives instructions to web crawlers about which pages they can and cannot access.
Important: robots.txt is a convention, not a security measure. Malicious bots ignore it. Only use it to manage crawl budget, not to hide sensitive data.
Common robots.txt Rules
# Allow all crawlers to access everything (default)
User-agent: *
Allow: /
# Block all crawlers from admin areas
User-agent: *
Disallow: /admin/
Disallow: /api/
# Block only GPTBot (OpenAI's web crawler)
User-agent: GPTBot
Disallow: /
# Point to sitemap
Sitemap: https://yoursite.com/sitemap.xmlWhat You Should Block
- Admin panels (
/admin/,/wp-admin/) - API endpoints (
/api/) - User-specific pages (
/dashboard/,/account/) - Checkout and payment pages
- Internal search result pages (duplicate content)
- Staging environments (better handled with
noindex)
Blocking AI Training Crawlers
Many website owners now block AI company crawlers to prevent their content from being used to train LLMs:
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Don't Forget the Sitemap Line
Every robots.txt should point to your sitemap so crawlers can find it without guessing. If you haven't generated one yet, the XML Sitemap Generator builds the file this tool's Sitemap: line should point to.
Crawl Budget and Large Sites
For sites with thousands of pages, robots.txt helps search engines prioritise crawl budget on your important pages. Book a consultation to review your crawl strategy.
Want this built against your real numbers?
A 30-minute call to scope the workflow, agent, or automation you actually need.
More seo tools
All toolsContent Brief Generator
Generate a complete SEO content brief from any keyword — ready to hand to a writer
Keyword Clustering Tool
Group any keyword list by search intent and topic — ready for content planning
Keyword Research & Intent Analyzer
Classify any keyword by search intent and know exactly what type of content to build
On-Page SEO Auditor
Get a full on-page SEO score with prioritised fixes for any URL in seconds
Schema Markup Generator
Generate valid JSON-LD schema markup for rich snippets — FAQ, Service, Person, and more
SEO Meta Tag Analyzer
Check any page's title, meta description, and OG tags in seconds
Have a workflow that's burning hours every week?
Bring me one real bottleneck. I'll tell you whether it's worth automating, and what it would take.