Robots.txt Generator Online – Free
Crawl rules without the syntax traps.
At a glance
- Processing:
- Runs locally in your browser
- Privacy:
- Nothing you type is sent to a server
Last updated: 2026-10-07
Common use cases
- Blocking staging admin paths before launch day
- Opening a new blog to crawlers with a sitemap line
- Hiding draft folders during content review
- Adding API limits for a docs portal
About this tool
Toggle common rules (block /admin and /api, allow everything else), add custom paths, point to your sitemap, and copy a valid robots.txt.
How to use it
- Set your domain and sitemap URL.
- Toggle default rules.
- Add custom allow/disallow lines and copy.
Practical tips
- Type the full site URL so the Sitemap line builds correctly. A trailing slash is trimmed for you.
- Use the admin and api boxes for quick defaults before custom lines. Untick either to open that path.
- Write one rule per line in Extra. Plain paths gain Disallow automatically. Prefix with Allow to open a path.
Common mistakes
- Listing private files here for secrecy. Crawlers may read the file, so use auth or noindex instead. Treat the file as public.
- Putting the file in a subfolder. Search tools read only the domain root copy. Upload to robots.txt exactly at the root.
- Skipping the Sitemap box. Tick Include Sitemap line so crawlers find the index. The line uses your typed domain.
How this compares
This tool assembles the text in your tab with no account and offers instant download, and nothing is transmitted. You give up cloud project lists and shared audits across devices. You keep a private draft you can paste at the domain root. There is no draft store to clean.
Good to know
- Output starts with User-agent star and Allow slash. The header is fixed.
- Extra lines accept Allow or Disallow. Other directives need manual edits.
- No live crawl test runs here. Upload to the root and check with console tools.
Frequently asked questions
Is my data uploaded?
No. Building robots.txt runs entirely in your browser — nothing leaves your device.
Does this guarantee pages stay private?
No — robots.txt is a crawl request, not access control. The tool warns about this and never suggests using it for secrets; use noindex/auth for real privacy.
Where do I put the robots.txt file?
At the root of your domain (e.g. https://example.com/robots.txt). Search engines only look there — a copy in a subfolder is ignored.
Can I block just one bad bot?
Not in this panel. Output targets all crawlers with User-agent star. Add a named agent block by hand after you copy the text. The Sitemap line stays last in the generated file.

