Robots.txt Generator
Crawl rules without the syntax traps.
Runs on your device
At a glance
- Processing:
- Runs locally in your browser
- Privacy:
- Your files stay on your device — nothing is uploaded to a server
About this tool
Toggle common rules (block /admin and /api, allow everything else), add custom paths, point to your sitemap, and copy a valid robots.txt.
How to use it
- Set your domain and sitemap URL.
- Toggle default rules.
- Add custom allow/disallow lines and copy.
Frequently asked questions
Is my data uploaded?
No. robots.txt building runs entirely in your browser — nothing leaves your device.
Does this guarantee pages stay private?
No — robots.txt is a crawl request, not access control. The tool warns about this and never suggests using it for secrets; use noindex/auth for real privacy.
Where do I put the robots.txt file?
At the root of your domain (e.g. https://example.com/robots.txt). Search engines only look there — a copy in a subfolder is ignored.