Robots.txt Generator
Build a clean, validated robots.txt file with presets for common setups.
User-agent groups
Sitemap URLs
Generated robots.txt
How to use
- Choose a preset or start from scratch.
- For each user-agent group, add Allow and Disallow paths and an optional crawl-delay.
- Paste one or more sitemap URLs.
- Copy the generated robots.txt and upload it to the root of your site.
- Test the live file at
https://yoursite.com/robots.txtand in Google Search Console.
Frequently asked questions
Where should I put my robots.txt file?
It must sit at the root of your domain, for example https://example.com/robots.txt. Search engines do not look elsewhere.
Does Disallow guarantee a page will not appear in search?
No. Disallow stops crawling but does not prevent indexing of URLs that are linked from elsewhere. Use a noindex meta tag or HTTP header on the page itself for a hard guarantee.
Do all crawlers honour robots.txt?
Major search engines do, but malicious or aggressive bots often ignore it. For security or rate limiting, server-side rules are required.
Can I block specific files like PDFs?
Yes. Use a pattern such as Disallow: /*.pdf$ if the bot supports wildcards. Google and Bing both do.
Advertisement