Robots.txt Generator

Build a clean, validated robots.txt file with presets for common setups.

User-agent groups

Sitemap URLs

Generated robots.txt

How to use

  1. Choose a preset or start from scratch.
  2. For each user-agent group, add Allow and Disallow paths and an optional crawl-delay.
  3. Paste one or more sitemap URLs.
  4. Copy the generated robots.txt and upload it to the root of your site.
  5. Test the live file at https://yoursite.com/robots.txt and in Google Search Console.

Frequently asked questions

Where should I put my robots.txt file?

It must sit at the root of your domain, for example https://example.com/robots.txt. Search engines do not look elsewhere.

Does Disallow guarantee a page will not appear in search?

No. Disallow stops crawling but does not prevent indexing of URLs that are linked from elsewhere. Use a noindex meta tag or HTTP header on the page itself for a hard guarantee.

Do all crawlers honour robots.txt?

Major search engines do, but malicious or aggressive bots often ignore it. For security or rate limiting, server-side rules are required.

Can I block specific files like PDFs?

Yes. Use a pattern such as Disallow: /*.pdf$ if the bot supports wildcards. Google and Bing both do.

What robots.txt controls, precisely

robots.txt regulates crawling, not indexing, and the difference matters. A disallowed URL can still appear in Google results (as a bare link without description) if other pages link to it, because Google never fetched the page to see any noindex instruction. To keep a page out of the index, allow crawling and use a noindex meta tag or header; to save crawl budget on infinite parameter spaces and admin areas, use robots.txt. Using disallow for secrecy is worse than useless... the file is public at a known URL, so it doubles as a map of everything you wanted hidden.

Syntax rules that actually trip people

  • Rules group under User-agent lines; the most specific matching group wins, and a bot matching a specific group ignores the * group entirely.
  • Disallow: with an empty value allows everything; Disallow: / blocks the entire site (the classic launch-day accident when a staging file ships to production).
  • Longest matching path wins between Allow and Disallow, which is how you re-open one folder inside a blocked tree.
  • Crawl-delay is ignored by Google (set rate in Search Console instead) though Bing and others respect it.
  • The Sitemap: line takes a full absolute URL and works from any position in the file.

A sensible default policy

Most sites need very little: allow everything, block genuinely crawl-wasteful paths (internal search results, cart and checkout flows, parameter permutations), and declare sitemaps. Audit the file whenever the site structure changes... a stale disallow from an old architecture quietly suppressing a live section is one of SEO's most common self-inflicted wounds.

Reading other sites' robots.txt as research

Because the file is public by design (append /robots.txt to any domain), it doubles as a window into how serious sites manage crawling. Large e-commerce files show systematic parameter blocking; news sites show AI-crawler policy in real time, with user-agent groups for GPTBot and friends appearing across the industry; and any site's Sitemap lines hand you its content inventory. Studying three or four well-run files in your own niche teaches more current practice than most tutorials... and occasionally reveals what a competitor considers crawl-wasteful, which is its own kind of intelligence.

Advertisement