The robots.txt file is small, but a single typo in it can hide an entire site from search. The ChrysoKit Robots.txt Generator builds the file from a checklist so the syntax is always correct.
Why use it
Reading the robots spec to add one disallow rule is overkill. A guided generator is faster and less error-prone.
How to use the Robots.txt Generator
- Pick which user-agents to target (all bots, or specific ones).
- Add allow and disallow rules from the form.
- Add your sitemap URL.
- Copy the generated robots.txt.
Features worth knowing
User-agent presets
Googlebot, Bingbot, common AI crawlers and the catch-all.
Multi-rule support
Combine allows and disallows per user-agent.
Sitemap reference
Adds the standard sitemap directive at the bottom.
Pro tips
- Disallow does not mean noindex. Pages you disallow can still appear in search results without their content. Use a meta tag for actual noindex.
- Always include a Sitemap line. It is the cheapest SEO win you can ship.
- Test the file with the Sitemap Validator and your search console before relying on it.
Privacy first. The Robots.txt Generator runs entirely in your browser. Nothing you enter is sent to a server.
If you write robots.txt by hand once a year, you will get it wrong once a year. Use the generator and stop.
Open the tool: Robots.txt Generator →
Post-mortem: the disallow that deindexed a product launch
A composite of incidents every SEO consultant has seen: a company launches a redesigned site, traffic collapses 90% over three weeks, panic ensues, and the cause is one line... Disallow: /... shipped from staging to production inside robots.txt. The staging environment was correctly hidden from crawlers; the deployment pipeline copied everything; nobody looks at a file that "never changes". Recovery took six weeks of recrawling after a thirty-second fix.
The quieter variants do slower damage. A Disallow: /products/ left over from an old architecture suppresses a section relaunched years later under the same path. A blanket block on /*? meant to kill parameter spam also blocks the paginated category pages that were the site's main crawl paths. An agency blocks /blog/ "temporarily during migration" and the comment in the ticket outlives everyone's memory of it.
The defences are boring and effective: keep robots.txt in version control so every change has an author and a date; diff it as part of deployment checks, exactly like code; test specific URLs in Search Console's checker after every structural change; and audit the file quarterly against the current site map, asking of each line "what does this block today?"... not what it blocked when written. A four-line file with the power to erase a site deserves at least that much ceremony.