Robots.txt Generator
Create a basic robots.txt policy with sitemap support. Your work stays on this device whenever possible.
How to use Robots.txt Generator
- Start from a preset button such as "Preset: Block AI Scrapers" or "Preset: WordPress", or fill the fields by hand.
- Choose a "User-agent preset" (All crawlers, Googlebot, Bingbot, GPTBot, CCBot, ClaudeBot, Applebot or Custom) and, for Custom, type the name into "Custom user-agent".
- List paths in "Disallow paths (one per line)" and "Allow paths (optional, one per line)", and pick a "Crawl-delay" if you want one.
- Add "Sitemap URLs (one per line)" and, only when targeting Yandex, the "Host directive (optional)".
- Press "Generate robots.txt", then "Copy robots.txt" or "Download robots.txt".
- Type a path into "URL Path to test" and press "Test path" to see which rule wins.
Worked example
Keep the defaults: user-agent *, disallow /admin/, /private/, /tmp/ and /api/, allow /admin/public-assets/ and /. The output starts with a comment, then "User-agent: *", four Disallow lines, two Allow lines, a blank line and the two Sitemap lines. Testing /admin/dashboard reports DISALLOWED by "Disallow: /admin/" with rule length 7, because it beats the one-character "/" allow.
When to use this tool
- Blocking common AI training crawlers on a personal blog in one click.
- Keeping cart and checkout paths out of a shop's crawl budget.
- Checking whether an Allow exception really overrides a broader Disallow before deploying.
Important limitations
robots.txt is an advisory crawl specification (RFC 9309), not security access control. Well-behaved search crawlers honor these rules, but malicious scrapers and unauthorized bots may ignore them.
Frequently asked questions
Does robots.txt keep a page out of search results?
No. It asks well-behaved crawlers not to fetch a URL, but a blocked address can still be listed if other sites link to it. Use a noindex directive or real authentication for pages that must stay out or private.
How does the path tester decide?
It treats each Allow and Disallow rule as a prefix, keeps the longest match and lets Allow win ties, as RFC 9309 describes for plain paths. Wildcards such as * and $ are not evaluated, so rules like the e-commerce preset's /*?*sort= cannot be tested here.
Does the Simulated Crawler menu change the answer?
Not currently. The tester applies whatever is in the Allow and Disallow boxes to every crawler, so switching between Any, Googlebot and GPTBot shows the same verdict.
Will Google follow Crawl-delay and Host?
Google ignores Crawl-delay, though some other crawlers honour it. Host is a Yandex-specific directive outside RFC 9309. Leave both off unless you know a target crawler reads them.
How do I write separate rules for several bots?
The form writes one User-agent group per run, apart from the AI preset, which lists several. Generate each group separately and join them by hand with a blank line between groups; Sitemap lines apply globally.