How to use
- 1 Select which bots to configure, or click "Block AI crawlers" to target GPTBot, Claude-Web, and other training bots in one click.
- 2 Add paths to Disallow (blocked) and Allow (explicitly permitted), one per line. Use "/" to block an entire bot.
- 3 Optionally set a Crawl-delay, add sitemap URLs, and drop in any # comments.
- 4 Click Copy or Download to grab the generated robots.txt and place it at the root of your website.
Key features
- Preset list of common bots: Googlebot, Bingbot, GPTBot, Claude-Web, and more
- Block-AI-crawlers preset and common blocked paths for one-click configuration
- Custom user-agent support for any crawler not in the preset list
- Crawl-delay, multiple sitemaps, and # comment support
- Validation warnings for paths, URLs, and delays to prevent broken rules
- 100% client-side: your site paths never leave the browser
What Is a Robots.txt Generator?
robots.txt is a plain-text file placed in the root of a website that tells crawlers which parts of the site they are allowed to access. It follows the Robots Exclusion Protocol (RFC 9309): each group starts with a User-agent: line naming the crawler, followed by Allow: and Disallow: rules for the paths it may or may not crawl.
A robots.txt generator builds this file from a form, so you don't have to remember the exact syntax or which crawler is which. Beyond the classic search bots, it now matters to decide how AI training crawlers (GPTBot, Claude-Web, Google-Extended) treat your content. This tool generates a spec-compliant file 100% in your browser.
Common Use Cases
Block AI training crawlers
Stop GPTBot, Claude-Web, Google-Extended, and similar bots from using your content to train models, without affecting your search rankings.
Hide staging and private areas
Keep staging environments, preview builds, and internal pages out of search results with a single disallow rule.
Protect admin and restricted paths
Block crawlers from wasting crawl budget on /admin/, /api/, or authentication pages that should never be indexed.
Control crawl budget
Direct search engines toward the pages that matter by disallowing low-value sections like filters, search results, and pagination.
Spec-compliant output
Generates valid RFC 9309 syntax with Allow, Disallow, Crawl-delay, and Sitemap directives ready to deploy.
Point crawlers to your sitemap
Add your sitemap URL so crawlers discover new and updated pages faster from day one.
Common Crawler User-agents
Which user-agent string each bot uses in robots.txt.
| User-agent | Owner | Purpose |
|---|---|---|
| Googlebot | Search indexing | |
| Google-Extended | AI training (Gemini) — separate from search | |
| Bingbot | Microsoft | Search indexing |
| GPTBot | OpenAI | AI training |
| Claude-Web | Anthropic | AI training |
| PerplexityBot | Perplexity | AI search answers |
| CCBot | Common Crawl | Web archive / AI datasets |
| Applebot | Apple | Siri, Spotlight, Apple Search |
| YandexBot | Yandex | Search indexing |
| DuckDuckBot | DuckDuckGo | Search indexing |
Frequently Asked Questions
Where do I place the robots.txt file?
In the root of your domain, so it is reachable at https://example.com/robots.txt. The path in rules is always relative to that root: /admin/ refers to https://example.com/admin/. The file applies per host, so a subdomain needs its own copy.
Does Googlebot respect Crawl-delay?
No. Googlebot ignores Crawl-delay entirely — use the crawl-rate limit in Google Search Console instead. Bing, Yandex, and a few smaller crawlers still honor it, which is why the field exists.
Should I block AI crawlers like GPTBot?
Only if you don't want your content used to train AI models. Blocking GPTBot, Claude-Web, or Google-Extended does not affect your search ranking — search crawling uses separate user-agents (Googlebot, Bingbot). The "Block AI crawlers" preset targets exactly those training bots while leaving search bots untouched.
Does robots.txt guarantee a page won't be indexed?
No. robots.txt only prevents crawling, and it's a suggestion, not an enforcement. If other sites link to a blocked URL, Google may still index it without the content. For pages you want out of the index, prefer a noindex meta tag or X-Robots-Tag header.
Is my data sent to a server?
No. Everything runs in your browser. The paths you enter and the generated file never leave your machine — useful if you're writing rules for a private or unreleased staging site.