Robots.txt Tester and Generator
Paste or fetch any robots.txt and see which URLs Googlebot, GPTBot or any crawler can reach, with the winning rule, its specificity and every candidate shown. Two named profiles: documented Google behavior and the RFC 9309 baseline.
Parsed locally under RFC 9309 rules: your file never leaves the browser.
Not sure what a crawler calls itself? Check the Bot User-Agent Lookup.
Google matches user-agent tokens by prefix and ignores a trailing *; the RFC baseline requires exact tokens. Results for other crawlers are baseline estimates.
| Path | Verdict | Matched rule |
|---|
Presets come from the sourced crawler registry below (verified 2026-07-18) and never replace your rules without a preview. Voluntary robots rules only reach crawlers that honor them.
Left empty, no Sitemap line is written. This field never ships a placeholder.
Generated and validated locally in your browser (round-tripped through the same parser as the tester). Upload it to your site root as /robots.txt. Robots.txt is public: never use Disallow to hide secrets.
How the robots.txt tester matches your URLs
The tester implements two named profiles rather than claiming one universal interpretation. The Google profile follows Google's documented behavior: user-agent tokens match by prefix (a Googlebot-Image crawler is caught by a Googlebot group), a trailing * on a rule adds nothing, and equally specific Allow and Disallow rules resolve to Allow, the least restrictive. The RFC 9309 baseline requires exact token matches and is the honest estimate for crawlers whose parsers are undocumented. Crawler compliance itself varies, and some agents ignore robots.txt entirely.
Inside the selected group, the most specific matching rule decides, measured on the effective matched path, not the raw pattern string. Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file because it matches more of the URL, which is exactly how the standard WordPress robots.txt keeps AJAX working while the admin stays blocked. The tester shows the winning rule, its line number, its specificity and every other candidate that matched, plus the normalized path it compared, so debugging stops being guesswork. Not sure what token a bot sends? Look it up in the Bot User-Agent Lookup.
Which AI crawlers can you block with robots.txt
Blocking AI bots is the biggest reason people edit robots.txt in 2026, and the tokens change constantly. This registry is data with sources, not folklore: every entry links to the provider's official documentation and was last verified on 2026-07-18. Purposes matter more than names: blocking a training crawler does not block the search indexer, and user-triggered fetch agents may not follow robots rules the way bulk crawlers do.
| Token | Provider | Purpose | What it does |
|---|---|---|---|
| GPTBot | OpenAI | AI model training | Crawls for training OpenAI models. Source |
| OAI-SearchBot | OpenAI | AI search indexing | Indexes pages for ChatGPT search results, with links. Source |
| ChatGPT-User | OpenAI | User-triggered fetch | Fetches a page when a ChatGPT user asks about it; may not behave like a bulk crawler. Source |
| ClaudeBot | Anthropic | AI model training | Crawls for training Anthropic models. Source |
| Claude-SearchBot | Anthropic | AI search indexing | Indexes pages to improve Claude search results. Source |
| Claude-User | Anthropic | User-triggered fetch | Fetches a page when a Claude user asks about it. Source |
| PerplexityBot | Perplexity | AI search indexing | Indexes pages for cited Perplexity answers, not model training. Source |
| Perplexity-User | Perplexity | User-triggered fetch | Fetches when a user asks; Perplexity documents that it may ignore robots rules for direct user requests. Source |
| Google-Extended | Use-control token (opt-out, not a crawler) | Not a crawler: an opt-out token controlling Gemini training and grounding. Blocking it does not affect Google Search. Source | |
| Applebot-Extended | Apple | Use-control token (opt-out, not a crawler) | Opt-out token for Apple Intelligence training; regular Applebot keeps serving Siri and Spotlight. Source |
| CCBot | Common Crawl | AI model training | Open web archive widely used as AI training raw material. Source |
| Meta-ExternalAgent | Meta | AI model training | Collects content Meta uses to train Llama and other AI systems. Source |
| Bytespider | ByteDance | AI model training | Feeds ByteDance AI models; known for aggressive request volume. Source |
| Amazonbot | Amazon | AI search indexing | Crawls for Alexa answers and Amazon AI features. Source |
Robots.txt stops polite bots. The rest need enforcement
Every rule in this file is a request, not a firewall. Reputable crawlers identify themselves and comply. Scrapers, click bots and content thieves send a fake Chrome user-agent and walk straight past your Disallow lines. If your analytics show bot traffic that robots.txt should have stopped, that is expected behavior, not a broken file. Enforcement has to happen server-side, based on how a visitor behaves rather than what it claims to be.
Robots.txt FAQ
Does robots.txt block AI bots like GPTBot and ClaudeBot?
Only the polite ones, and purpose matters. Bulk crawlers like GPTBot, ClaudeBot and CCBot declare their names and honor robots.txt, so a Disallow group keeps them out. User-triggered fetch agents such as ChatGPT-User, Claude-User and Perplexity-User act on behalf of one person asking about your page and may not follow robots rules the same way, which their providers document. And robots.txt is a convention, not an enforcement layer: scrapers that fake a browser user-agent ignore it completely. For those you need server-side detection that inspects behavior, not self-declared names.
What is the difference between robots.txt and noindex?
Robots.txt controls crawling, noindex controls indexing, and mixing them up is the classic trap. If you block a page in robots.txt, Google cannot fetch it, which means it never sees a noindex tag on that page. To remove a page from search results, allow crawling and add a meta robots noindex tag or an X-Robots-Tag header.
Does Google respect Crawl-delay?
No. Googlebot has never supported the Crawl-delay directive, while Bing and Yandex still read it. If Googlebot crawls too aggressively, reduce the crawl rate in Search Console or return temporary 429 or 503 responses. The tester surfaces every Crawl-delay line it finds so you know which crawlers will actually obey it.
How do wildcards work in robots.txt?
Two special characters exist: * matches any sequence of characters and $ anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /private blocks anything starting with /private, including /private-notes. When several rules match, the longest one wins, and Allow beats Disallow at equal length.
Why does a page blocked in robots.txt still show up in Google?
Because blocking crawling does not block indexing. If other sites link to the URL, Google can index the bare address without ever visiting it, showing it with a "no information available" snippet. Unblock the page and serve a noindex tag if you want it gone, or use the removal tool in Search Console for urgent cases.
What should a WordPress robots.txt look like?
Keep it short: disallow /wp-admin/, allow /wp-admin/admin-ajax.php so front-end plugins keep working, and list your XML sitemap. Do not block /wp-content/ or /wp-includes/, an old myth that stops Google from rendering your CSS and JavaScript and can hurt rankings. Add per-bot groups only when you want different treatment, for example blocking AI training crawlers.
Where does the robots.txt file have to live?
At the root of the host, spelled exactly robots.txt in lowercase: https://example.com/robots.txt. Crawlers never look in subdirectories, so /blog/robots.txt is ignored. Each subdomain and protocol needs its own file, meaning shop.example.com and example.com are separate. The file must be plain text, UTF-8 encoded, and served with a 200 status code.
More free tools: Bot User-Agent Lookup · Meta Tag Checker · Redirect Checker