Robots.txt Tester and Generator
Paste any robots.txt and see instantly which URLs Googlebot, Bingbot, GPTBot or any crawler can reach, with the exact rule that decides. Or build a clean file from presets. Everything runs in your browser.
Parsed locally under RFC 9309 rules: your file never leaves the browser.
Not sure what a crawler calls itself? Check the Bot User-Agent Lookup.
| Path | Verdict | Matched rule |
|---|
The AI preset blocks GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider, Meta-ExternalAgent, Amazonbot and Applebot-Extended. Each one is explained below the tool.
RulesGenerated locally in your browser. Upload it to your site root as /robots.txt.
How the robots.txt tester matches your URLs
The tester implements RFC 9309, the Robots Exclusion Protocol that Google, Bing and every serious crawler have followed since it became an official standard. Group selection works by token: a crawler picks the group whose User-agent line is the most specific match for its own name, and only falls back to the * group when nothing else matches. That is why a Googlebot-Image group overrides a Googlebot group, which overrides the wildcard.
Inside the winning group, the longest matching rule decides. Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file because the rule is longer, which is exactly how the standard WordPress robots.txt keeps AJAX working while the admin stays blocked. The tester shows the precise rule that fired for every path, so debugging stops being guesswork. Not sure what token a bot sends? Look it up in the Bot User-Agent Lookup.
Which AI crawlers can you block with robots.txt
Blocking AI bots is the biggest reason people edit robots.txt in 2026. These are the tokens the generator preset writes, and what each one actually feeds:
- GPTBot: OpenAI's training crawler. Pages it collects can appear in future GPT models.
- Google-Extended: an opt-out token, not a crawler. Blocking it keeps content out of Gemini training without touching your Google Search rankings.
- CCBot: Common Crawl's archiver. Its open dataset is raw material for many AI training runs.
- ClaudeBot: Anthropic's crawler gathering training data for Claude models.
- PerplexityBot: indexes pages for Perplexity's live, cited answers rather than model training.
- Bytespider: ByteDance's crawler feeding its AI models, known for aggressive request volume.
- Meta-ExternalAgent: collects content Meta uses to train Llama and other AI systems.
- Amazonbot: crawls for Alexa answers and Amazon's AI features.
- Applebot-Extended: an opt-out token that removes your content from Apple Intelligence training while regular Applebot keeps serving Siri and Spotlight.
Robots.txt stops polite bots. The rest need enforcement
Every rule in this file is a request, not a firewall. Reputable crawlers identify themselves and comply. Scrapers, click bots and content thieves send a fake Chrome user-agent and walk straight past your Disallow lines. If your analytics show bot traffic that robots.txt should have stopped, that is expected behavior, not a broken file. Enforcement has to happen server-side, based on how a visitor behaves rather than what it claims to be.
Robots.txt FAQ
Does robots.txt block AI bots like GPTBot and ClaudeBot?
Only the polite ones. GPTBot, ClaudeBot, CCBot, PerplexityBot and Google-Extended declare their names and honor robots.txt, so a Disallow group keeps them out. But robots.txt is a convention, not an enforcement layer: scrapers that fake a browser user-agent ignore it completely. For those you need server-side detection that inspects behavior, not self-declared names.
What is the difference between robots.txt and noindex?
Robots.txt controls crawling, noindex controls indexing, and mixing them up is the classic trap. If you block a page in robots.txt, Google cannot fetch it, which means it never sees a noindex tag on that page. To remove a page from search results, allow crawling and add a meta robots noindex tag or an X-Robots-Tag header.
Does Google respect Crawl-delay?
No. Googlebot has never supported the Crawl-delay directive, while Bing and Yandex still read it. If Googlebot crawls too aggressively, reduce the crawl rate in Search Console or return temporary 429 or 503 responses. The tester surfaces every Crawl-delay line it finds so you know which crawlers will actually obey it.
How do wildcards work in robots.txt?
Two special characters exist: * matches any sequence of characters and $ anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /private blocks anything starting with /private, including /private-notes. When several rules match, the longest one wins, and Allow beats Disallow at equal length.
Why does a page blocked in robots.txt still show up in Google?
Because blocking crawling does not block indexing. If other sites link to the URL, Google can index the bare address without ever visiting it, showing it with a "no information available" snippet. Unblock the page and serve a noindex tag if you want it gone, or use the removal tool in Search Console for urgent cases.
What should a WordPress robots.txt look like?
Keep it short: disallow /wp-admin/, allow /wp-admin/admin-ajax.php so front-end plugins keep working, and list your XML sitemap. Do not block /wp-content/ or /wp-includes/, an old myth that stops Google from rendering your CSS and JavaScript and can hurt rankings. Add per-bot groups only when you want different treatment, for example blocking AI training crawlers.
Where does the robots.txt file have to live?
At the root of the host, spelled exactly robots.txt in lowercase: https://example.com/robots.txt. Crawlers never look in subdirectories, so /blog/robots.txt is ignored. Each subdomain and protocol needs its own file, meaning shop.example.com and example.com are separate. The file must be plain text, UTF-8 encoded, and served with a 200 status code.
More free tools: Bot User-Agent Lookup · Meta Tag Checker · Redirect Checker