WordPress Robots.txt Generator
Build the right robots.txt for WordPress in seconds. Pick a preset, flip plain-English toggles, and get a file that blocks the junk, keeps admin-ajax.php working and shuts out AI crawlers if you want. Runs entirely in your browser.
Yoast and Rank Math publish at /sitemap_index.xml, core WordPress at /wp-sitemap.xml. Leave empty to skip the line; no placeholder is ever written into the file.
Generated entirely in your browser and validated by the same parser as the Robots.txt Tester. Honesty note: robots.txt controls crawling by compliant bots only. It does not remove already-indexed URLs (use a noindex tag on a crawlable page for that), and it does not stop scrapers that ignore the file.
How to install robots.txt on WordPress
WordPress does not ship an editable robots.txt file. When no physical file exists, it answers the /robots.txt URL with a small virtual file generated on the fly. You have three ways to replace it with the file you built above.
1. Upload a physical file (recommended)
Download the file above, then upload it to your site root, the folder that holds wp-config.php, often called public_html. Use SFTP or your host's file manager. Visit yourdomain.com/robots.txt to confirm it is live. A physical file overrides the virtual one immediately.
2. Use the robots_txt filter (no file access needed)
If you cannot reach the server, WordPress lets code rewrite the virtual file through the robots_txt filter. This snippet contains EXACTLY the rules you built above and updates live as you change them. Drop it into a child theme's functions.php or a small mu-plugin. Important: it only works while no physical robots.txt file exists; a physical file at the site root overrides the filter completely.
3. Edit through your SEO plugin
Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt) and All in One SEO (Tools, then Robots.txt Editor) all include a robots.txt editor. Paste the generated output, save, and the plugin serves it for you.
WordPress robots txt example: what to block, what to leave open
This table is the reasoning behind the toggles. The most common mistake in a robots.txt for WordPress is blocking too much: Google renders pages like a browser, so your CSS, JavaScript and Ajax endpoints must stay reachable. When in doubt, leave it open.
| WordPress path | Verdict | Why |
|---|---|---|
| /wp-admin/ | Block | Dashboard and login screens have no search value. This is the one Disallow rule nearly every WordPress site should ship, always paired with the admin-ajax.php Allow line. |
| /wp-admin/admin-ajax.php | Keep open | The front-end Ajax endpoint. Themes and plugins call it from public pages for live search, filters and forms, so an Allow line must punch through the /wp-admin/ block. |
| /wp-includes/ | Keep open | An old myth says to block it. Google renders pages like a browser and needs these core scripts; blocking them can make pages look broken to Google and hurt rankings. |
| /wp-content/plugins/ | Keep open | Plugin CSS and JavaScript are part of how your pages render. Google explicitly asks site owners not to block assets it needs for rendering. |
| /wp-content/uploads/ | Keep open | Blocking uploads removes every image from Google Images, which is usually a meaningful chunk of free traffic. |
| /wp-json/ | Keep open | The REST API serves oEmbed data and powers front-end features in many themes and page builders. Block it only if you fully understand the trade. |
| /?s= and /search/ | Block | Internal search results are an infinite space of thin pages and a favorite injection point for search spam. |
| /feed/ and /*/feed/ | Optional | Feeds duplicate every post as XML. Blocking them tidies crawl budget, and RSS readers ignore robots.txt, so subscribers are unaffected either way. |
| /trackback/ | Block | A legacy ping protocol from early blogging that today attracts only spam bots. |
| /cart/, /checkout/, /my-account/ | Block (Woo) | Session-bound WooCommerce pages that are personal to each shopper and should never appear in search results. |
| wp-login.php | Optional | WordPress already marks the login page noindex. A Disallow line is harmless but mostly just trims bot noise from your logs. |
Block bots on WordPress: robots.txt asks, enforcement decides
The AI toggles are purpose-based because AI agents are not one thing. Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent) collect content for model training and honor a Disallow. Search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so AI answers can cite you with links, and blocking them is a different decision most sites do not want. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) act for one person asking about your page and may not follow robots rules the same way. Google-Extended and Applebot-Extended are opt-out control tokens, not crawlers: blocking them affects model use of your content without touching Google Search or Siri. The token list comes from official provider documentation, last verified 2026-07-18.
But WordPress sites attract a second class of bots that never announce themselves: scrapers cloning your content, brute-force scripts hammering wp-login.php, click bots polluting your analytics. They send fake Chrome user-agents and skip robots.txt entirely. A robots.txt is a polite request. Enforcement means detecting bots by how they behave and stopping them at the server, which is exactly what DevDome Bot Protection does.
WordPress robots.txt FAQ
What is the best robots.txt for WordPress?
Shorter than you think. Block /wp-admin/, allow /wp-admin/admin-ajax.php, list your XML sitemap, and stop there unless you have a specific reason to add more. Add WooCommerce, feed or AI crawler rules only when they solve a problem you actually have. Every extra Disallow line is another chance to block something Google needs to render your pages.
Should I block /wp-includes/ or /wp-content/ in robots.txt?
No. This advice was everywhere ten years ago and it is now actively harmful. Google renders pages like a browser, so it needs the CSS and JavaScript that live in /wp-includes/ and /wp-content/. Blocking them can make your pages look broken to Google and hurt rankings. Google documentation asks site owners not to block these assets.
Why does my robots.txt need to allow admin-ajax.php?
Because blocking /wp-admin/ without an exception also blocks /wp-admin/admin-ajax.php, the endpoint many themes and plugins call from the public site for live search, filters, sliders and forms. Crawlers that cannot reach it may fail to render those features. The Allow line is longer than the Disallow line, so it wins under robots.txt precedence rules and unblocks exactly that one file.
Does WordPress create a robots.txt file automatically?
Yes, a virtual one. If no physical robots.txt exists at your site root, WordPress answers the URL itself with a minimal file: it blocks /wp-admin/, allows admin-ajax.php and lists the core wp-sitemap.xml. You can replace it three ways: upload a physical file, which overrides the virtual one, use the robots_txt filter in code, or use the editor in an SEO plugin like Yoast or Rank Math.
Can robots.txt block bots on WordPress?
Only bots that choose to obey it, and robots.txt controls crawling, not indexing or access. Declared crawlers like GPTBot, ClaudeBot and CCBot honor a Disallow, so the training toggle keeps compliant bots out. User-triggered AI fetchers may not follow the file the same way, scrapers with fake browser user-agents ignore it entirely, and a blocked URL that other sites link to can still appear in search results. Stopping non-compliant bots requires server-side detection based on behavior, which is what DevDome Bot Protection does.
Will blocking AI crawlers hurt my Google rankings?
Blocking the TRAINING toggle: no. It never names Googlebot or Bingbot, and Google-Extended is a pure opt-out that removes your content from Gemini training while your Google Search position stays untouched. Blocking the AI SEARCH toggle is different: it removes your pages from AI assistants that would otherwise cite you with links, which is a visibility trade you should make deliberately. That is exactly why they are two separate switches.
Where does the robots.txt file go on a WordPress site?
In the site root, the same folder as wp-config.php, so it is reachable at yourdomain.com/robots.txt. Crawlers only ever check that exact location. If WordPress lives in a subdirectory like /blog/ but serves your main domain, the file still belongs at the domain root, not inside the subdirectory.
More free tools: Robots.txt Tester · Bot User-Agent Lookup · XML Sitemap Generator · LLMs.txt Generator · Security Headers Checker