WordPress Robots.txt Generator
Build the right robots.txt for WordPress in seconds. Pick a preset, flip plain-English toggles, and get a file that blocks the junk, keeps admin-ajax.php working and shuts out AI crawlers if you want. Runs entirely in your browser.
Yoast and Rank Math publish at /sitemap_index.xml, core WordPress at /wp-sitemap.xml. Clear the field to skip the line.
Generated entirely in your browser. When you are done, paste the file into the Robots.txt Tester to see exactly which URLs Googlebot, GPTBot or any other crawler can reach.
How to install robots.txt on WordPress
WordPress does not ship an editable robots.txt file. When no physical file exists, it answers the /robots.txt URL with a small virtual file generated on the fly. You have three ways to replace it with the file you built above.
1. Upload a physical file (recommended)
Download the file above, then upload it to your site root, the folder that holds wp-config.php, often called public_html. Use SFTP or your host's file manager. Visit yourdomain.com/robots.txt to confirm it is live. A physical file overrides the virtual one immediately.
2. Use the robots_txt filter (no file access needed)
If you cannot reach the server, WordPress lets code rewrite the virtual file through the robots_txt filter. Drop a snippet like this into a child theme's functions.php or a small mu-plugin, returning your generated rules. It only works while no physical robots.txt exists.
add_filter( 'robots_txt', function ( $output ) {
$output = "User-agent: *\n";
$output .= "Disallow: /wp-admin/\n";
$output .= "Allow: /wp-admin/admin-ajax.php\n";
$output .= "Sitemap: https://yourdomain.com/sitemap_index.xml\n";
return $output;
}, 20 );3. Edit through your SEO plugin
Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt) and All in One SEO (Tools, then Robots.txt Editor) all include a robots.txt editor. Paste the generated output, save, and the plugin serves it for you.
WordPress robots txt example: what to block, what to leave open
This table is the reasoning behind the toggles. The most common mistake in a robots.txt for WordPress is blocking too much: Google renders pages like a browser, so your CSS, JavaScript and Ajax endpoints must stay reachable. When in doubt, leave it open.
| WordPress path | Verdict | Why |
|---|---|---|
| /wp-admin/ | Block | Dashboard and login screens have no search value. This is the one Disallow rule nearly every WordPress site should ship, always paired with the admin-ajax.php Allow line. |
| /wp-admin/admin-ajax.php | Keep open | The front-end Ajax endpoint. Themes and plugins call it from public pages for live search, filters and forms, so an Allow line must punch through the /wp-admin/ block. |
| /wp-includes/ | Keep open | An old myth says to block it. Google renders pages like a browser and needs these core scripts; blocking them can make pages look broken to Google and hurt rankings. |
| /wp-content/plugins/ | Keep open | Plugin CSS and JavaScript are part of how your pages render. Google explicitly asks site owners not to block assets it needs for rendering. |
| /wp-content/uploads/ | Keep open | Blocking uploads removes every image from Google Images, which is usually a meaningful chunk of free traffic. |
| /wp-json/ | Keep open | The REST API serves oEmbed data and powers front-end features in many themes and page builders. Block it only if you fully understand the trade. |
| /?s= and /search/ | Block | Internal search results are an infinite space of thin pages and a favorite injection point for search spam. |
| /feed/ and /*/feed/ | Optional | Feeds duplicate every post as XML. Blocking them tidies crawl budget, and RSS readers ignore robots.txt, so subscribers are unaffected either way. |
| /trackback/ | Block | A legacy ping protocol from early blogging that today attracts only spam bots. |
| /cart/, /checkout/, /my-account/ | Block (Woo) | Session-bound WooCommerce pages that are personal to each shopper and should never appear in search results. |
| wp-login.php | Optional | WordPress already marks the login page noindex. A Disallow line is harmless but mostly just trims bot noise from your logs. |
Block bots on WordPress: robots.txt asks, enforcement decides
The aggressive preset writes a Disallow group for ten declared AI crawlers, from GPTBot and ClaudeBot to CCBot and Bytespider. Those bots identify themselves and honor the file, so robots.txt genuinely keeps your WordPress content out of AI training sets. Google-Extended even works as a pure opt-out: blocking it removes your content from Gemini training without touching your Google Search rankings.
But WordPress sites attract a second class of bots that never announce themselves: scrapers cloning your content, brute-force scripts hammering wp-login.php, click bots polluting your analytics. They send fake Chrome user-agents and skip robots.txt entirely. A robots.txt is a polite request. Enforcement means detecting bots by how they behave and stopping them at the server, which is exactly what DevDome Bot Protection does.
WordPress robots.txt FAQ
What is the best robots.txt for WordPress?
Shorter than you think. Block /wp-admin/, allow /wp-admin/admin-ajax.php, list your XML sitemap, and stop there unless you have a specific reason to add more. Add WooCommerce, feed or AI crawler rules only when they solve a problem you actually have. Every extra Disallow line is another chance to block something Google needs to render your pages.
Should I block /wp-includes/ or /wp-content/ in robots.txt?
No. This advice was everywhere ten years ago and it is now actively harmful. Google renders pages like a browser, so it needs the CSS and JavaScript that live in /wp-includes/ and /wp-content/. Blocking them can make your pages look broken to Google and hurt rankings. Google documentation asks site owners not to block these assets.
Why does my robots.txt need to allow admin-ajax.php?
Because blocking /wp-admin/ without an exception also blocks /wp-admin/admin-ajax.php, the endpoint many themes and plugins call from the public site for live search, filters, sliders and forms. Crawlers that cannot reach it may fail to render those features. The Allow line is longer than the Disallow line, so it wins under robots.txt precedence rules and unblocks exactly that one file.
Does WordPress create a robots.txt file automatically?
Yes, a virtual one. If no physical robots.txt exists at your site root, WordPress answers the URL itself with a minimal file: it blocks /wp-admin/, allows admin-ajax.php and lists the core wp-sitemap.xml. You can replace it three ways: upload a physical file, which overrides the virtual one, use the robots_txt filter in code, or use the editor in an SEO plugin like Yoast or Rank Math.
Can robots.txt block bots on WordPress?
Only bots that choose to obey it. GPTBot, ClaudeBot, CCBot and other declared crawlers honor a Disallow, so the aggressive preset in this WordPress robots.txt generator really does keep them out. Scrapers, brute-force scripts and click bots send fake browser user-agents and ignore the file entirely. Stopping those requires server-side detection based on behavior, which is what DevDome Bot Protection does.
Will blocking AI crawlers hurt my Google rankings?
No. The AI preset only names AI training and answer-engine crawlers, never Googlebot or Bingbot. Google-Extended is even designed as a pure opt-out: blocking it removes your content from Gemini training while your position in Google Search stays untouched. Your pages keep ranking exactly as before.
Where does the robots.txt file go on a WordPress site?
In the site root, the same folder as wp-config.php, so it is reachable at yourdomain.com/robots.txt. Crawlers only ever check that exact location. If WordPress lives in a subdirectory like /blog/ but serves your main domain, the file still belongs at the domain root, not inside the subdirectory.
More free tools: Robots.txt Tester · Bot User-Agent Lookup · XML Sitemap Generator · LLMs.txt Generator · Security Headers Checker