Crawl Budget and Bots: What Small Sites Should Actually Worry About

Crawl budget is real, and it is almost certainly not your problem. Google’s own guidance on the subject is explicitly aimed at large sites, and says in plain terms that most site owners do not need to think about it.

What people usually have instead is one of two different problems wearing crawl budget’s name: low-value URLs eating crawl attention, or non-search bots eating server capacity. Both are worth fixing. Neither is crawl budget.

What the Term Actually Describes

Two things combined.

Crawl rate is how fast Googlebot is willing to fetch from your server without degrading it. If your site responds slowly or returns errors, Google backs off.

Crawl demand is how much Google wants from you, based on how popular your pages are and how often they change.

The budget is the interaction of the two. It becomes a constraint when a site has so many URLs that Google’s willingness to fetch cannot keep up with the number of pages that need attention. That threshold is high.

Who Actually Has This Problem

Very large sites. Ecommerce catalogues with faceted navigation producing endless URL combinations. News publishers pushing hundreds of items a day. Sites with millions of programmatically generated pages.

A WordPress blog with three hundred posts is not in that category and will not be for a long time. If your pages are not being indexed, the cause is much more likely to be quality, duplication, or an accidental noindex than a budget ceiling.

That last one is worth checking before anything else, because it is common and completely invisible from the front end. A staging site that went live with search engines discouraged, an SEO plugin setting applied to a whole post type, a robots.txt line copied from somewhere else: any of them stop indexing entirely while your pages load perfectly for visitors. Search Console will tell you in a minute, which is a better use of that minute than reading about crawl budget.

The Problem You Probably Do Have

WordPress generates URLs enthusiastically, and many of them are worth nothing.

Paginated archives that go on for forty pages. Tag pages holding a single post. Author archives on a one-author site. Search result pages that got indexed. Attachment pages for every uploaded image. Date archives nobody links to.

None of this is a budget crisis, but it is genuine waste: crawl attention spent on pages that will never rank, and in some cases thin pages competing with your real content.

The cleanup is standard and useful regardless: noindex the archives that add nothing, disable attachment pages, keep search results out of the index, and make sure your sitemap contains only URLs you actually want ranked.

Bots That Are Not Google

This is where the conflation happens, and it is worth separating cleanly.

Your server gets hit by a lot of automated traffic that has nothing to do with search: AI training crawlers, SEO tool crawlers, scrapers, uptime monitors, security scanners.

They consume bandwidth, CPU and database queries. They can genuinely slow a small site down. What they do not do is consume Google’s crawl budget, because that is a measure of Googlebot’s own activity.

So the question “are these bots hurting my SEO” has a slightly awkward answer: only indirectly, and only if they slow your server enough that Googlebot backs off its crawl rate. That happens, but it is a performance problem with an SEO symptom rather than a crawl budget problem.

Find Out What Is Actually Visiting

Before acting on any of this, look at your own data.

Search Console crawl stats tells you what Googlebot fetched, how often, how fast your server responded and what it got back. That is the only authoritative answer about your crawl relationship with Google, and it is free.

For everything else, your analytics needs to be classifying bots rather than hiding or ignoring them. DevDome Analytics reports bot activity separately from human visitors using six independent signals, so you can see whether a load problem is one aggressive crawler or a general rise, and you can watch whether it is growing. Because plans are metered on human pageviews only, that visibility does not come with a bigger bill attached.

Server logs are the other honest source, showing every request whether or not any script ran.

If You Do Need to Act

Be specific and be careful, in that order.

Identify the actual offender before writing any rule. One badly behaved crawler making thousands of requests an hour is a different problem from broad automated traffic, and it has a narrower fix.

Use robots.txt for well behaved crawlers that respect it, and rate limiting at the server or CDN for those that do not. Never write a rule broad enough to catch a search crawler as collateral, and always verify a claimed Googlebot before deciding it is fake.

Then check Search Console for a week afterwards. If crawl errors appear or crawl rate drops, your rule caught something it should not have, and that costs far more than the bandwidth it saved.

The Short Version

Stop optimising crawl budget. Clean up the low-value URLs your platform generates, because that is worth doing on its own merits. Watch your bot share so you notice when something aggressive arrives. Keep your server fast, because that is the one thing that genuinely affects how much Google is willing to crawl.

That covers the real problems, and it does not require thinking about a budget you do not have.