AI Crawlers Are Already Hitting Your Site: What We Measured

A small store we run logged 11 human visitors and roughly 2,770 bot hits in one 30 day window, so 99.6% of everything that touched the site was automated. The bots also produced about 1,510 click events on that site. The humans produced none. A bigger affiliate site we run told a milder version of the same story in the same period: 9,490 humans against 12,870 bot hits, so about 58% of all hits were machines.

Not all of that is AI, and none of it is unusual. Search crawlers, uptime monitors and vulnerability scanners have always been there. What changed is that a new category joined the queue, it identifies itself politely, and almost nobody has looked at how much of their traffic it now accounts for. Here is who is knocking, what it does to your numbers, and what blocking actually involves.

Who Is Actually Crawling You

The major AI crawlers publish their names, which is more than most bots do. OpenAI documents three: GPTBot for model training, OAI-SearchBot for surfacing sites in ChatGPT search, and ChatGPT-User for pages fetched because a person asked. Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot along the same split. Google separates AI training from Search with Google-Extended.

Others turn up regularly in logs with less documentation, including PerplexityBot, CCBot from Common Crawl, Bytespider and Applebot-Extended. The names churn, so treat any list, this one included, as a snapshot rather than an inventory.

What It Looks Like on a Small Site

The instinct is that this is a big publisher problem. Our own numbers say the opposite: the smaller the human audience, the worse the ratio looks. Bots find you by scanning addresses and following links, so a quiet site gets close to the same baseline of automated attention as a busy one, spread over a fraction of the real traffic.

That is how you end up with 2,770 automated hits against 11 people. Nothing dramatic happened to that store, it simply has very little human traffic to dilute the machines. If you have ever looked at a dashboard for a new site and wondered why a page nobody shares gets steady views, this is usually the answer.

Why Most Analytics Never Show Them

Browser-based analytics miss nearly all of it, because the measurement only exists if JavaScript runs, and most crawlers do not run it. So Google Analytics can show a quiet week while your server handles thousands of automated requests.

Server-side measurement has the opposite failure. It sees every request, which is the honest picture, but it counts crawlers as visitors unless something classifies them. Built-in bot filtering only helps with declared, listed crawlers, so the total drifts upward and you never learn which half was real. We unpacked that gap in why Google Analytics overcounts your visitors.

There is a billing angle too. If your analytics plan meters every event, a crawl wave is something you pay for. DevDome Analytics counts human pageviews only for plan limits, so bot volume never pushes you up a tier. Details are on our pricing page.

What robots.txt Can and Cannot Do

For the documented crawlers, robots.txt is the intended control, and each vendor publishes the token to use. Blocking OpenAI’s training crawler is two lines: User-agent: GPTBot then Disallow: /. Repeat the pair for ClaudeBot, Google-Extended, OAI-SearchBot or any other name you want to exclude. In WordPress, edit the file your SEO plugin manages, or drop a real robots.txt in your web root.

Two caveats. First, robots.txt is a request, not a wall: it works because reputable operators choose to honour it, and a scraper with no reputation to protect simply ignores it. Second, do not confuse this with your search visibility. Googlebot is controlled separately from Google-Extended, and Google states plainly that Google-Extended does not affect inclusion in Search.

While you are editing that file, it is worth confirming crawlers are not fetching each of your pages twice under two addresses, which is exactly what happens when both the slashed and unslashed version of a URL answers with a 200. That one is a five minute fix: see trailing slash SEO and how to redirect it properly.

Blocking for Real

If you want enforcement rather than a polite request, it has to happen before WordPress runs. That means a rule at your host, your firewall or your CDN that rejects the user agent outright, which also saves the PHP and database work a WordPress page normally costs.

The infrastructure layer has moved in this direction on its own. Cloudflare now asks new domains upfront whether AI crawlers should be allowed at all, a shift from opting out to opting in. If your site sits behind a CDN, check what its current default is before you write your own rules, because you may be solving a problem your provider already handles.

One warning about heavy handed blocking: do not block by IP address alone. Vendors publish IP ranges that change, and an IP block can stop a crawler from even reading the robots.txt that would have told it to go away.

Should You Block Them At All?

This is a judgement call, not a best practice. Training crawlers take your content and give nothing back you can measure, and that is a reasonable thing to decline. AI search crawlers are different: they are how your pages get cited in AI answers, and blocking them removes you from that surface the way blocking Googlebot removes you from Search.

A middle position suits most WordPress sites: disallow the training tokens, keep the search and user-initiated ones, and revisit it in six months. What you should not do is block everything reflexively because a bot count looked alarming. Some of those crawlers are the reason people find you.

Measure First, Then Decide

You cannot make this call on someone else’s statistics. Split your own traffic into humans and machines, then look at which crawlers are actually spending your server’s time. DevDome Analytics classifies each hit against six signals, including verified user agent, missing browser profile, automation markers, datacenter origin and scanner behaviour, and keeps both figures visible instead of blending them into one number. The mechanics are in bot traffic vs real visitors, and the wider context is in what percentage of website traffic is bots.

Once the split is on screen the decision gets easy, which is the goal. You see your real audience, you see which machines are worth the bandwidth, and you stop optimising a site for readers who were never there.