Scraper Bots Are Copying Your Content: How to Stop Them

You cannot stop every scraper, so aim to make your site expensive to copy instead of impossible, and work in this order: see the bots in your traffic, ask the polite ones to leave with robots.txt, throttle or block the rest at your host or CDN, make the content itself less worth taking, and only then chase down copies. Skipping straight to blocking is what breaks sites, usually by accidentally locking out a search engine.

Scraping is not rare or targeted, either. It is background weather on the open web, and most site owners only notice when a copy of their post turns up in a search result.

What Scrapers Actually Take

A scraper is just an automated client that fetches your pages fast and republishes what it finds, sometimes on a spam blog, sometimes to feed a price comparison site, sometimes into a dataset. It does not need to break in. Your content is already public, and a script can read it as easily as a reader can.

Certain pages attract more of it than others. Product roundups, comparison tables and anything with prices in it get copied constantly, which creates a second problem for affiliates: a copied price is a frozen price, and it is wrong within a day. If you publish product prices at all, the rules on showing Amazon prices are worth reading, because they apply to your own pages long before anyone copies them.

See Them Before You Block Them

Blocking blind is how people lock out Googlebot and lose rankings. Start by finding out what is actually hitting you: which pages, how fast, from which networks, and whether the requests report a real browser, operating system and device. Anything fetching hundreds of pages an hour with no device information is not a reader.

The scale usually surprises people. Across our own sites, one affiliate site logged 9,490 human visitors against 12,870 bot hits in a 30-day window, so about 58% of all hits were automated, and a small store in the same period recorded 11 humans against roughly 2,770 bot hits. Not all of that is scraping, but none of it is an audience. DevDome Analytics classifies every request server side using six signals, including whether the client is a verified crawler, reports no browser profile, shows automation, uses a script user agent, comes from a datacenter, or matches a known scanner, and it shows the human and bot columns separately instead of blending them. The bot traffic vs real visitors guide covers reading those signals by hand if you would rather.

Robots.txt Only Stops the Polite Ones

Add rules to robots.txt for the crawlers you do not want, and the well-behaved ones will respect them. Google is explicit that robots.txt cannot enforce anything: the instructions are a request, and it is up to each crawler to obey. A scraper built to take your content will simply ignore the file, and some read it as a map of what you consider valuable.

So use robots.txt for what it is good at, which is managing crawl load from identified, cooperative bots. It is the first step because it is free and takes two minutes, not because it solves the problem.

Block and Slow the Rest at the Edge

The real enforcement lives in front of WordPress, in your CDN or host controls. Three settings do most of the work: a rate limit that allows normal reading but not hundreds of requests a minute from one source, a challenge for traffic arriving from datacenter networks rather than consumer connections, and a block list for user agents that identify themselves as scripting libraries.

Two cautions. Verify before you block anything claiming to be a search engine, because Google documents that its crawlers can be confirmed by user agent, IP address and reverse DNS, and impersonating Googlebot is a standard scraper trick. And keep limits generous, since a rate limit set too tight blocks real readers first, usually on the pages that matter most.

Make the Copy Less Useful

Some of the best defences are editorial. Publish summaries rather than full posts in your RSS feed, because full-text feeds are the single easiest way to automate republishing. Put internal links inside your body copy, since scrapers usually keep them, so every copy quietly links back to you. Keep a clear publication date and make sure new posts get indexed quickly, which is what settles authorship questions in practice.

None of this stops a determined copy. It changes the economics, which is the realistic goal: the scraper gets an excerpt full of links to your site, and the copy reads as the derivative it is.

Getting a Copy Taken Down

When a copy does appear, act in order rather than in anger. Contact the site, then its host, then the search engines through the relevant legal removal process, keeping evidence of your original publication date at each step. Copyright removal requests are handled by hosts and search engines every day, so a well-documented one usually works.

Keep the effort proportional. Google’s spam policies already list scraped content as a violation, and most copies never rank, so this is housekeeping rather than a crisis. Spend the time on the copies that actually appear in search results for your own titles.

Measure Whether Any of It Worked

Give your changes two weeks and compare the bot share before and after, not just your traffic total. The direction you want is a falling bot count with an unchanged human count. If humans dropped, a rule is too aggressive and should be loosened immediately.

This is also where blended analytics fail you, because a tool that counts machines as people will show your “traffic” falling after every successful block. When human and bot numbers are separate, the picture is obvious. DevDome plans are metered on human pageviews only, so scrapers never eat your allowance while you work through the list, and you can see the plans on our pricing page. For the wider context on how much of the web is automated, see what percentage of web traffic is bots and why Google Analytics shows too many visitors.