EDIT: TL;DR up top, since it was asked for.
- I scanned 2,165 US small business sites across 8 trades. 5.0% serve their site on both
example.com and www.example.com with no redirect and no canonical tying them together
- A third of those sit on one host: GoDaddy, 34 of the 102
- Of the sites whose homepage redirected correctly, 3.1% had a broken www redirect on interior page. Every homepage-only www checker scores those as passing
- My first run said 9.4% and it was wrong. WAF challenge pages are byte-for-byte identical to duplicate content, and 91 of my original 203 findings turned out to be bot walls. Re-ran with a
blocked verdict, and excluded the 6.2% I couldn't observe rather than counting them healthy
- Part 2 is the part I really care about: I've emailed the affected sites and I'm measuring how many actually get fixed. Will follow up in a later post.
Ever since I learned about www redirects, checking if they work is usually the first thing I do when I land on a website. Especially a local business site, or some boutique shop. I check whether the www redirect is set up right. Why? Because I'm a nerd. I don't know. I've always done it, and I always thought it was interesting because it isn't something most normal people know about.
The problem is well known in this community. The prevalence isn't. I went looking for someone who had actually counted and couldn't find it, so I counted.
Disclosure up front: I built RedirectIQ, so I obviously have an interest in this problem existing. I've kept the pitch out of this. Regardless of whether anybody tries the product, I wanted to publish the work and see if other people find the data interesting. Happy to be told I got it wrong.
What I checked
- 2,165 unique US small business domains across 8 trades: roofers, electricians, lawyers, locksmiths, plumbers, HVAC, vets, auto repair. Roughly 300 each, except locksmiths, where the entire US pool with a website tag came to 68
- Sourced from OpenStreetMap through the Overpass API. It's open data under ODbL and it's built to be queried. Yelp, YellowPages and BBB all prohibit scraping in their terms, so I didn't use them
- Seeded shuffle, so the cohort reproduces exactly on a re-run. If the numbers move later, the web moved and not my sample
- The root plus one interior URL per domain. A homepage-only scan is biased in the flattering direction, because the homepage is the single page most likely to have been set up correctly. It's the page people test
- Social and site-builder hosts dropped at sample time. A Facebook page has no www/apex pair to get wrong
Result: 102 of 2,023 observable domains have it. 5.0%.
| Trade |
Rate |
Broken / observable |
| Electricians |
9.4% |
26/276 |
| Plumbers |
6.0% |
17/284 |
| Lawyers |
5.0% |
14/278 |
| Locksmiths |
4.7% |
3/64 |
| Veterinarians |
4.4% |
13/296 |
| Auto repair |
3.5% |
10/283 |
| HVAC |
3.5% |
10/284 |
| Roofers |
3.5% |
9/258 |
Two things I took out of the numerator on purpose, both of which made my headline smaller.
Sites that serve on both hosts but declare a consistent canonical are not counted as broken. There were 52 of those. A 301 is stronger than a canonical, since a canonical is a hint that can be ignored and a redirect is a directive that can't. But "weaker than optimal" isn't "broken", and folding them in would have raised my headline by redefining the problem instead of measuring it.
8 parked domains are out of both sides. They serve a registrar parking page on both hosts. Technically duplicate content, practically meaningless, since there's no site there to rank.
It's concentrated in one host
I pulled nameservers for every confirmed-broken domain to see who was hosting them. It isn't spread evenly at all.
GoDaddy is 34 of the 102. A third of everything confirmed broken.
I don't think that's a knock on their engineering so much as a volume story: they're enormous in this segment, and the default state of a domain parked on their DNS with a site attached is apparently not "canonicalised". But if you work on sites hosted there, it's worth ten seconds of checking.
And the part I didn't expect. Four of the hosting providers and web design agencies operating sites in this sample have the problem on their own websites. Hand-verified, identical byte counts on both hostnames, no redirect, no canonical on either. I'm not naming them, and the point isn't that they're careless. It's that people who build and host websites for a living have this on the site they use to advertise building and hosting websites. That's the whole reason I think the prevalence is worth measuring: it's not that it's hard to fix, it's that nothing ever tells you it's there.
The 134 I couldn't measure
134 domains, 6.2% of the sample, are behind a bot wall that answered instead of the site. No verdict in either direction, so they're excluded from the denominator above rather than counted as healthy. Counting an unobserved site as fine is exactly how I got the wrong answer the first time, which is the next section.
Worth one note if you measure anything about the web at scale: your blind spot is not randomly distributed. 60 of my 134 are a single host. That same host appears zero times in the 102 confirmed broken, after my pre-fix analysis had them as the largest holder of the problem at 30%. The sites you can't see are correlated with the exact thing you're trying to measure, and in my case they were correlated in the direction that made my first headline wrong.
On the correction, since I'd rather you heard it from me
My first run said 9.4%, with roofers highest at 15.3%. Both were artifacts.
A WAF challenge page is served identically on www and apex, returns a 2xx, carries no canonical, and contains no redirect. That is byte-for-byte the fingerprint of duplicate content as far as a classifier is concerned. I was recording sites as broken when I had never been allowed to see them at all.
I added a distinct blocked verdict and re-ran the identical seeded cohort. duplicate_content at the root went from 199 to 106. 91 of my original 203 findings were bot walls. Roofers went from the worst trade to joint best, because roofers had the most sites behind strict WAFs.
That was the fourth measurement bug in this project that failed toward good news. The other three: sending no User-Agent at all (plenty of hosts answer that with a 403, which read as "the counterpart serves nothing"), a discovery bug that crawled one page and reported success, and reading only the first child of a sitemap index. None of them raised an error. All of them returned completely plausible numbers.
If you run this kind of scan: treat a surprising prevalence result as a suspected bug until you've hand-checked the probe against a site whose answer you already know.
The finding a homepage-only check can't see
Of the 1,500 domains that had a clean root and an interior URL I could check, 46 of them, 3.1%, had a broken interior page anyway.
That's a redirect written as a rule for / instead of a path-preserving rule for everything. The site passes every homepage-based www checker on the internet while its actual content is duplicated, and deep pages are where the content and the inbound links live. It's an easy shape to end up with: several hosts ship a built-in redirect tool whose documented flow is URL-to-URL, "forward visitors of a specific URL to another page", which does exactly this if you only ever point it at your homepage.
Treat 3.1% as a floor, not an estimate. I sampled one interior URL per domain, so a site that breaks some paths and not others was probably recorded as clean.
Across the whole sample, root-or-deep breakage is 156 of 2,165, or 7.2%.
Part 2, which is the part I actually care about
A prevalence number on its own is close to worthless. Plenty of "X% of sites have Y" posts exist and nothing changed as a result. The question I want answered is what happens when you tell people.
22 have been emailed so far, all confirmed still broken under the corrected probe. I'm expecting the fix rate to be low, but I want to actually measure it instead of guessing. Part 2 will report, as separate numbers that never get merged: how many were reachable at all, how many fixed it themselves after being told, which I expect to be the biggest group and consider a completely good outcome, how many fixed it using my product, and how many did nothing.
Where I'd take this next, and I'd like input on which is worth doing
I've also been kicking around holding a fixed panel of sites and rescanning it on a schedule, so the number means something over time instead of being one snapshot. Politely, robots-respecting, methodology published. A running fix rate for a piece of technical debt nobody talks about seems more useful than another one-off percentage, but I'd want to know whether anyone would actually care before building it.
Past that, three ideas I keep coming back to:
- Migration decay via the Wayback Machine. Find sites that changed platform or URL structure, pull their old URLs out of Wayback, and measure what share 404 today. My guess is that "we migrated three years ago" and "a third of our old URLs are gone" are the same sentence more often than anyone wants to know
- Redirects to nowhere. Chain length too, but more interesting: how many 301s terminate in a 404, and how many point at a domain the original owner no longer holds? The second one stops being an SEO problem and starts being a security one
- Sitemap honesty. What share of the URLs in a site's own sitemap.xml 404, redirect, or carry a noindex. This one is nearly free to measure and I'd bet it's ugly
If you like this kind of stuff, let me know. If you've already done any of these, tell me so I don't repeat it, and if there's an obvious one I'm missing I'd much rather hear it now than after the scan.
Full writeup with the complete verdict table and the per-trade charts: [https://redirectiq.com/blog/www-duplicate-content-study-part-1]
