Technical SEO crawlers compared: what each one actually finds

The only honest way to compare crawlers is to break a site on purpose, so you know in advance exactly what there is to find. Here is how that test is built, and what it reveals.

The test site

Four thousand pages on a staging domain, seeded with 91 known issues across nine classes: broken canonical chains, orphaned pages, redirect loops, duplicate title clusters, malformed hreflang, blocked resources, thin content, mixed content warnings, and a genuinely nasty pagination trap that produced near-infinite crawl paths.

The exact count is known in advance because every issue is planted deliberately. That is the part most crawler comparisons skip, and it is the only thing that makes the scores mean anything — without a known denominator you are comparing report lengths, not detection.

Detection: three cleared the bar

Three tools found 88 or more of the 91. Two landed in the seventies. The remaining four sat between 51 and 66, and the pattern in what they missed was consistent enough to be worth naming: almost everyone finds broken links and duplicate titles; almost nobody catches a canonical chain that resolves correctly but points somewhere strategically wrong.

The pagination trap was the real separator. Four crawlers walked straight into it and burned their entire budget generating URLs that should never have been crawled. Two detected the pattern and stopped. One flagged it as a finding, which is the correct behaviour and, frankly, the only one that would have saved us money on a real site.

Prioritisation: where the field collapses

This is the interesting result. Detection is close to solved. Prioritisation is not.

The gap between the best and the worst was not features. It was whether the tool told you which of the ninety-one problems was the one worth fixing on Monday morning.

Two tools produced a genuinely ranked action list with a defensible rationale. Three produced severity labels that appeared to be assigned by issue type rather than by actual impact on the site in question — every missing meta description flagged identically, whether on a top-converting page or on page 3,900 of a tag archive.

One produced a 340-page PDF. We want to be fair to it: everything in that document was accurate. It is also, functionally, not a report. Nobody has read it. Nobody will.

What we would tell a team of five

Buy the cheapest tool that produces a ranked list you will actually work through. Depth you never reach is not depth. Most small teams we speak to are over-tooled and under-shipped, and the second problem is usually caused by the first.

What we would tell a team of fifty

Your constraint is not detection either — it is routing. The question worth paying for is which of your six squads owns each finding and whether that assignment happens without a human in the middle. Two tools do this properly. The rest assume someone will triage by hand, and that someone is your bottleneck.

The short version

Stop evaluating crawlers on what they find. Evaluate them on what they make you do. Then run this test on your own site — plant a dozen issues you know about, see what comes back. It takes an afternoon and it will save you a year of subscription drift.