Audit a whole site

Give us a domain. We read its sitemap and crawl it, auditing every page we reach, and report the findings grouped by defect rather than by page.

Why grouping by template matters more than counting pages

A site is not a pile of pages, it is a handful of templates rendered many times over. When a product template writes the pre-discount price into offers.price, that is one defect — but a page-by-page report shows it four thousand times and buries the three genuinely one-off problems underneath.

So we group. Each finding says how many of the sampled pages carry it and which templates are affected, and anything appearing on most of the sample is marked template-wide and sorted to the top. Those are the fixes worth doing first, because each one is a single change to a single file.

The order pages are crawled in

Not sitemap order. We group every discovered URL by its path shape — /products/blue-kettle and /products/red-kettle are both /products/{slug} — and then work round-robin across those groups, largest first.

A crawl that runs to completion reaches everything either way. A crawl that stops at its budget has, this way round, covered the product template, the article template and the contact page rather than the first few hundred pages of one section. Since most crawls stop somewhere, the ordering is what decides whether a partial result is useful.

How we behave while crawling

Three pages at a time, with a pause between requests. If the site’s robots.txt states a Crawl-delay we drop to one at a time and wait as long as it asks — a crawler that reads that directive and ignores it is worse than one that never looked. Disallowed paths are skipped, checked per URL rather than once for the site, because a site can disallow a section and a crawler that only checks the home page ignores that.

There is a page budget and a twenty-minute wall-clock limit. Neither is a technical ceiling — the crawler goes as far as it is configured to — but someone else’s server is paying for every request, so the number is deliberate rather than absent. You can stop a crawl at any point and keep the results gathered so far.

What a partial crawl does and does not tell you

Results appear while the crawl runs, and every count is against the pages audited to that point. A finding marked template-wide at page fifty may settle differently by page five hundred, which is why the report shows pages-seen-on rather than a percentage presented as a fact about the site.

If the crawl stops at its budget, it has covered a breadth of templates but not every page. A defect on three pages out of four thousand is exactly the kind of thing only a complete crawl finds — raise the budget when that is what you are looking for.

For a single page in full detail, with visibility state and evidence on every value, use the page checker.