Your SEO ticket says "fixed." The deploy went out. How do you know it actually worked, and that it didn't break something else?
Most SEO audit tools answer with a net count: "Broken internal links: 16 → 17." That number is almost useless for a developer. It could mean one new broken link. It could also mean ten new ones hidden behind nine fixes.
What you want is a diff: for each issue type, which URLs were fixed and which are new. This post shows the logic, a small script you can run on any two crawl exports, and the gotchas we hit doing this on our own site.
The model: issues are sets of URLs
A crawl audit is just a mapping from issue type to the set of URLs that have it:
before["Client error (4xx)"] = {/blog/a, /blog/b, /pricing-old, ...}
after["Client error (4xx)"] = {/pricing-old, ...}
Two set differences give you everything:
- fixed = before minus after
- new = after minus before
Net change (len(after) - len(before)) throws that information away.
A real example
We ran this on siteauditlint.com between two audits on the same day:
| Issue | Before | After | Fixed | New |
|---|---|---|---|---|
| Client error (4xx) | 13 | 11 | 2 | 0 |
| Links to broken internal pages | 16 | 17 | 0 | 1 |
Net change says "one better, one worse." The diff says the fixes worked, and a post published that afternoon introduced exactly one new broken link. That is a ticket a developer can close in five minutes.
The script
Export issue rows from your crawler as CSV with at least an issue column and a url column (most SEO crawlers can do this). Export the full list of crawled URLs too, so you can tell "fixed" apart from "disappeared."
import csv
import sys
from collections import defaultdict
def load_issues(path):
issues = defaultdict(set)
with open(path, newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
issues[row["issue"]].add(row["url"])
return issues
def load_urls(path):
with open(path, newline="", encoding="utf-8") as f:
return {row["url"] for row in csv.DictReader(f)}
before, after = load_issues(sys.argv[1]), load_issues(sys.argv[2])
urls_before, urls_after = load_urls(sys.argv[3]), load_urls(sys.argv[4])
# Scope check: comparisons only mean something if both crawls cover the same site
ratio = len(urls_after) / max(len(urls_before), 1)
if not 0.9 <= ratio <= 1.1:
print(f"WARNING: crawl scope changed ({len(urls_before)} vs {len(urls_after)} URLs)")
for issue in sorted(set(before) | set(after)):
b, a = before.get(issue, set()), after.get(issue, set())
fixed, new = b - a, a - b
gone = {u for u in fixed if u not in urls_after} # left the crawl, not truly fixed
if fixed or new:
print(f"{issue}: {len(b)} -> {len(a)} | fixed {len(fixed) - len(gone)} "
f"| dropped {len(gone)} | new {len(new)}")
for u in sorted(new):
print(f" NEW {u}")
Run it:
python crawl_diff.py issues_before.csv issues_after.csv urls_before.csv urls_after.csv
Three gotchas we hit
1. "Fixed" can mean "not crawled." In our example, one of the two fixed 4xx URLs didn't start returning 200. It dropped out of the crawl because the links pointing to it were removed. Total URLs went from 711 to 710. That still counts as resolved for most purposes, but your report should say which is which. That's what the dropped count is for.
2. Scope changes fake regressions. Compare a 100-URL crawl with a 710-URL crawl and critical issues will "rise" even if nothing broke. We had exactly this in our own history. The 10% scope check above catches it.
3. Settings must match. If the baseline ran with JavaScript rendering on and the new crawl ran with it off, rendered-only links and canonicals will appear as new issues. Same user agent, same robots.txt handling, same limits.
Putting it in the pipeline
The useful version of this runs automatically:
- Save a baseline crawl of production (or staging) as your reference.
- After each deploy, run a headless crawl with the same settings.
- Diff against the baseline and fail or flag the job if any critical issue has new URLs.
- Post the new URLs to your team channel or issue tracker, with the source page and broken target.
Treat it like a regression test: you don't need zero issues, you need zero new critical ones.
Tools that do this for you
If you'd rather not maintain a script:
- Screaming Frog (paid) has a compare mode showing URLs added to and removed from each filter.
- Semrush and Ahrefs show new and fixed or removed issues between scheduled cloud crawls.
- Sitebulb Pro has audit comparison and history.
- SiteAuditLint, which we build, shows fixed and new URLs per issue, labels every issue row New or Existing, supports headless scheduled audits on Windows, and the Agency edition can send issues to Linear and Slack.
We wrote a longer comparison of eight site audit tools, including pricing and JavaScript rendering support: the best SEO audit tools in 2027, compared.
A closed ticket isn't proof. A passing recrawl is.


Top comments (3)
This is a useful distinction to make explicit — net counts really do hide the churn, especially on a deploy where a content change and a template fix land together.
The 'fixed vs dropped' split is the part I'd keep. A URL disappearing because its incoming links were removed is resolved in practice, but it's not the same evidence as a live URL returning the right status and content. Search Console draws a similar line: an issue is treated as fixed for a URL when it's crawled and the issue is no longer found, or when the page is no longer available to Google for some reason. Keeping those two buckets separate in your own diff avoids overstating a fix.
One addition I'd make for the pipeline version: pair this crawl diff with the Search Console side. The crawl tells you what changed on your site right now; validation there is the slower check — Google notes it typically takes up to about two weeks — so a clean diff and a pending validation can both be true at the same time. That's also why the scope check matters: if the baseline crawl and the post-deploy crawl don't cover the same set of URLs, the 'new' list is mostly noise.
Good framing overall — zero new critical issues is a much more workable gate than zero issues.
Fixed vs new per issue is way more useful than the net count, agreed. One gotcha on Next.js sites: streamed metadata can put the canonical in the body for some user agents, so if the crawler UA changes between two runs the same page can "fix" or "break" with no deploy at all. I'd pin the UA for both crawls. Do you store it with each export?
Net counts lying is so true. Have you tried running the same diff against a staging crawl before the deploy goes out? Failing CI when the "new" column isn't empty for internal 4xx links would turn this from a report into a gate.