Click depth is the smallest number of link clicks it takes to get from your home page to a page. The home page is depth 0, anything in the main nav is depth 1, and a blog post that only appears on page 4 of the blog archive might be depth 5 or 6. Nobody plans for a page to end up six clicks deep. It happens when a blog grows one "Next page" link at a time, when docs get nested one level per release, or when the only link to an important page is a button that is not really a link.
This post builds a click depth auditor in Python using only the standard library (urllib, urllib.robotparser, html.parser, xml.etree, csv, concurrent.futures). It reads your sitemap or sitemap index, crawls your site breadth first from the home page, and records the shortest click path to every page. It flags sitemap URLs deeper than a threshold you choose (3 by default), marks pages that can only be reached by paging through an archive, and writes a CSV with the full path for each URL, plus a console summary by depth and by site section and an exit code you can use in CI.
Google Search Central's Help Google understand your ecommerce website structure, SEO Starter Guide and Pagination, incremental page loading, and their impact on Google Search (all three showing "Last updated 2025-12-10 UTC") were opened on 15 Nov 2026, and every Google statement below comes from those three pages. Appie free-tool descriptions are as of 15 Nov 2026, checked on the tool pages that day. The script was tested on Python 3.13.5 against a local fixture site with planted deep pages; the output below is real, from runs on 15 Nov 2026.
Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.
Where this fits
This is another script in a small standard-library series. The previous one found orphan pages by comparing sitemap URLs with internal links: pages nothing links to at all. An earlier post found broken links and redirect chains. This one asks the next question for the pages that are linked: how far from the home page are they, which path gets there, and is that path a real navigation route or a long walk through archive pages?
What Google actually says (and what it does not)
The most direct statement is on the ecommerce site structure page, and it applies to any site with a lot of pages: "Google can use information such as the number of links it needs to follow to reach a page and the number of links to a page to infer the relative importance of a page over the rest of your site." That is click depth and inbound links, named together.
The same page says that structure comes from links, not from URLs: "Google generally doesn't look at the structure of URLs to work out the structure of a site. Instead, it analyzes the linkages between pages to gain insights about the relative importance of different pages on a site." So /docs/getting-started/install/proxy/sso is not deep because it has five folders in its path. It is deep only if it takes five clicks to reach. The script measures clicks and ignores folder depth.
It also gives the fix for an important page: "if you have a best selling product, consider linking to it from the home page or in other content, such as blog posts or newsletters on your site." And it warns about pages that can only be reached through a search box: "Googlebot generally doesn't try to submit searches into a search box as part of crawling a site."
The pagination page explains why archives push pages down. Its first best practice is to "include links from each page to the following page using <a href> tags," and it suggests "linking from all individual pages in a collection back to the first page of the collection." It also notes that Google's crawlers "don't 'click' buttons and generally don't trigger JavaScript functions that require user actions," which is why a "Load more" button can hide everything after the first batch. The page says Google "no longer uses" rel="next" and rel="prev" link tags, so the script ignores them and follows only <a href> links.
The SEO Starter Guide adds a reality check before you start moving pages around: "Don't drop everything and start reorganizing your site right now though: while these suggestions can be helpful long term (especially if you're working on a larger website), search engines will likely understand your pages as they are right now, regardless of how your site is organized." Its internal linking advice is "Every page you care about should have a link from at least one other page on your site," and "There's no magical ideal number of links a given page should contain."
What none of these pages give is a maximum click depth. "Three clicks" is a common rule of thumb in site architecture, not a Google number. The script uses 3 as a default because it is a reasonable line for "a visitor can get here from the nav without hunting," and the flag is --max-depth so you can set your own. Treat a deep page as a question to answer (does anyone need to reach this?), not as an error.
How the crawl works
-
Sitemap URLs. It reads a
urlsetor a sitemap index (recursively, gzip supported) and keeps every<loc>. These are the pages the report is about. -
Breadth-first crawl. It fetches the home page (or any
--startURLs), then every same-host page linked from it, then every page linked from those, one level at a time. Because each level is finished before the next one starts, the first time the crawl sees a URL is its shortest click distance. That is the main reason to use breadth first: a depth-first crawl would report whatever path it happened to take first. -
What counts as a link. Only
<a href>links on the same host. Links withrel="nofollow"and links on pages whose robots meta saysnofolloware not followed, but nofollow links are counted so the report can explain why a page was never reached.<base href>is honoured.robots.txtis respected. Links to files such as PDFs, images or.txtcount as reached but are not fetched. -
Redirects. A link to a URL that redirects is credited to the final URL, at the depth of the link. A sitemap URL that redirects is reported as
redirectedand counted under its target. -
Shortest paths. After the crawl, every link from a page at depth
dto a page at depthd + 1is a shortest-path link. The script stores how many there are (shortest_paths_in) and prints one full path per URL, for example/ > /blog > /blog/page/2 > /blog/post-7. -
Pagination. A URL matching
/page/Nor apage=,p=,pg=orpaged=query parameter is treated as a paginated listing. A page isonly_via_paginationwhen every one of its shortest paths runs through one. Those are the pages a visitor only finds by clicking "Next" again and again. -
Matching. URLs are compared after removing fragments, lowercasing scheme and host, dropping default ports and ignoring a trailing slash.
--drop-queryalso ignores query strings.
What the script flags
-
Flag:
deeper_than_N; Meaning: Shortest path from the start page is more than--max-depthclicks; What to check: Does this page matter enough to link from a hub, a category page or the nav? -
Flag:
only_via_pagination; Meaning: Every shortest path goes through page 2, 3, ... of a listing; What to check: Add numbered page links, category or topic hubs, or related-post links -
Flag:
not_reached; Meaning: In the sitemap but never reached from the start page; What to check: Run an orphan check; thenofollow_inlinkscolumn shows whether nofollow links are the reason -
Flag:
redirected; Meaning: The sitemap URL redirects; What to check: List the final URL in the sitemap instead -
Flag:
http_error; Meaning: The page returned something other than 200; What to check: Fix it or remove it from the sitemap -
Flag:
not_in_sitemap; Meaning: A crawled HTML page the sitemap does not list (CSV only); What to check: Informational; paginated archive pages often land here
Exit codes: 0 when no sitemap URL has an issue other than redirected, 1 when at least one is too deep, only reachable through pagination, unreached or erroring, and 2 when the sitemap cannot be read.
The script
Save as click_depth.py:
#!/usr/bin/env python3
"""Measure click depth from the home page for every URL in a sitemap.
Standard library only. Usage:
python3 click_depth.py https://example.com/sitemap.xml --max-depth 3 --out depth.csv
Exit codes: 0 = nothing deeper than --max-depth, 1 = deep or unreached pages, 2 = sitemap unreadable.
"""
import argparse
import csv
import gzip
import re
import sys
import time
import urllib.error
import urllib.request
import urllib.robotparser
import xml.etree.ElementTree as ET
from collections import Counter, defaultdict
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit, urlunsplit
UA = "click-depth-audit/1.0 (+python stdlib)"
SKIP_EXT = (".pdf", ".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".zip",
".mp4", ".mp3", ".css", ".js", ".xml", ".txt", ".json", ".ico")
PAGINATION = re.compile(r"(/page/\d+/?$)|([?&](page|p|pg|paged)=\d+)", re.I)
def norm(url, drop_query=False):
"""Comparable form of a URL: no fragment, lowercase host, no default port,
no trailing slash (except the root)."""
s = urlsplit(url.strip())
scheme, host = s.scheme.lower(), (s.hostname or "").lower()
if s.port and not ((scheme == "http" and s.port == 80) or (scheme == "https" and s.port == 443)):
host = f"{host}:{s.port}"
path = s.path or "/"
if len(path) > 1:
path = path.rstrip("/")
return urlunsplit((scheme, host, path, "" if drop_query else s.query, ""))
def fetch(url, timeout=15):
"""Return (status, final_url, content_type, body_bytes). Follows redirects."""
req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
body = r.read()
if r.headers.get("Content-Encoding") == "gzip":
body = gzip.decompress(body)
return r.status, r.geturl(), r.headers.get("Content-Type", ""), body
except urllib.error.HTTPError as e:
return e.code, e.geturl() or url, "", b""
except Exception as e: # DNS, timeout, TLS
return None, url, str(e), b""
def read_sitemap(url, seen=None):
"""All <loc> values from a urlset or sitemap index (recursive, .gz ok)."""
seen = seen if seen is not None else set()
if url in seen:
return []
seen.add(url)
status, _, _, body = fetch(url)
if status != 200:
raise ValueError(f"{url} returned {status}")
if body[:2] == b"\x1f\x8b":
body = gzip.decompress(body)
root = ET.fromstring(body)
locs = [el.text.strip() for el in root.iter() if el.tag.endswith("loc") and el.text]
if root.tag.endswith("sitemapindex"):
out = []
for child in locs:
out.extend(read_sitemap(child, seen))
return out
return locs
class LinkParser(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True)
self.links, self.nofollow_page, self.base = [], False, None
def handle_starttag(self, tag, attrs):
a = {k.lower(): (v or "") for k, v in attrs}
if tag == "base" and a.get("href"):
self.base = a["href"]
elif tag == "meta" and a.get("name", "").lower() in ("robots", "googlebot"):
if "nofollow" in a.get("content", "").lower() or "none" in a.get("content", "").lower():
self.nofollow_page = True
elif tag == "a" and a.get("href"):
rel = a.get("rel", "").lower().split()
self.links.append((a["href"], "nofollow" in rel))
def crawl(starts, host, args, robots):
"""Breadth-first crawl, one depth level at a time, so the first time a page
is seen is its shortest click distance from the start page(s)."""
depth, parents, edges, info = {}, defaultdict(set), [], {}
nofollow_in = Counter()
alias = {} # normalized requested URL -> normalized final URL (redirects)
frontier = []
for s in starts:
k = norm(s, args.drop_query)
depth[k] = 0
frontier.append(k)
level = 0
blocked = set()
def visit(key):
status, final, ctype, body = fetch(key)
return key, status, final, ctype, body
with ThreadPoolExecutor(max_workers=args.workers) as pool:
while frontier and level <= args.max_crawl_depth and len(info) < args.limit:
batch = frontier[: args.limit - len(info)]
nxt = []
for key, status, final, ctype, body in pool.map(visit, batch):
fkey = norm(final, args.drop_query)
if fkey != key:
alias[key] = fkey
if fkey in info: # redirect target already crawled
continue
depth[fkey] = min(depth.get(fkey, level), level)
info[fkey] = {"status": status, "html": "html" in ctype.lower()}
if status != 200 or "html" not in ctype.lower():
continue
p = LinkParser()
p.feed(body.decode("utf-8", "replace"))
if p.nofollow_page:
continue
base = urljoin(final, p.base) if p.base else final
for href, nofollow in p.links:
if href.startswith(("mailto:", "tel:", "javascript:", "#")):
continue
target = urljoin(base, href)
t = urlsplit(target)
if t.scheme not in ("http", "https") or (t.hostname or "").lower() != host:
continue
tkey = norm(target, args.drop_query)
tkey = alias.get(tkey, tkey)
if nofollow:
nofollow_in[tkey] += 1
continue
edges.append((fkey, tkey))
if tkey in depth:
continue
if robots and not robots.can_fetch(UA, target):
blocked.add(tkey)
continue
depth[tkey] = level + 1
if t.path.lower().endswith(SKIP_EXT):
info[tkey] = {"status": None, "html": False}
continue
nxt.append(tkey)
if args.delay:
time.sleep(args.delay)
frontier, level = sorted(set(nxt)), level + 1
# Redirect sources share the depth of their target; resolve edges through aliases.
edges = [(alias.get(a, a), alias.get(b, b)) for a, b in edges]
for a, b in edges:
if a != b and a in depth and b in depth and depth[a] + 1 == depth[b]:
parents[b].add(a)
return depth, parents, edges, info, alias, blocked, nofollow_in
def is_pagination(url):
return bool(PAGINATION.search(url))
def build_report(sitemap, depth, parents, edges, info, alias, nofollow_in, args):
inlinks = Counter(b for a, b in set(edges) if a != b)
# A page is "only via pagination" when every shortest path to it goes
# through a paginated listing (page 2, 3, ...).
only_pag = {}
for key in sorted(depth, key=lambda k: depth[k]):
ps = parents.get(key)
if not ps:
only_pag[key] = False
continue
only_pag[key] = all(is_pagination(p) or only_pag.get(p, False) for p in ps)
def path_of(key):
chain, cur, guard = [key], key, 0
while parents.get(cur) and guard < 100:
cur = sorted(parents[cur])[0]
chain.append(cur)
guard += 1
return " > ".join(urlsplit(c).path or "/" for c in reversed(chain))
in_map = {norm(u, args.drop_query) for u in sitemap}
rows = []
keys = list(dict.fromkeys([norm(u, args.drop_query) for u in sitemap] +
[k for k, v in info.items() if v["html"] and v["status"] == 200]))
for key in keys:
final = alias.get(key, key)
d = depth.get(final)
st = info.get(final, {}).get("status")
issues = []
if final != key:
issues.append("redirected")
if d is None:
issues.append("not_reached")
elif d > args.max_depth:
issues.append(f"deeper_than_{args.max_depth}")
if d is not None and only_pag.get(final):
issues.append("only_via_pagination")
if st is not None and st != 200:
issues.append("http_error")
if key not in in_map:
issues.append("not_in_sitemap")
rows.append({
"url": key,
"in_sitemap": "yes" if key in in_map else "no",
"status": st if st is not None else "",
"depth": "" if d is None else d,
"shortest_paths_in": len(parents.get(final, ())),
"inlinks": inlinks.get(final, 0),
"nofollow_inlinks": nofollow_in.get(final, 0),
"section": "/" + (urlsplit(key).path.strip("/").split("/")[0] if urlsplit(key).path.strip("/") else ""),
"path": path_of(final) if d is not None else "",
"issues": ";".join(issues),
})
rows.sort(key=lambda r: (r["depth"] == "", -(r["depth"] or 0) if r["depth"] != "" else 0, r["url"]))
return rows
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("sitemap")
ap.add_argument("--start", action="append", help="start URL(s); default is the site root")
ap.add_argument("--max-depth", type=int, default=3, help="flag pages deeper than this (default 3)")
ap.add_argument("--max-crawl-depth", type=int, default=25, help="stop crawling past this level")
ap.add_argument("--limit", type=int, default=5000, help="max pages to fetch")
ap.add_argument("--workers", type=int, default=4)
ap.add_argument("--delay", type=float, default=0.0, help="seconds to pause between levels")
ap.add_argument("--drop-query", action="store_true", help="treat ?a=1 and ?a=2 as one page")
ap.add_argument("--no-robots", action="store_true", help="ignore robots.txt (your own staging only)")
ap.add_argument("--out", default="click_depth.csv")
args = ap.parse_args()
try:
sitemap = read_sitemap(args.sitemap)
except Exception as e:
print(f"Could not read sitemap: {e}")
return 2
s = urlsplit(args.sitemap)
root = f"{s.scheme}://{s.netloc}/"
host = (s.hostname or "").lower()
starts = args.start or [root]
robots = None
if not args.no_robots:
robots = urllib.robotparser.RobotFileParser(urljoin(root, "/robots.txt"))
try:
robots.read()
except Exception:
robots = None
print(f"Sitemap: {len(sitemap)} URLs from {args.sitemap}")
depth, parents, edges, info, alias, blocked, nofollow_in = crawl(starts, host, args, robots)
rows = build_report(sitemap, depth, parents, edges, info, alias, nofollow_in, args)
fetched = sum(1 for v in info.values() if v["status"] is not None)
print(f"Crawled {fetched} pages breadth first from {', '.join(starts)}")
if blocked:
print(f"Skipped {len(blocked)} URLs disallowed by robots.txt")
in_map = [r for r in rows if r["in_sitemap"] == "yes"]
# Redirecting sitemap URLs are reported, but counted under their target.
direct = [r for r in in_map if "redirected" not in r["issues"]]
hist = Counter(r["depth"] for r in direct if r["depth"] != "")
print("\nClick depth of sitemap URLs:")
for d in sorted(hist):
flag = " <- deeper than %d" % args.max_depth if d > args.max_depth else ""
print(f" depth {d:>2}: {hist[d]:>4} {'#' * min(hist[d], 40)}{flag}")
unreached = sum(1 for r in direct if r["depth"] == "")
if unreached:
print(f" not reached: {unreached:>2}")
sections = defaultdict(list)
for r in direct:
if r["depth"] != "":
sections[r["section"]].append(r["depth"])
print("\nBy section (sitemap URLs reached):")
w = max([len(x) for x in sections] + [7]) + 2
print(f" {'section':<{w}}{'pages':>6}{'max':>5}{'deep':>6}")
for sec in sorted(sections, key=lambda x: (-max(sections[x]), x)):
ds = sections[sec]
print(f" {sec:<{w}}{len(ds):>6}{max(ds):>5}{sum(d > args.max_depth for d in ds):>6}")
flagged = [r for r in in_map if r["issues"]]
print(f"\nFlagged sitemap URLs ({len(flagged)}), deepest first:")
for r in flagged:
extra = f" ({r['nofollow_inlinks']} nofollow links)" if r["nofollow_inlinks"] and r["depth"] == "" else ""
print(f" {str(r['depth']) or '-':>2} {r['issues']:<34} {urlsplit(r['url']).path or '/'}{extra}")
extra_pages = sum(1 for r in rows if r["in_sitemap"] == "no")
if extra_pages:
print(f" (+{extra_pages} crawled pages not in the sitemap; see the CSV)")
with open(args.out, "w", newline="", encoding="utf-8") as fh:
w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()) if rows else ["url"])
w.writeheader()
w.writerows(rows)
print(f"\nReport: {args.out}")
benign = {"", "redirected", "not_in_sitemap"}
bad = any(set(r["issues"].split(";")) - benign for r in in_map)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
Testing it against a local fixture
I built a small site for a made-up field data app, "Fieldnote," with a sitemap index pointing at two child sitemaps (35 URLs). Every page shares a header nav (Home, Features, Pricing, Blog, Docs, About) and a footer with a Privacy link and a Careers link marked rel="nofollow". The planted problems:
-
The blog has 12 posts, three per listing page, and the listing pages only link to the next page (
/blogto/blog/page/2to/blog/page/3to/blog/page/4). The exception is/blog/post-1, which the features page also links to. -
The docs nest one level per page:
/docsto/docs/getting-startedto/installto/install/proxyto/install/proxy/sso. -
/pricing/enterpriseis linked from the pricing page only through<a onclick="go('/pricing/enterprise')">, with nohref. The only real link to it is at the bottom of the SSO docs page. -
Case studies: the home page features
/customers/acme, the customers index is only linked from the about page, and/customers/initechis only linked from the Globex case study. -
/old-integrationsis in the sitemap and returns a 301 to/integrations; the home page links to the old URL. -
/careersis only linked through the nofollow footer link, and/webinar-2025is in the sitemap with no links to it. -
robots.txtdisallows/admin/, which the home page links to.
The fixture is generated by a short script, and served with a handler that maps clean URLs to .html files and answers the one redirect:
"""Serve ./site with clean URLs (/pricing -> pricing.html) and one redirect."""
import functools
import http.server
import os
REDIRECTS = {"/old-integrations": "/integrations"}
class Handler(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
path = self.path.split("?")[0].split("#")[0]
if path in REDIRECTS:
self.send_response(301)
self.send_header("Location", REDIRECTS[path])
self.end_headers()
return
path = path.rstrip("/") or "/"
if os.path.isdir("site" + path):
path = path.rstrip("/") + "/index.html"
elif not os.path.splitext(path)[1]:
path += ".html"
self.path = path
super().do_GET()
def log_message(self, *args):
pass
http.server.ThreadingHTTPServer(
("127.0.0.1", 8794),
functools.partial(Handler, directory="site")).serve_forever()
Run the server, then from another terminal:
python3 click_depth.py http://127.0.0.1:8794/sitemap.xml --out click_depth.csv
echo "exit code: $?"
Real output on Python 3.13.5:
Sitemap: 35 URLs from http://127.0.0.1:8794/sitemap.xml
Crawled 35 pages breadth first from http://127.0.0.1:8794/
Skipped 1 URLs disallowed by robots.txt
Click depth of sitemap URLs:
depth 0: 1 #
depth 1: 8 ########
depth 2: 9 #########
depth 3: 5 #####
depth 4: 5 ##### <- deeper than 3
depth 5: 3 ### <- deeper than 3
depth 6: 1 # <- deeper than 3
not reached: 2
By section (sitemap URLs reached):
section pages max deep
/pricing 2 6 1
/blog 13 5 5
/docs 6 5 2
/customers 4 4 1
/integrations 3 2 0
/about 1 1 0
/features 1 1 0
/privacy 1 1 0
/ 1 0 0
Flagged sitemap URLs (15), deepest first:
6 deeper_than_3 /pricing/enterprise
5 deeper_than_3;only_via_pagination /blog/post-2
5 deeper_than_3;only_via_pagination /blog/post-3
5 deeper_than_3 /docs/getting-started/install/proxy/sso
4 deeper_than_3;only_via_pagination /blog/post-4
4 deeper_than_3;only_via_pagination /blog/post-5
4 deeper_than_3;only_via_pagination /blog/post-6
4 deeper_than_3 /customers/initech
4 deeper_than_3 /docs/getting-started/install/proxy
3 only_via_pagination /blog/post-7
3 only_via_pagination /blog/post-8
3 only_via_pagination /blog/post-9
1 redirected /old-integrations
- not_reached /careers (35 nofollow links)
- not_reached /webinar-2025
(+3 crawled pages not in the sitemap; see the CSV)
Report: click_depth.csv
exit code: 1
The CSV has 38 rows: the 35 sitemap URLs plus three crawled archive pages the sitemap does not list. These are the first ten rows and the last two, sorted deepest first:
url,in_sitemap,status,depth,shortest_paths_in,inlinks,nofollow_inlinks,section,path,issues
http://127.0.0.1:8794/pricing/enterprise,yes,200,6,1,1,0,/pricing,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy > /docs/getting-started/install/proxy/sso > /pricing/enterprise,deeper_than_3
http://127.0.0.1:8794/blog/post-2,yes,200,5,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4 > /blog/post-2,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-3,yes,200,5,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4 > /blog/post-3,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/docs/getting-started/install/proxy/sso,yes,200,5,1,1,0,/docs,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy > /docs/getting-started/install/proxy/sso,deeper_than_3
http://127.0.0.1:8794/blog/page/4,no,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4,deeper_than_3;only_via_pagination;not_in_sitemap
http://127.0.0.1:8794/blog/post-4,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-4,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-5,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-5,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-6,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-6,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/customers/initech,yes,200,4,1,1,0,/customers,/ > /about > /customers > /customers/globex > /customers/initech,deeper_than_3
http://127.0.0.1:8794/docs/getting-started/install/proxy,yes,200,4,1,1,0,/docs,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy,deeper_than_3
...
http://127.0.0.1:8794/careers,yes,,,0,0,35,/careers,,not_reached
http://127.0.0.1:8794/webinar-2025,yes,,,0,0,0,/webinar-2025,,not_reached
What the run shows:
-
The deepest page is a sales page, not a docs page.
/pricing/enterprisesits one click below the pricing page for a visitor who clicks the "Talk to sales" button, but that button is anonclickhandler with nohref, so the crawl does not count it. The only real link is at the bottom of a docs page five clicks down, which makes the enterprise plan six clicks from the home page. Thepathcolumn shows the route the crawler had to take. -
Eight of the 12 blog posts are
only_via_pagination. Posts 10 to 12 are on the first listing page (depth 2), post 1 is linked from the features page (depth 2), and everything else needs one more "Next" click per three posts. Posts 2 and 3 are five clicks deep. - The docs chain adds one click per level. Four of the six docs URLs are fine; the proxy and SSO pages are at 4 and 5.
- The case studies are uneven. Acme is one click away because the home page features it. Initech is four clicks away because the only link to it is inside another case study.
-
/old-integrationsis reported once, asredirected, and is not counted in the depth histogram, because the crawl credits that link to/integrations. -
/careerswas never reached, and the CSV shows why: 35 nofollow links and no followed ones./webinar-2025has no links of any kind. Both need the orphan check from the previous post, not a depth fix. -
/admin/was skipped becauserobots.txtdisallows it.
After fixing the fixture
I changed three things in the fixture: every blog listing page now links to every other listing page (numbered pages instead of only "Next"), the pricing page links to /pricing/enterprise with a real <a href>, and the getting-started docs page links directly to the proxy and SSO pages. Same command, real output:
Sitemap: 35 URLs from http://127.0.0.1:8794/sitemap.xml
Crawled 35 pages breadth first from http://127.0.0.1:8794/
Skipped 1 URLs disallowed by robots.txt
Click depth of sitemap URLs:
depth 0: 1 #
depth 1: 8 ########
depth 2: 10 ##########
depth 3: 12 ############
depth 4: 1 # <- deeper than 3
not reached: 2
By section (sitemap URLs reached):
section pages max deep
/customers 4 4 1
/blog 13 3 0
/docs 6 3 0
/integrations 3 2 0
/pricing 2 2 0
/about 1 1 0
/features 1 1 0
/privacy 1 1 0
/ 1 0 0
Flagged sitemap URLs (12), deepest first:
4 deeper_than_3 /customers/initech
3 only_via_pagination /blog/post-2
3 only_via_pagination /blog/post-3
3 only_via_pagination /blog/post-4
3 only_via_pagination /blog/post-5
3 only_via_pagination /blog/post-6
3 only_via_pagination /blog/post-7
3 only_via_pagination /blog/post-8
3 only_via_pagination /blog/post-9
1 redirected /old-integrations
- not_reached /careers (35 nofollow links)
- not_reached /webinar-2025
(+3 crawled pages not in the sitemap; see the CSV)
Report: click_depth_after.csv
exit code: 1
Every docs page is now within three clicks, the enterprise page dropped from 6 to 2, and posts 2 and 3 went from 5 to 3. The blog posts are still flagged only_via_pagination, which is correct: numbered page links shorten the walk, but a reader still has to page through the archive to find post 2. Topic hubs, "related posts" links, or a link from a relevant product page would give those posts a second route. Initech is still at 4 because I left it alone.
When the sitemap URL is wrong, the script stops early:
Could not read sitemap: http://127.0.0.1:8794/missing.xml returned 404
exit code: 2
Running it on your own sitemap
Point it at your sitemap. On a large site, keep the worker count low, add a delay between levels, and cap the crawl on a first pass:
python3 click_depth.py https://example.com/sitemap.xml --workers 2 --delay 0.5 --limit 2000 --out depth.csv
Some options worth knowing:
-
--max-depth 4changes the flag line. Pick the number for your site and keep it the same between runs so the CSVs are comparable. -
--start https://example.com/blog/measures depth from a section hub instead of the home page. Repeat--startto begin from several pages at once (for example, home plus the docs index if docs live on the same host and most visitors land there). -
--drop-querytreats?utm_source=...and sort or filter parameters as the same page. Leave it off if your pagination uses?page=, or the archive pages collapse into one URL. -
--max-crawl-depth(default 25) stops the crawl from following an endless calendar or filter trail forever. -
--no-robotsis for your own staging site only. On a live site, letrobots.txtdecide.
If the crawl reaches far fewer pages than your sitemap lists, check whether your navigation is rendered by JavaScript. The script reads only the HTML your server returns and does not run JavaScript. A site that builds its menu in the browser will look much deeper (or unreachable) to this script than it does to a person.
If you want a quick look before running a crawl, two free tools on our site can help. Sitemap Analytics and Insights reads a urlset (pasted or by URL) and counts path depth, file extensions, http versus https and query strings. That is URL folder depth, not click depth, and as the Google page says, folder structure is not how Google works out site structure, so use it to spot over-nested URL patterns and use the script for clicks. The Website URL Extractor collects same-host links from up to 80 HTML pages or until 12 seconds pass, and skips a host whose robots.txt disallows all crawling, which is enough to see which URLs a small site actually links to. Both ask for a captcha before each run; the script has no page cap and gives you the path for every URL.
Fixing what it finds
Work from the CSV, sorted by depth:
- Decide which deep pages matter. A five-click changelog entry from 2022 may be fine. A five-click pricing or signup page usually is not. Mark the pages that should be close to the home page before changing anything.
-
Turn button "links" into real links. Any
deeper_than_Npage that people reach in a browser through a button, a JavaScript menu or a search box is a candidate. Google's pages are clear that crawlers follow<a href>and do not click buttons or submit searches. - Shorten archives. Link every listing page to the others (or at least to the first page and a few around the current one), as the pagination page recommends for linking back to the first page of a collection.
- Add hubs, not more nav items. Topic or category pages that link to every post on a subject give old posts a route that does not depend on publish date. A "related posts" block does the same at the post level.
- Flatten long chains. If a docs section nests five levels deep, link the section index to every page in it.
- Link the pages you want to promote from the home page. That is the ecommerce structure page's own example for a best seller, and it works for a flagship case study or an integration page too.
- Rerun and compare. Keep the CSV from each run. A page that jumps from depth 2 to depth 5 after a redesign is usually a nav item someone removed.
Extending it
-
Track depth over time. Save the CSV per deploy and diff the
depthcolumn. Fail CI only when a page that was within the threshold moves past it, so existing deep pages do not block every build. -
Join with analytics or Search Console. Export clicks per page, join on
url, and sort deep pages by traffic. A deep page with traffic is worth a better route; a deep page with none may be a candidate to merge or retire. -
Count JavaScript-only anchors. Add a counter in
LinkParserfor<a>tags that haveonclickorrouterLinkand nohref(the orphan post does this), and print which pages have them. In the fixture, that would have pointed straight at the pricing page. -
Weight by section. Use a different
--max-depthper section by running the script with--startset to each section hub, or by post-processing the CSV by thesectioncolumn.
FAQ
What is a good maximum click depth?
Google's pages do not give one. They say Google can use "the number of links it needs to follow to reach a page" to infer importance, which means shallower pages look more important relative to the rest of the site. Three clicks is a common rule of thumb; choose the number that fits your site's size and keep it fixed between runs.
Is click depth the same as URL depth?
No. /a/b/c/d can be one click from the home page, and /pricing/enterprise was six clicks away in the fixture. Google's ecommerce page says it "generally doesn't look at the structure of URLs to work out the structure of a site." The script only measures clicks.
Why breadth first instead of following links as they come?
Breadth first finishes every page at depth 1 before looking at depth 2, so the first time a page is found is its shortest distance. A depth-first crawl can reach a page through a long route first and report the wrong number.
Does a sitemap fix deep pages?
A sitemap helps a crawler find URLs, and Google's ecommerce page recommends one when "it's not possible to link to all pages." It does not change how many clicks a person needs, and it does not add links between your pages. The script reads the sitemap only to know which URLs to report on.
Can I run it in CI?
Yes. It exits 1 when a sitemap URL is too deep, only reachable through pagination, unreached or erroring, 0 when clean and 2 when the sitemap cannot be read. Run it against a staging build with --start pointed at the staging home page.
Wrap-up
Click depth is a site structure signal you can measure directly, and Google names it in its own documentation. A breadth-first crawl from the home page gives you the shortest path to every page in your sitemap, and the path column shows exactly where a route gets long: an archive that only has "Next" links, a docs tree that adds a level per release, a sales page behind a button. Fix the pages that matter, link them from hubs or the home page, and rerun the script after every redesign so a removed nav item shows up as a depth jump before anyone notices the traffic.
RG, Founder, Appie.run
Top comments (0)