Your scraper works perfectly on 50 pages. At 5,000, it starts returning 403s, then 429s, then CAPTCHA pages that your parser happily treats as real data. The code didn’t break. The target site just noticed that one IP address is making thousands of requests.
The fix isn’t a cleverer parser. It’s a pipeline with a proxy layer that spreads traffic across IPs, retries intelligently, and stops using addresses that have gone bad. This guide builds one in Python, step by step.
Why Scrapers Get Blocked
Sites rarely block you for one reason. They combine signals:
- Request volume per IP: Hundreds of requests per minute from one address is the easiest thing to flag.
- IP reputation: Datacenter ranges and previously abused IPs get stricter treatment.
-
Missing or odd headers: Default library User-Agents like
python-requests/2.xare an instant giveaway. - Unnatural timing: Perfectly even intervals look like a machine, because they are.
A proxy layer addresses the first two directly and makes the other two easier to manage, because every request now passes through one place where you control headers, delays, and retries.
The Architecture
Keep the pipeline in four small stages, each with one job:
- Proxy pool: Holds your proxies and tracks which are healthy.
- Fetcher: Makes requests through a proxy, retries on failure, and backs off.
- Parser: Turns HTML into structured data.
- Storage: Writes results somewhere durable.
Separating them means you can swap a proxy provider, change a parser, or move from SQLite to Postgres without touching the rest.
Step 1: Choose Your Proxies
Proxy type matters more than most tutorials admit:
- Datacenter proxies are fast and cheap. They suit sites with light protection and high-volume jobs.
- Residential proxies use real household IPs. They cost more but pass stricter checks.
- ISP and mobile proxies sit in between, useful when you need stable, trusted addresses.
For this tutorial, I used Proxy-Seller, which offers several of these types and supports both IP allowlisting and username/password authentication. If you sign up, you can use the referral code HARSH15. Whichever provider you pick, the code below works the same, because it only needs a list of proxy URLs in this format:
http://username:password@host:port
Step 2: Build the Proxy Pool
The pool picks a proxy, tracks failures, and puts misbehaving proxies in a timeout instead of deleting them. IPs often recover after a rest.
# proxy_pool.py
import random
import time
from dataclasses import dataclass
@dataclass
class Proxy:
url: str
failures: int = 0
cooldown_until: float = 0.0
def available(self) -> bool:
return time.time() >= self.cooldown_until
class ProxyPool:
def __init__(self, urls, max_failures=3, cooldown=300):
self.proxies = [Proxy(u) for u in urls]
self.max_failures = max_failures
self.cooldown = cooldown
def get(self) -> Proxy:
live = [p for p in self.proxies if p.available()]
if not live:
raise RuntimeError("No healthy proxies available")
return random.choice(live)
def report_success(self, proxy: Proxy):
proxy.failures = 0
def report_failure(self, proxy: Proxy):
proxy.failures += 1
if proxy.failures >= self.max_failures:
proxy.cooldown_until = time.time() + self.cooldown
proxy.failures = 0
Random selection is a good default. If you want fairness, swap random.choice for round-robin.
Step 3: Build the Fetcher With Retries
This is the core of the pipeline. It treats block-style status codes as failures, rotates to a different proxy on each retry, and uses exponential backoff with jitter.
# fetcher.py
import random
import time
import httpx
BLOCK_CODES = {403, 407, 429, 503}
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
class BlockedError(Exception):
pass
def fetch(url: str, pool, retries: int = 4):
for attempt in range(retries):
proxy = pool.get()
try:
with httpx.Client(
proxy=proxy.url,
headers=HEADERS,
timeout=15,
follow_redirects=True,
) as client:
resp = client.get(url)
if resp.status_code in BLOCK_CODES:
raise BlockedError(resp.status_code)
resp.raise_for_status()
pool.report_success(proxy)
return resp.text
except (httpx.HTTPError, BlockedError):
pool.report_failure(proxy)
delay = min(2 ** attempt + random.random(), 30)
time.sleep(delay)
return None
Three details matter here:
- A fresh proxy per attempt. Retrying on the same blocked IP is wasted effort.
- Jitter in the delay. The
random.random()keeps retries from landing in a perfect rhythm. - Block codes as failures. A 429 should count against the proxy, not be silently accepted.
Step 4: Parse and Detect Soft Blocks
Some sites return a 200 status with a CAPTCHA or “access denied” page. If you parse that, you store garbage. Add a quick sanity check before extracting data.
# parser.py
from bs4 import BeautifulSoup
BLOCK_MARKERS = ("captcha", "access denied", "unusual traffic")
def looks_blocked(html: str) -> bool:
text = html.lower()
return any(marker in text for marker in BLOCK_MARKERS)
def parse_product(html: str) -> dict | None:
if looks_blocked(html):
return None
soup = BeautifulSoup(html, "html.parser")
title = soup.select_one("h1")
price = soup.select_one(".price")
if not title:
return None
return {
"title": title.get_text(strip=True),
"price": price.get_text(strip=True) if price else None,
}
Adjust the selectors to your target. The important habit is returning None when the page doesn’t look right, so bad pages never reach storage.
Step 5: Store Results
SQLite is plenty for most scrapers and needs no setup. Using INSERT OR REPLACE keeps reruns from creating duplicates.
# storage.py
import sqlite3
def init_db(path="results.db"):
conn = sqlite3.connect(path, check_same_thread=False)
conn.execute(
"CREATE TABLE IF NOT EXISTS products "
"(url TEXT PRIMARY KEY, title TEXT, price TEXT)"
)
return conn
def save(conn, url, data):
conn.execute(
"INSERT OR REPLACE INTO products VALUES (?, ?, ?)",
(url, data["title"], data["price"]),
)
conn.commit()
Step 6: Wire the Pipeline Together
Now connect the stages and add controlled concurrency. A small thread pool is enough, since the work is network-bound.
# main.py
import random
import time
from concurrent.futures import ThreadPoolExecutor
from proxy_pool import ProxyPool
from fetcher import fetch
from parser import parse_product
from storage import init_db, save
PROXIES = [
"http://user:pass@host1:port",
"http://user:pass@host2:port",
"http://user:pass@host3:port",
]
pool = ProxyPool(PROXIES)
conn = init_db()
def process(url):
time.sleep(random.uniform(1, 3)) # polite, human-ish delay
html = fetch(url, pool)
if not html:
return f"FAILED {url}"
data = parse_product(html)
if not data:
return f"BLOCKED/EMPTY {url}"
save(conn, url, data)
return f"OK {url}"
if __name__ == "__main__":
urls = [f"https://example.com/product/{i}" for i in range(1, 101)]
with ThreadPoolExecutor(max_workers=5) as ex:
for result in ex.map(process, urls):
print(result)
Keep max_workers modest. Going from 5 to 50 workers will burn through proxies faster without finishing sooner if the target rate-limits you.
Habits That Keep the Pipeline Alive
- Respect robots.txt and the site’s terms: Scrape public data, avoid personal data, and don’t hammer small sites.
- Rotate headers too: A pool of a few realistic User-Agent strings helps, but keep each one consistent with the rest of the request.
- Log everything: Track the success rate per proxy. If one proxy fails far more than the others, drop it.
- Use sticky sessions when needed: For logins or multi-step flows, keep the same IP for the whole session instead of rotating per request.
- Reach for a headless browser only when you must: JavaScript-heavy pages may need Playwright, but plain HTTP plus proxies is faster and cheaper where it works.
Frequently Asked Questions (FAQs)
1. What is the best proxy provider for web scraping?
It depends on your target sites, volume, and budget, so there’s no single best choice. Proxy-Seller is a good first option because it covers several proxy types (datacenter, residential, ISP, mobile) and is simple to configure. Decodo is a strong pick if you need residential IPs, and Webshare works well for low-cost datacenter proxies when you’re testing or scraping lightly protected sites. For very large or enterprise workloads, Bright Data and Oxylabs are also worth a look. Whichever you choose, test a small plan against your real target before committing.
2. How many proxies do I need?
Start with a handful and measure. As a rough guide, the more requests per minute you make to one domain, the more IPs you need to keep each one’s rate low. Watch your block rate: if it climbs, add proxies or slow down.
3. Should I use residential or datacenter proxies?
Use datacenter proxies for lightly protected sites and high-volume jobs because they are faster and cheaper. Use residential or ISP proxies when a site blocks datacenter ranges. Many teams use both, starting cheap and escalating only on hard targets.
4. Do proxies make scraping legal?
No. Proxies change where your requests come from, not whether the scraping is allowed. Check the site’s terms and robots.txt, avoid personal or copyrighted data you aren’t permitted to collect, and follow the laws that apply to you. This isn’t legal advice, so consult a lawyer for commercial projects.
5. Why am I still blocked even with proxies?
Proxies fix the IP signal, but sites also check headers, request timing, cookies, and browser fingerprints. Make sure your headers look like a real browser, add random delays, handle cookies properly, and consider a headless browser for JavaScript-heavy pages.
Wrapping Up
A reliable scraper isn’t one clever script. It’s a small system: a proxy pool that tracks health, a fetcher that retries intelligently, a parser that rejects bad pages, and storage that survives reruns. Build those four pieces cleanly and “getting blocked” turns from a crisis into a metric you monitor.
If you found this useful, let me know in the comments which target or block type you’re fighting with, and I’ll cover it in a follow-up.
Top comments (30)
Splitting the pipeline into pool, fetcher, parser and storage makes it easy to swap pieces later. Two parts stood out to me:
A few things I’d add when scaling this up:
Retry-Afteron 429s. When a site sends it, it’s a better signal than a fixed exponential backoff.ThreadPoolExecutor, the failure counters inProxyPooland the shared SQLite connection can race. Athreading.Lockaroundreport_failureand thesavecall avoids odd edge cases.Curious whether you’ve compared block rates between datacenter and residential proxies on the same target. A follow-up with real numbers would be great.
Thanks, glad the separation of pool, fetcher, parser and storage came across. Keeping the pieces swappable was the main goal. The soft-block check and the cooldown logic both came from getting burned in practice. A 200 with a CAPTCHA page in the database is a nasty one because nothing errors out, and I was throwing away proxies that worked fine after a short rest.
Your three additions are all good ones:
threading.Lockaroundreport_failureand the save call is the simplest fix. A per-thread connection or a single writer thread would also work.I haven’t run a clean datacenter vs residential comparison on the same target yet. A follow-up with real block rates, cost per successful request and where each type falls over would be useful, so I’ll consider it. Thanks for the detailed feedback.
Thanks for the detailed reply, and for confirming the Retry-After and per-domain points. The CAPTCHA-behind-a-200 case is the one that bites most people, since a bad row looks exactly like a good one.
I’d definitely read the datacenter vs residential follow-up. Metrics I’d find most useful:
If you run it, test one site with light protection and one with aggressive bot detection, because the gap tends to show up there. Looking forward to it!
The soft-block check in Step 4 is the part most tutorials skip, and parsing a CAPTCHA page as real data is exactly how bad rows end up in a database without anyone noticing.
Two small things I’d add for anyone taking this to production:
Retry-Afteron 429s. If the server tells you how long to wait, using that value instead of a fixed backoff saves a lot of burned retries.Also, a 407 usually means a proxy auth/config problem rather than a blocked IP, so it might be worth logging separately from 403/429.
Curious how you’d handle sticky sessions in this setup. Pin a proxy per session ID in the pool?
Totally agree on the soft-block check. A CAPTCHA page that returns a 200 is the sneakiest failure, because everything looks healthy until someone audits the data.
Your three additions are all solid:
On sticky sessions: yes, pinning a proxy per session ID is how I’d do it. Keep a
session_id → proxymap and reuse that proxy for the whole session (login, cart, pagination). Only reassign it if the proxy gets blocked or hits a failure threshold, and in that case start a fresh session rather than switching IPs mid-session, since the site will often invalidate the cookies anyway. Add a TTL so idle sessions don’t hold proxies forever.Thanks, the session_id → proxy map with a TTL is what I’d go with too. One addition: pin the whole identity to the session, not just the IP. Keep the same User-Agent, headers, and cookie jar for the session’s lifetime, because a stable IP with a changing fingerprint is just as easy to flag.
When a session dies (block, failure threshold, or TTL), retire the entire session object and start a clean one, so no half-valid cookies carry over. It’s also worth logging which session a block happened on and how many requests it had made. That shows you the real per-domain threshold, so you can rotate proactively just before it instead of reacting after the ban.
Good addition, and it fits the TLS fingerprint point from earlier in this thread. A stable IP with a changing fingerprint is just as easy to flag as the reverse, so the session should own the whole identity, not only the proxy.
curl_cffi, the impersonation profile should be pinned too, so the User-Agent never disagrees with the TLS handshake.For proactive rotation, I’d rotate at a margin below the observed threshold (say, around 70-80% of the typical block point) rather than right at it. Thresholds drift with concurrency, time of day and site changes, so a hard cutoff will eventually be wrong. I haven’t benchmarked this myself, so treat those numbers as a starting point to tune against your own logs.
I’ll fold this into the session-handling section of the hardening follow-up. Thanks for the detailed addition.
Keeping the pool, fetcher, parser, and storage separate makes this easy to extend. Putting proxies in a cooldown instead of deleting them is a good touch, since IPs do recover.
A few things I’d tweak if you take this to production:
proxy. resp.raise_for_status()raises httpx.HTTPStatusError, which theexcept httpx.HTTPErrorbranch catches. A plain 404 or 410 therefore penalizes a healthy proxy and triggers pointless retries. I’d return early on those, or retry only on the block codes and 5xx.Retry-Afteron 429s. If the server tells you how long to wait, using that value beats a generic exponential backoff.ThreadPoolExecutor, several workers updatefailuresandcooldown_untilat once. Athreading.Lockaroundreport_failureorgetavoids odd race conditions.httpxandrequestshave a recognizable TLS/JA3 fingerprint.curl_cffiwith browser impersonation often fixes this without a headless browser.On your question about which targets cause trouble: Cloudflare-protected sites are the ones I hit most, and the TLS fingerprint was usually the culprit rather than the IP.
Would you consider a follow-up on async (
httpx.AsyncClientplus semaphores) for the same architecture?You’re right about the 404s. As written,
raise_for_status()feeds every HTTP error into the proxy penalty path, so a dead URL gets blamed on a healthy proxy and retried for nothing. The fix is to return early on 404/410 and only retry on the block codes (403, 429) and 5xx.Respecting
Retry-Afteron 429s is also better than blind exponential backoff. I’d parse it, fall back to backoff when the header is missing, and cap the wait so one bad response can’t stall a worker.On soft-block detection, keyword matching is the weakest part of the example. A reCAPTCHA script on a normal login page would trip it. Checking for an expected selector plus a minimum content length is much more reliable, and I’ll add that as the recommended approach.
The thread-safety point is fair too. With
ThreadPoolExecutor,failuresandcooldown_untilare shared state, so athreading.Lockaroundgetandreport_failureis the right call for production.The TLS fingerprinting note is the one I most want to work into the post. It explains why some sites keep returning 403 even with clean proxies and good headers.
curl_cffiwith browser impersonation is a lighter fix than jumping straight to a headless browser. And since your experience with Cloudflare-protected sites points to the fingerprint rather than the IP, that’s a useful pointer for anyone debugging the same thing.An async follow-up with
httpx.AsyncClientand semaphores is a good idea, and the pool/fetcher/parser/storage split should carry over cleanly. I’ll plan on it, and I’ll fold your fixes into the production section.Nice walkthrough. Splitting it into pool, fetcher, parser and storage makes it easy to swap parts without rewriting everything. Putting proxies in a cooldown instead of deleting them is a good detail, since IPs often recover.
One thing I’d like to see in a follow-up: right now
looks_blocked()runs in the parser, but the proxy pool never hears about it. A CAPTCHA page that comes back as a 200 still counts asreport_success, so a burned proxy stays in rotation. Feeding soft blocks back into the pool’s failure count would make the health tracking more accurate.Two small things I noticed:
httpx.Clienton every attempt skips connection reuse, which adds up at scale.report_failure, so dead URLs can push healthy proxies into cooldown.For me the hard part is sites that fingerprint TLS, where good proxies still get blocked with a clean header set. Are you planning to cover that, or the Playwright fallback, next?
The soft-block gap is the real flaw.
looks_blocked()lives in the parser, so a CAPTCHA page that returns a 200 still callsreport_successand the burned proxy stays in rotation. The fix is to run that check inside the fetcher (or return a signal the fetcher can act on) and callreport_failureon a soft block, so the pool’s health tracking matches what the target is really doing.The 404 point is also fair.
raise_for_status()throws anHTTPError, which lands in the sameexceptbranch, so dead URLs count against healthy proxies. 404s and 410s should skipreport_failureand return early. And a singlehttpx.Clientper proxy, reused across attempts, would give you connection reuse instead of a new handshake every time.On TLS fingerprinting, you’ve hit the limit of this setup. Proxies only change the IP, and plain
httpxstill presents a very recognizable TLS signature, so good IPs plus clean headers can still get blocked. Two usual ways out arecurl_cffiwith browser impersonation, or a real browser via Playwright. I’ll cover both in a follow-up, along with the soft-block feedback loop and the fixes above.Separating the pool, fetcher, parser and storage means you can swap a proxy provider or move from SQLite to Postgres without touching the rest. I especially liked the cooldown approach in
ProxyPool: putting a proxy in timeout aftermax_failuresinstead of deleting it fits how rotating IPs behave, since many recover after a rest. The soft-block check in the parser is also the right call. A 200 response with a CAPTCHA page is the failure that quietly corrupts a dataset, and returningNonebefore storage keeps that out.Two things I’d consider adding. First, honor the
Retry-Afterheader on 429 responses instead of a fixed exponential backoff, and weight proxy selection by recent success rate rather thanrandom.choice, so a degrading IP gets picked less before it hits the cooldown threshold. Second, withcheck_same_thread=Falseand five workers sharing one SQLite connection, you may hit “database is locked” errors or interleaved commits. Athreading.Lockaroundsave(), or WAL mode(PRAGMA journal_mode=WAL), would make it safer under concurrency. Great guide overall!Thanks for the detailed read, really glad the separation of pool, fetcher, parser and storage came across as intended. The cooldown in ProxyPool was a deliberate choice for exactly the reason you gave: rotating IPs often recover after a rest, so permanent removal wastes good capacity.
Both suggestions are fair. Honoring Retry-After on 429s is more respectful to the target and more efficient than blind exponential backoff, and weighting proxy selection by recent success rate would let a degrading IP fade out before it hits the cooldown threshold instead of after.
You’re also right about the SQLite setup. With check_same_thread=False and five workers on one connection, “database is locked” errors or interleaved commits are a real risk. A threading.Lock around save() is the simplest fix, and WAL mode (PRAGMA journal_mode=WAL) is the better option once reads and writes overlap. For heavier concurrency, moving to Postgres is where the modular storage layer pays off.
Treating soft blocks (200 responses with a CAPTCHA page) as failures is the part most tutorials skip, and it’s what keeps bad data out of storage.
A few things I’d tweak when taking this to production:
check_same_thread=Falseand 5 worker threads sharing one connection, concurrentsave()calls can collide. Athreading.Lockaround the write, or a single writer thread fed by a queue, avoids “database is locked” errors.httpx.Clientper attempt skips connection pooling. One client per proxy would speed things up.Good call on the cooldown instead of deleting proxies too, since IPs often recover. Have you tried a weighted pick based on per-proxy success rate instead of
random.choice?Thanks for the detailed feedback, this is exactly the kind of hardening the tutorial version skips.
You’re right on all four:
threading.Lockaroundsave()is the quick one.httpx.Clientper proxy keeps connection pooling and cuts per-request overhead.On weighted picking, I haven’t benchmarked it against
random.choiceyet, but it’s the natural next step. I’d track success and failure per proxy over a rolling window, weight by success rate, and keep a small exploration share so recovering proxies still get tested. Have you tried this in your own setup? I’d be curious how much it improved your success rate.Quick note for anyone trying this: the soft-block check in Step 4 saved me the most headaches. A 200 response with a CAPTCHA page quietly poisons your data if you don’t catch it.
Curious what everyone else is running into: are you getting hard blocks (403/429), soft blocks (CAPTCHA pages), or JavaScript-rendered pages where plain HTTP just doesn’t work? Drop your target type below and I’ll cover it in a follow-up.
mostly soft blocks on our side, a 200 that is really a login or captcha page and the parser stores it as a row
what helped was counting rows and checking one required field per run, and failing the run when either looks off instead of writing it
Soft blocks are the nastiest kind, because a 200 hides the failure and the bad row looks like real data. Your fix is a good one. A row count plus one required-field check per run catches most of these without much code.
Failing the run instead of writing is the key part. A bad batch that never lands is easier to deal with than one you have to clean out of the dataset later. Do you also keep the raw HTML of the failed pages? It makes it quick to tell a captcha from a login wall, and that tells you whether to rotate the proxy or slow down.
Totally agree on the soft-block check. A 200 with a CAPTCHA page is the sneakiest failure because nothing errors out, your pipeline just keeps happily saving garbage. Checking for known CAPTCHA markers and suspiciously small or oddly structured responses before parsing is cheap insurance.
For the follow-up, I’d love to know more about what people are hitting:
Drop your target type (e-commerce, social, travel, listings, etc.) and roughly what scale you’re working at, and I’ll cover the most common one in the next post.
Separating pool, fetcher, parser and storage makes this easy to extend. A few things I’d harden before running it at scale:
looks_blocked()returnsNonein the parser, but by thenfetch()has already calledreport_success(proxy). A proxy that only serves CAPTCHA pages keeps a clean failure count forever. Running the soft-block check inside the fetcher fixes this.ProxyPoolmutatesfailuresandcooldown_untilfrom 5 worker threads with no lock. The same goes for the shared SQLite connection withcheck_same_thread=False. Athreading.Lockaround both, or a single writer thread fed by a queue, avoids “database is locked” errors.Retry-Afteron 429s instead of a fixed exponential delay.Also, creating a new
httpx.Clientper attempt drops connection reuse, so you pay a TLS handshake every time. Is a per-proxy client cache worth it in your testing?Thanks, this is a really useful list.
report_success()fires before the parser ever sees the CAPTCHA page, so a proxy that only serves challenges never accrues failures. Moving the block check into the fetcher and callingreport_failure()from there is the right fix.curl_cffiwith impersonation is the better option for protected targets.I’ll probably cover these in a follow-up on hardening the pipeline. Thanks for taking the time to write it up.
A few quick tips if you’re testing this on a real target:
What are you scraping at the moment, and are you seeing 403s, 429s, or CAPTCHA pages? Share it here, and I’ll try to cover it in a follow-up.
The 3-5 workers point is the one I’d stress most. Cranking up threads usually just burns through the proxy pool faster and spikes the block rate.
Per-proxy success logging is worth doing early too. A single bad proxy can quietly drag down the whole run, and you only spot it once you track failures per IP. Sticky sessions for login and multi-step flows are the right call as well, since rotating mid-session often triggers extra checks.
On your question, I mostly see 429s on high-volume listing pages, and 403s once a site’s fingerprinting kicks in. CAPTCHAs tend to show up when both happen together. What are you running into on your side?
That 429 vs 403 split matches what I see too. 429s on listing pages are usually a rate problem, which fewer workers, jittered delays and a wider pool fix. 403s after fingerprinting are a different issue. Rotating IPs won’t help if the TLS fingerprint and headers still look like a script, so that’s where I’d look at a real browser profile or a client that mimics one.
For my own runs, the blocks mostly show up on paginated search results and product listings, with a few CAPTCHAs once the request volume ramps up. I’m now watching the block rate per proxy rather than overall, which makes it much easier to see whether the problem is the IP or the request pattern.
Since you’re seeing 429s on listing pages, do they start at a consistent request count per IP, or only when concurrency goes up? That would tell us whether to fix pacing or the pool size first, and I’ll cover it in the follow-up.
Agreed on the split. 429s are a pacing problem and 403s are an identity problem, and mixing them up is how people end up burning through proxy pools for nothing. Tracking block rate per proxy is a great call too. It’s the quickest way to tell a bad IP from a bad request pattern.
On your question, I haven’t seen a fixed request count per IP that triggers 429s. It tends to depend on concurrency and how bursty the traffic is, more than on a set number of requests. If the same IP is fine at low concurrency and starts getting 429s when workers go up, that points to pacing. If a fresh IP gets blocked within a few requests, that points to the fingerprint or headers. So I’d fix pacing first (fewer workers, jitter, backoff on 429), then size the pool.
I’ll cover the pacing-vs-pool-size test in the follow-up. If you’ve seen a pattern on your side, I’d like to hear it.
Solid tutorial! The cooldown approach for proxy health management is such an underrated pattern. So many beginner scraping guides just pop() a failing proxy out of the list permanently, which wastes good residential/datacenter IPs that were just temporarily rate-limited.
A few thoughts and additions from building production scraping pipelines:
Pro-tip: If you find yourself getting instant 403s despite using clean proxies, swap httpx for libraries like curl_cffi or tls-client, which impersonate actual browser TLS signatures.
Python
Quick snippet for async pool concurrency control
import asyncio
semaphore = asyncio.Semaphore(10) # limit global active connections
async def safe_fetch(url):
async with semaphore:
return await async_fetch(url, pool)
Thanks for laying out the architecture so cleanly—separating the Fetcher, Pool, and Sanity Parser is definitely the right way to write maintainable scrapers!
permanent
pop()is one of those habits that looks fine in a tutorial and then quietly burns through your good IPs in production.Your three additions are all good ones:
TLS fingerprinting: Agreed, and it’s an easy one to miss. People rotate proxies and User-Agents for hours before realizing the block happens at the handshake.
curl_cffiis the cleanest swap since its API is close torequests, so it drops into the Fetcher without much change. I kept the tutorial onhttpxto keep the focus on the architecture, but this belongs in a “when you still get 403s” follow-up.ThreadPoolExecutor vs. asyncio: You’re right that threads work for learning and small jobs but stop scaling well past a handful of concurrent requests. A semaphore per proxy, on top of your global one, would also stop a single IP from getting hammered, which is a nice way to keep the cooldowns from triggering in the first place.
Circuit breaker: This is the missing piece in the pool as written. Without it, a dead target just drains every proxy’s cooldown and your bandwidth for nothing. A failure-rate threshold with a long pause is a much better failure mode than retrying a dead endpoint.
Thanks for the production notes. I may fold the async and circuit breaker ideas into a follow-up. And yes, keeping the Fetcher, Pool, and Parser separate is what makes swaps like
curl_cffipainless.Some comments may only be visible to logged-in visitors. Sign in to view all comments.