The problem
The first version of every streaming integration I have seen is the same five lines: open the stream, loop over chunks, append each chunk to a string, show the string. It demos beautifully. Then production gets a vote.
The failures arrive in a predictable order. First, the product team wants structured output, a JSON object with a title, a summary, and a score, and someone tries to json.loads the accumulated string on every chunk. That throws on every chunk until the last one, because partial JSON is not JSON. Second, a client disconnects mid-stream and the server keeps generating, writing into a dead socket or, worse, leaving the provider connection half-open. Third, a slow consumer shows up: the UI thread stalls, the downstream writer blocks, and the unbounded append buffer quietly becomes a memory problem on the longest responses. Fourth, the provider fails at chunk 40 of 50, after half the answer is already on screen, and the code has no vocabulary for "here is what you got, and here is where it stopped."
None of these need a framework. They need three small pieces with explicit contracts: a buffer that can say no, a parser that emits what is finished instead of demanding what is not, and a consumer loop that treats abort and error as normal outcomes, not exceptions that escape, all in dependency-free Python.
The approach
Three design decisions shape the code.
First, nothing in this layer parses a partial document. An incremental extractor consumes characters and emits each top-level field at the moment its value is complete: a string when its closing quote arrives, a nested object when its depth returns to zero, a scalar when its delimiter lands. Only then does json.loads run, and only on that field's raw text. Consumers get usable values early.
Second, the buffer is bounded in characters and its offer() returns False when full. That refusal is the entire backpressure mechanism: the producer may not advance until the consumer drains room. An unbounded queue with a fast producer and a slow consumer is not a pipeline, it is a memory leak with good intentions.
Third, the consumer always returns a report. Complete, aborted, and errored streams all produce the same shape: the fields delivered, how many chunks arrived, what was mid-flight, and the error if there was one. Callers branch on a status field instead of catching exceptions from three layers. The demo runs against a stub token source that yields a canned response in fixed-size chunks, so everything runs offline with no API key.
Step 1: The incremental field extractor
The extractor is a small state machine over characters. It tracks whether it is reading a key, waiting for a colon, or inside a value, and inside a value whether that value is a string (watching for escapes), a nested structure (counting depth, ignoring brackets inside strings), or a scalar like a number or true. When a value completes, it decodes and emits.
class FieldExtractor:
"""Emit each top-level field of a JSON object as soon as it completes."""
def feed(self, text):
out = []
for ch in text:
self._step(ch, out)
return out
def _complete(self, out):
value = json.loads(self._value_raw)
out.append((self._key, value))
self._key = None
self._value_raw = ""
The full class is in streaming_llm.py, about 90 lines, most of it the nested-value case. Two properties matter more than the mechanics. The extractor is chunking-agnostic: feed it one character at a time or a kilobyte at a time and it emits the same fields in the same order, which makes it trivially testable. And it can name the field that is mid-flight, via a pending_key property. That name turns out to be the most useful debugging output in the whole layer, as the error scenario below shows. One honest limit: it handles a single top-level object, the shape most structured-output prompts produce.
Step 2: A bounded buffer that can refuse
class BoundedBuffer:
"""A character-capped FIFO. offer() returns False when full."""
def __init__(self, capacity_chars):
self.capacity = capacity_chars
self._chunks = deque()
self.size = 0
self.refused = 0
def offer(self, chunk):
if self.size + len(chunk) > self.capacity:
self.refused += 1
return False
self._chunks.append(chunk)
self.size += len(chunk)
return True
# drain() joins and clears held chunks
Capacity is counted in characters, not chunks, because chunk sizes vary by provider and memory is spent on characters. The refused counter is not decorative: it is the metric that tells you your consumer is the bottleneck, and by how much. A producer that gets a refusal has exactly two honest options, wait or drop. What it may not do is append anyway and let the process find out later.
Step 3: The consumer: aborts and errors as outcomes
run_stream pulls chunks, offers them to the buffer, drains into the extractor, and records each emitted field with the chunk index at which it completed. Three exits, one report:
def run_stream(source, capacity_chars=1024, abort_after_fields=None):
buffer = BoundedBuffer(capacity_chars)
extractor = FieldExtractor()
report = StreamReport(status="complete")
while True:
if (abort_after_fields is not None
and len(report.fields) >= abort_after_fields):
report.status = "aborted"
close = getattr(source, "close", None)
if close is not None:
close() # run the source's cleanup, do not abandon it
break
try:
chunk = next(source)
except StopIteration:
break
except StreamError as exc:
report.status = "error"
report.error = str(exc)
break
# offer, drain into the extractor, record emitted fields...
report.pending_key = extractor.pending_key
return report
The abort path deserves attention. Aborting is not breaking out of the loop and forgetting the source. The consumer calls close() on the generator, which raises GeneratorExit at its current yield point and runs its finally block, where a real client returns its HTTP connection to the pool. Skip that step often enough and you leak connections under exactly the conditions, flaky clients disconnecting early, that cause aborts in the first place. The demo proves cleanup ran, rather than assuming it.
Step 4: Run it
The demo (run_streaming_demo.py) streams a six-field JSON object in 9-character chunks through four scenarios. Actual output:
== Scenario 1: fields emitted as they complete ==
chunk 4: field 'title' -> 'Refund policy update'
chunk 12: field 'summary' -> 'Customers can request a refund within 30 days of purchase.'
chunk 15: field 'sentiment' -> 'positive'
chunk 17: field 'score' -> 0.87
chunk 20: field 'tags' -> ['billing', 'refund']
chunk 25: field 'meta' -> {'model': 'stub-1', 'tokens': 42}
status=complete, 25 chunks, first field at chunk 4 of 25
== Scenario 2: abort after 2 fields ==
status=aborted
delivered: {'title': 'Refund policy update', 'summary': 'Customers can request a refund within 30 days of purchase.'}
chunks consumed: 12 of 25
field in flight when aborted: None
source cleanup ran: True
== Scenario 3: bounded buffer vs a producer that never waits ==
offered 10 chunks of 8 chars into a 32-char buffer
accepted=4, refused=6, size pinned at 32 (capacity 32)
after drain: size=0, drained 32 chars intact
== Scenario 4: provider fails mid-stream ==
status=error, error='provider connection reset mid-stream'
fields delivered before the failure: ['title']
field in flight when it failed: 'summary'
Read scenario 1 the way a user experiences it: the title is usable after chunk 4 of 25, so a UI can render the headline while 84 percent of the response is still in flight. Scenario 2 is the disconnect case: two complete fields survive, the partial third field is never half-parsed, and the source confirms its cleanup ran. Scenario 4 saves debugging sessions: the report says the failure hit during summary, not just that it hit.
When streaming is worth it, and when it is not
Stream when the consumer can act on partial output. Rendering text as it arrives, populating fields as they close, starting a downstream step on the first completed field: these cut latency from full generation time to first-field arrival, and on long responses that difference is the product. Stream also when you need the abort path, because a non-streaming request gives a disconnecting client no way to stop the work.
Wait for the full response when the consumer needs the whole object before doing anything, which is more common than streaming advocates admit. If the next step is a single database write of the complete record, streaming adds a parser, a buffer policy, and three exit paths to buy latency nobody can perceive. Wait also when the response is short: below a few hundred tokens, generation finishes before the machinery pays for itself. And wait when correctness demands validating the whole object against a schema before any of it is shown; a stream that must be fully revalidated at the end has bought nothing.
Real gotchas
Chunk boundaries are adversarial. A chunk can end mid-string, mid-escape, or in the middle of a \u escape. Any consumer logic that pattern-matches on chunk text, looking for a delimiter inside one chunk, will eventually split its needle across a boundary and miss it. Accumulate into the parser and match on parser state, never on raw chunks.
Half-shown output is a product decision, not a bug to hide. When a stream errors at chunk 40, the user has read 39 chunks of an answer that will never finish. Decide in advance what the UI does: mark it incomplete, offer a retry that regenerates, or keep the partial with a visible truncation notice. The one unacceptable option is leaving the partial on screen looking finished. The report's status field exists for exactly this call.
Retries do not compose with streams. A non-streaming call can be retried transparently because nothing was delivered. A stream that delivered six fields cannot be restarted without the user seeing the answer twice. If you retry, retry from a clean slate and say so in the UI, or do not retry at all. Idempotency keys do not save you here; the side effect was on the user's screen.
Backpressure has to reach the socket. A bounded buffer only helps if refusing actually pauses reads from the provider connection. If a background task keeps reading the socket and piling chunks into an internal queue, you have rebuilt the unbounded buffer one layer down. In async code this means awaiting the drain before the next read; in threaded code it means the reader thread blocks on the bounded queue. Audit the whole path, not just your class.
Abort checks happen at chunk boundaries. In scenario 2 the abort landed between fields, so no field was in flight. That is the general case: you can only act on an abort between chunks, which means a single very long chunk delays both aborts and error detection. Providers that emit large chunks under load will make cancellation feel sluggish; all your consumer can do is measure chunk sizes and complain with data.
Wrap-up
Streaming stops being fragile when three contracts are explicit: fields are emitted when they complete, the buffer refuses instead of growing, and every exit, success, abort, or failure, returns the same report describing what was delivered. That is about 250 lines of dependency-free Python, converting the four production failures above into statuses your code already handles. Reach for your provider's higher-level streaming helpers when they offer the same contracts; keep this layer when they do not, or when you need to prove exactly what your abort path delivers.
Top comments (0)