Streaming looks solved. You flip stream=True, loop over deltas, append text to a div, and the demo feels magical. Then it ships, and the bug reports start. The response cuts off halfway on flaky mobile connections. The JSON you were parsing mid-stream throws because you got half a key. A tool call fires but your renderer already committed to showing text. A user hits stop and the backend keeps burning tokens for another thirty seconds.

None of that is a streaming problem. Streaming the tokens is the easy 20%. The other 80% is state management over an unreliable, half-complete byte stream, and almost nobody writes about it. This post walks the failure modes in the order you actually hit them in production, and gives you the pattern that fixes each one.

The One-Liner That Fools You

Here is the version that ships in every tutorial:

from anthropic import Anthropic

client = Anthropic()

with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": prompt}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

On the frontend it is just as short. You open a fetch with a readable stream, decode chunks, and paint them. It works on your laptop on office wifi against a healthy API. Everything past this point is what breaks that illusion.

The core issue: a stream is a sequence of partial states, and your UI has to render every one of them without ever showing something malformed, and recover cleanly when the sequence is interrupted. That is a different job than “print the tokens.”

Failure Mode 1: You Are Parsing JSON That Is Not Finished Yet

The moment you want structured output while streaming, you are in trouble. Say the model is returning a JSON object and you want to render fields as they arrive so the UI does not sit blank. You get deltas like this:

{"tit
{"title": "Quarterly
{"title": "Quarterly Report", "sta
{"title": "Quarterly Report", "status": "dra
{"title": "Quarterly Report", "status": "draft"}

Every intermediate state except the last is invalid JSON. If you call json.loads on any of them, you get a JSONDecodeError. The naive fix is to wrap it in try/except and only parse when it succeeds. That works but it defeats the point: you buffer the entire response, parse once at the end, and the user stared at nothing the whole time. You threw away streaming to keep your parser happy.

The real fix is a parser that tolerates truncation and completes the structure for you. You close open strings, arrays, and objects with a small stack, then parse the repaired string.

def repair_partial_json(buf: str) -> str:
    """Best-effort close of an incomplete JSON fragment so it parses."""
    stack = []
    in_string = False
    escaped = False

    for ch in buf:
        if escaped:
            escaped = False
            continue
        if ch == "\\" and in_string:
            escaped = True
            continue
        if ch == '"':
            in_string = not in_string
        elif not in_string:
            if ch in "{[":
                stack.append(ch)
            elif ch in "}]" and stack:
                stack.pop()

    repaired = buf
    if in_string:
        repaired += '"'
    # a trailing comma or dangling key makes this fail; drop to last safe point
    repaired = repaired.rstrip().rstrip(",")
    for opener in reversed(stack):
        repaired += "}" if opener == "{" else "]"
    return repaired

Then on each delta you attempt a parse of the repaired buffer and keep the last version that succeeded:

import json

buffer = ""
last_valid = {}

for delta in stream:
    buffer += delta
    try:
        last_valid = json.loads(repair_partial_json(buffer))
    except json.JSONDecodeError:
        pass  # not enough yet, keep the previous render
    render(last_valid)

This is not a toy. It is exactly how streaming structured UIs work: the fields pop in as they complete, and a half-typed value simply shows the partial string rather than blowing up. Libraries like partial-json-parser and Vercel’s parsePartialJson do the same thing with more edge cases handled. Do not hand-roll the full grammar; do understand what it is doing, because you will debug it.

One rule that saves you: never render a numeric or boolean field until it is closed. A partial "total": 14 might become 142.50, and showing 14 then 142 then 142.5 looks like a bug. Only commit scalars when the token after them proves they are complete.

Failure Mode 2: Tool Calls Arrive In The Middle Of Text

Streaming gets genuinely hard the moment the model can call tools. The response is no longer a flat token stream. It is a sequence of typed events: text starts, text deltas, text stops, a tool-use block starts, its arguments stream in as partial JSON, the tool block stops, then possibly more text after the tool result comes back.

If your renderer assumes “everything is text,” a tool call corrupts the display. You need an event-driven state machine, not a string accumulator. The Anthropic SDK exposes these as raw events, and you route on the block type:

with client.messages.stream(model="claude-sonnet-4-6", max_tokens=2048,
                             tools=tools, messages=messages) as stream:
    for event in stream:
        if event.type == "content_block_start":
            block = event.content_block
            if block.type == "text":
                begin_text_block()
            elif block.type == "tool_use":
                begin_tool_block(block.id, block.name)

        elif event.type == "content_block_delta":
            d = event.delta
            if d.type == "text_delta":
                append_text(d.text)
            elif d.type == "input_json_delta":
                # tool arguments stream as partial JSON, same problem as mode 1
                append_tool_args(d.partial_json)

        elif event.type == "content_block_stop":
            finalize_current_block()

Two things to internalize. First, tool arguments stream as partial JSON, so the repair pattern from failure mode 1 applies here too if you want to show “searching for quarterly rep…” while the arg is still assembling. Second, index matters. In a single response the model can open multiple blocks, each with an index. If you flatten by name you will merge two parallel tool calls into garbage. Key your in-progress buffers by block index, not by tool name.

The UX payoff is real: instead of the interface freezing while a tool runs, you show “Calling search_docs” the instant the block opens, stream the arguments into a readable status line, then swap to the result. The user always sees forward motion.

Failure Mode 3: The Transport Itself

You have to move these events from server to browser. Three options, and most teams pick the wrong one out of habit.

TransportDirectionReconnectBest for
Plain fetch + ReadableStreamServer to clientManualSimple, one-shot generations
Server-Sent Events (SSE)Server to clientBuilt in, automaticStreaming chat, the default choice
WebSocketBidirectionalManualVoice, interrupts, collaborative sessions

For text generation, SSE is almost always right and almost always underused. It is a plain HTTP response with Content-Type: text/event-stream, it survives proxies and load balancers that mangle WebSockets, and the browser’s EventSource reconnects on its own and tells the server where to resume via the Last-Event-ID header. WebSockets give you a bidirectional channel you usually do not need, plus a reconnection story you have to build yourself.

The one thing SSE’s EventSource cannot do is send a request body or custom headers, which matters for auth. In practice you either use a POST with fetch and parse the SSE frames manually, or you use a library like @microsoft/fetch-event-source that gives you SSE semantics over fetch. Server side, the frame format is dead simple:

async def event_stream(prompt: str):
    seq = 0
    async with client.messages.stream(
        model="claude-sonnet-4-6", max_tokens=2048,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        async for text in stream.text_stream:
            seq += 1
            yield f"id: {seq}\n"
            yield f"event: delta\n"
            yield f"data: {json.dumps({'text': text})}\n\n"
    yield "event: done\ndata: {}\n\n"

Emit an id: on every frame. That is what makes reconnection possible, which is the next failure mode.

Failure Mode 4: The Connection Dies Mid-Response

This is the one that generates the most support tickets and the one demos never surface, because demos run on good networks. On mobile, connections drop constantly: tunnels, elevators, network handoffs. If a stream dies at 60% and your UI just stops, the user sees a truncated answer with no error and assumes your product is broken.

There are two honest strategies, and the right one depends on whether your generation is idempotent.

Resume from a checkpoint. The server buffers the full generation keyed by a request ID and tracks how many events it has sent. On reconnect the client sends Last-Event-ID, and the server replays only the frames after that point. This is the correct approach for a single generation you do not want to pay for twice.

# server holds an append-only log of frames per request_id
async def stream_with_resume(request_id: str, last_event_id: int):
    log = await get_or_create_log(request_id)

    # replay everything the client missed
    for frame in log.frames_after(last_event_id):
        yield frame.render()

    # if generation already finished while disconnected, we are done
    if log.complete:
        yield "event: done\ndata: {}\n\n"
        return

    # otherwise keep forwarding new frames as they are produced
    async for frame in log.follow():
        yield frame.render()

The client side that drives it:

let lastEventId = 0;

async function connect(requestId) {
  const res = await fetch(`/stream/${requestId}`, {
    headers: { "Last-Event-ID": String(lastEventId) },
  });
  const reader = res.body.getReader();
  const decoder = new TextDecoder();

  try {
    for (;;) {
      const { value, done } = await reader.read();
      if (done) break;
      for (const frame of parseSSE(decoder.decode(value, { stream: true }))) {
        if (frame.id) lastEventId = Number(frame.id);
        if (frame.event === "delta") appendText(frame.data.text);
        if (frame.event === "done") return;
      }
    }
  } catch (err) {
    // network dropped mid-stream: back off and resume from lastEventId
    await sleep(backoff());
    return connect(requestId);
  }
}

Because the server buffers and the client tracks lastEventId, a reconnect stitches the response back together with no duplicate text and no gap. The user sees a brief pause, not a broken answer.

Regenerate from scratch. If you did not buffer, the only option is to rerun the prompt. This costs tokens again and, because sampling is nondeterministic, produces different text, so you cannot naively append it to what the user already saw. If you go this route, replace the whole message rather than continue it, and set temperature=0 if you need the regeneration to at least resemble the original. Honestly, buffering and resuming is worth the complexity for anything user-facing.

A subtle but important detail: buffer with a TTL and a size cap. A per-request log that lives for five minutes and holds at most a few hundred KB is fine. One that lives forever is a memory leak that looks like a slow crash.

   Client                         Server
     |                              |
     |--- POST /stream ------------>|  create log(request_id)
     |<-- id:1 delta "The" ---------|  append frame 1
     |<-- id:2 delta " qu" ---------|  append frame 2
     |         X  network drop      |  generation continues,
     |                              |  frames 3..7 buffered
     |--- reconnect ---------------- |
     |    Last-Event-ID: 2          |
     |<-- id:3..7 (replayed) -------|  replay from log
     |<-- id:8 delta (live) --------|  resume live tail
     |<-- event: done --------------|

Failure Mode 5: Cancellation That Does Not Actually Cancel

A user hits stop. On the frontend you clear the reader and hide the spinner. Feels done. But if you did not propagate the cancel to the model provider, the backend is still generating, still holding the upstream connection open, and still billing you for every token until max_tokens. At scale this is a real line item and a real source of connection-pool exhaustion.

Cancellation has to travel the whole chain: UI abort, to your server, to the provider. On the browser, use an AbortController:

const controller = new AbortController();
stopButton.onclick = () => controller.abort();

await fetch(`/stream/${requestId}`, { signal: controller.signal });

Aborting the fetch closes the HTTP connection. Your server has to notice that disconnect and break out of its generation loop, which then closes the upstream stream to the provider. In FastAPI, check await request.is_disconnected(); in the Anthropic SDK, exiting the with stream block or breaking the async iteration closes the upstream request. The chain only works if every link forwards the signal. Test it by watching your provider dashboard: hit stop, and token consumption for that request should flatline within a second, not keep climbing.

What Actually Works

Streaming is not one feature, it is five separate reliability problems wearing a trench coat. Here is the honest scorecard after building this more than once.

  • Partial JSON repair works and you should adopt it, but use a maintained library instead of your own parser, and refuse to render scalars until they are closed. This is the single biggest win for structured streaming UIs.
  • Event-driven rendering for tool calls is mandatory the moment tools enter the picture. Key buffers by block index. A string accumulator will silently corrupt multi-tool responses, and you will lose an afternoon before you see why.
  • Use SSE for text, not WebSockets, unless you genuinely need to send data back mid-stream. The automatic reconnection and proxy friendliness are worth more than the bidirectional channel you probably will not use.
  • Resume-from-checkpoint is the difference between “reliable” and “demo,” and it is the piece teams skip because it never fails locally. Buffer frames per request with a TTL, emit an id on every frame, honor Last-Event-ID. Do this before launch, not after the first angry mobile user.
  • Cancellation has to reach the provider or it is theater. Verify it on the billing dashboard, not in the UI.

What does not work: treating the stream as a string, parsing only at the end, or assuming the network is your laptop’s network. The magic-feeling demo and the production system share about ten lines of code. The rest is the boring plumbing that decides whether users trust the thing. Build the plumbing first.