Streaming Response Studio

TTFT—

tok/s—

tokens0

elapsed0.0s

80 text tokens. Shows TTFT and token rate with nothing else in the way.

Speed20 tok/s

Non-streaming

⌁

Select a preset and click Send to stream

Tokens will appear here as the stream arrives

The 5 Hard Problems — Toggle to see the difference

Streaming Markdownsolved

Problem: Partial Markdown

react-markdown renders incomplete fences gracefully

Tool Fragment Accumulatorsolved

Problem: Tool-call fragments

Buffer fragments, try-parse, resolve when JSON is complete

Structured Output Parsersolved

Problem: Structured output mid-stream

Regex detects resolved key:value pairs incrementally

Mid-Stream Error Handlingsolved

Problem: Mid-stream errors

Error event renders inline ErrorInline with message

Render Storm (OFF = bad)solved

Problem: Render storms

rAF batching — accumulate in ref, flush at ~60fps

How streaming works

TTFT = prefill latency

The model runs a forward pass over your entire prompt before generating token 1. TTFT measures this. Streaming cuts perceived latency by exposing the decode loop immediately after prefill.

SSE wire format

Every major provider (OpenAI, Anthropic, Gemini) uses Server-Sent Events. Each event is data: {…}\n\n. Chunks from ReadableStream don't align with event boundaries — the consumer must buffer.

AbortController

Calling abort() signals the stream factory to stop emitting. Partial text stays visible. In production, this cancels the HTTP request, saving GPU time and reducing costs.

rAF batching

Accumulate tokens in a ref, flush to state on requestAnimationFrame. Decouples the 30–80 tok/sec stream rate from React reconciler. Without this, you flood the scheduler and drop frames.