Streaming

Set stream: true and Zumik returns a server-sent event stream of OpenAI-shaped chunks. Tokens arrive as the provider produces them - on every provider, including Anthropic and Gemini - with a clean terminator and (when you ask for it) a trailing usage chunk with the exact token counts you were billed for.

Stream a chat completion

stream = client.chat.completions.create(
    model="code.fast",
    messages=[{"role": "user", "content": "Explain the diff."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage:
        print("\ncached:", chunk.usage.prompt_tokens_details.cached_tokens)

A streamed response sets Content-Type: text/event-stream and Cache-Control: no-cache, and the body is a sequence of data: lines.

The chunk shape

Each event is data: followed by a JSON object whose object is chat.completion.chunk. The frames arrive in this order:

{"id":"chatcmpl-…","object":"chat.completion.chunk","created":1750000000,"model":"code.fast",
 "choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

{"id":"chatcmpl-…","object":"chat.completion.chunk","created":1750000000,"model":"code.fast",
 "choices":[{"index":0,"delta":{"content":"The patch "},"finish_reason":null}]}

{"id":"chatcmpl-…","object":"chat.completion.chunk","created":1750000000,"model":"code.fast",
 "choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

{"id":"chatcmpl-…","object":"chat.completion.chunk","created":1750000000,"model":"code.fast",
 "choices":[],
 "usage":{"prompt_tokens":1240,"completion_tokens":180,"total_tokens":1420,
          "prompt_tokens_details":{"cached_tokens":1024}}}

data: [DONE]

stream_options.include_usage

By default a streamed response carries no usage object - matching OpenAI. Set stream_options.include_usage: true to receive one trailing chunk after the finish frame whose usage block reports prompt_tokens, completion_tokens, total_tokens, and prompt_tokens_details.cached_tokens. The cached count is how you confirm prompt-cache capture on a streamed call.

Token-by-token relay

Streaming is a true relay by default: Zumik opens the provider's own streaming endpoint and pipes chunks through as they arrive, so time to first token is the provider's, not the full generation time. OpenAI, xAI, and Fireworks already speak the OpenAI chunk dialect and relay verbatim. Anthropic (Messages API) and Gemini (streamGenerateContent) stream in their own SSE dialects, and Zumik translates them frame by frame into the same chat.completion.chunk shape - text deltas, tool-call deltas, finish reasons, and all - so your client never sees the difference.

Billing stays exact. Before the stream opens, the request is pre-checked against your budget for the worst case (input plus the requested max_tokens at the published rate), so a near-empty balance is rejected as a normal error rather than cut off mid-stream. The actual charge lands once the stream closes, computed from the provider's own reported usage - the same numbers the trailing usage chunk shows you.

Two paths still fall back to buffered delivery (the full completion fetched, charged, then re-emitted as chunks): a provider route with no usable streaming key, and deployments that set ZUMIK_STREAM_RELAY=0 to opt out of the relay entirely.

On /v1/responses and /v2/responses

The OpenAI-compatible /v1/responses supports stream: true with the Responses SSE event protocol (response.created through response.completed, including response.output_text.delta and function-call argument deltas) — see its streaming section for the event sequence. Under the hood it resolves and charges the generation, then emits the events; for the lowest time-to-first-token use /v1/chat/completions with stream: true, which relays provider tokens live. The native /v2/responses stays buffered and adds execution_profile and a formal qos_outcome to the response object.