Streaming

Set stream: true and the response body becomes text/event-stream. Each event is a line data: <JSON> followed by a blank line. The JSON is a ChatCompletionChunk. The stream ends with data: [DONE].

data: {"id":"chatcmpl-0192a1b2c3d4","object":"chat.completion.chunk","created":1758700000,"model":"polish-pro","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-0192a1b2c3d4","object":"chat.completion.chunk","created":1758700000,"model":"polish-pro","choices":[{"index":0,"delta":{"content":"Warszawo, "},"finish_reason":null}]}

data: {"id":"chatcmpl-0192a1b2c3d4","object":"chat.completion.chunk","created":1758700000,"model":"polish-pro","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-0192a1b2c3d4","object":"chat.completion.chunk","created":1758700000,"model":"polish-pro","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":40,"total_tokens":52}}

data: [DONE]

The usage chunk has no choices

When you send stream_options: {"include_usage": true}, the last chunk before [DONE] has choices: [] and a usage object. Code that reads chunk.choices[0] without checking will throw on that chunk. Guard it:

for chunk in stream:
    if chunk.choices:
        text = chunk.choices[0].delta.content or ""
    elif chunk.usage:
        total = chunk.usage.total_tokens

for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content ?? "";
  if (chunk.usage) total = chunk.usage.total_tokens;
}

Without include_usage you never receive the usage chunk, but the request is billed all the same.

Timeouts

The gateway waits up to 180 seconds for the first byte from the model server — long enough to cover a cold start, when a GPU has to spin up before it can serve anything. Set your HTTP client’s read timeout to at least that, and show a “thinking” state in your UI rather than failing at 30 seconds.

A separate 600-second total budget starts the moment the gateway sends the request upstream — before it connects, before the first byte, before any streaming — and covers the whole attempt on that deployment. Time spent waiting out a cold start counts against this budget too, so a slow first byte leaves that much less of the 600 seconds for the streaming that follows it.

Both the first-byte and the total timeout can fire before any byte has reached you, and either way you get a normal JSON error (502 upstream_timeout), the same as any other pre-first-byte failure. Only a first-byte timeout gets one transparent failover to another deployment; a total-timeout expiry does not retry.

Once the first byte has arrived, a separate 60-second idle timeout takes over: it fires only if the gap between chunks goes over 60 seconds. By then the HTTP status is already 200, so this doesn’t produce a JSON error — it just ends the stream without [DONE], the same as any other mid-stream failure; see Errors during a stream below.

Disconnects

If your client closes the connection mid-stream — a page navigation, a killed process, a client-side timeout — the gateway detects it and cancels the upstream request. You are still charged: for the tokens produced up to that point.

Because a cancelled request never gets a usage report from the model server, the gateway estimates it instead: output tokens are counted as the number of content chunks you received, and input tokens are estimated from the length of your messages (characters, not bytes, divided by 3.5). The usage shown for that request is marked estimated in the console rather than exact.

A disconnect before the very first token arrives — for example while a model is still cold-starting — is charged the estimated input tokens only, since there’s no output to count.

There is no way to resume a stream. Retry the whole request if you need the complete answer.

Errors during a stream

An error that happens before the first byte reaches you — a bad key, an empty balance, an unknown model, every deployment of a model being down — arrives as a normal JSON error response with the matching HTTP status, exactly as it would for a non-streaming request; see Errors. If a deployment fails before sending its first byte, the gateway automatically retries once on another deployment of the same model, transparently to you.

Once a byte has reached you, the HTTP status is already 200 and the gateway will not retry. If the model server then fails, the stream simply stops: no more data: lines, no data: [DONE], and no in-band error event. This is the failure mode most worth guarding against, because the official OpenAI SDKs don’t raise an exception for it — they just stop iterating — so it’s easy for your code to treat a truncated answer as a complete one.

Detecting it:

• Reading raw SSE yourself: a stream that never sends a data: [DONE] line failed.
• Reading through an SDK, which hides [DONE] from you: track whether any choice ever carried a finish_reason. If none did by the time iteration ends, the stream failed before it finished.

finished = False
pieces = []
for chunk in stream:
    if chunk.choices:
        delta = chunk.choices[0].delta.content
        if delta:
            pieces.append(delta)
        if chunk.choices[0].finish_reason:
            finished = True

answer = "".join(pieces)
if not finished:
    raise RuntimeError("stream ended without finishing; answer may be truncated")

let finished = false;
let answer = "";
for await (const chunk of stream) {
  if (chunk.choices.length) {
    answer += chunk.choices[0].delta?.content ?? "";
    if (chunk.choices[0].finish_reason) finished = true;
  }
}
if (!finished) {
  throw new Error("stream ended without finishing; answer may be truncated");
}

You’re billed for whatever the model produced before the failure — the same estimated-usage accounting described under Disconnects above, since a stream that fails this way never reaches a real usage chunk either.