Streaming
Server-sent events from POST /chat/completions: the chunk format, the final usage chunk, errors during a stream, and what cancelling does and does not save.
Set "stream": true on a chat completion to receive the answer as it is generated. Add "stream_options": {"include_usage": true} to receive the token counts and cost at the end.
Examples
curl -N https://api.safeguard.sh/gpt/v1/chat/completions \
-H "Authorization: Bearer $SAFEGUARD_INFERENCE_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "MODEL_ID",
"messages": [{"role": "user", "content": "Write a haiku about dependency updates."}],
"stream": true,
"stream_options": {"include_usage": true}
}'What the stream looks like
The response has Content-Type: text/event-stream, Cache-Control: no-cache and an X-Generation-Id header holding the request id (gen-…), the same id every chunk carries. Each event is one data: line followed by a blank line. The stream ends with data: [DONE]. The values below are illustrative.
data: {"id": "gen-6f1c2e0a9b4d4c7e8a1f3b5d7c9e0a2b", "object": "chat.completion.chunk", "created": 1791475000, "model": "MODEL_ID", "provider": "openai", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "Lockfile"}, "finish_reason": null}]}
data: {"id": "gen-6f1c2e0a9b4d4c7e8a1f3b5d7c9e0a2b", "object": "chat.completion.chunk", "created": 1791475000, "model": "MODEL_ID", "provider": "openai", "choices": [{"index": 0, "delta": {"content": " turns green"}, "finish_reason": null}]}
data: {"id": "gen-6f1c2e0a9b4d4c7e8a1f3b5d7c9e0a2b", "object": "chat.completion.chunk", "created": 1791475000, "model": "MODEL_ID", "provider": "openai", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}
data: {"id": "gen-6f1c2e0a9b4d4c7e8a1f3b5d7c9e0a2b", "object": "chat.completion.chunk", "created": 1791475000, "model": "MODEL_ID", "provider": "openai", "choices": [], "usage": {"prompt_tokens": 16, "completion_tokens": 18, "total_tokens": 34, "cost": 8.8e-05}}
data: [DONE]Every chunk carries the same id, created, model and provider. choices[0].delta holds the new piece of the answer, including tool_calls pieces when the model calls a tool. The gateway leaves out chunks that carry no change.
The usage chunk
With stream_options.include_usage set, the last chunk before [DONE] has an empty choices array and a usage object with the same fields as a non-streamed response: prompt_tokens, completion_tokens, total_tokens, cost, and prompt_tokens_details or estimated when they apply. Without the option, the stream has no usage chunk, and the request is billed all the same.
Some providers do not report usage in a stream. The gateway then counts the tokens from the text itself, bills that estimate, and marks the usage "estimated": true.
Errors
Before the first chunk. Until the provider has produced its first chunk, nothing is sent to you, so a refusal or a provider failure comes back as an ordinary HTTP error with a JSON body, exactly as for a non-streamed request. When you allow fallbacks, the gateway's retry and fallback happen in this window too. See Errors.
After the stream has started. The HTTP status is already 200. If the provider's stream breaks, the gateway sends an error event, then the usage chunk if you asked for one, then [DONE]:
data: {"error": {"message": "The provider stream ended with an error.", "type": "api_error", "code": "upstream_error", "param": null}}A stream that runs past 900 seconds ends the same way with the code upstream_timeout. The OpenAI SDKs raise an error when they read an error event, so handle it as you would a failed request. The request is billed for what the provider produced before the failure; when the provider reported no usage, that is an estimate marked "estimated": true.
If the provider's stream ends without a finish reason and without an error, the gateway sends a last chunk with "finish_reason": "stop".
Cancelling a stream
To stop early, close the connection: break out of the loop and call stream.close() in the Python SDK, or call stream.controller.abort() in the Node.js SDK.
Closing the connection stops the gateway sending to you. It does not stop the provider. The gateway keeps reading the provider's answer to the end, for up to 900 seconds, so that it can bill the provider's own token counts, and the request is billed for everything the provider generated. To limit what a request can cost, set max_completion_tokens or max_tokens.
The Stop button in the Console Workbench behaves the same way.