Streaming

Start a stream

Set the JSON boolean stream: true on chat completions, responses or messages. Use curl -N to disable client buffering. Image generation does not stream. Replace the model placeholder with a compatible enabled ID; SDK examples use LLMFLUX_MODEL.

curl -N https://api.llmflux.dev/v1/chat/completions \
  -H "Authorization: Bearer $LLMFLUX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "YOUR_MODEL_ID",
  "messages": [
    {
      "role": "user",
      "content": "Hello from LLMFLUX"
    }
  ],
  "stream": true
}'

Wire format

The gateway sends HTTP 200 with Content-Type: text/event-stream, Cache-Control: no-cache, no-transform and X-Accel-Buffering: no. SSE bytes pass through unchanged. Chat completions commonly use data: chunks and [DONE]; Responses and Messages use their own event schemas. Do not assume every endpoint has chat-style choices[].delta.

A network chunk is not an event boundary. Buffer until complete SSE frames are available, or use the SDK iterator/parser. Reverse proxies and clients must also avoid buffering.

Usage

For chat completions, the gateway forces stream_options.include_usage: true. Usage is accumulated from complete data: lines with top-level usage or message.usage. OpenAI prompt/completion tokens and Anthropic split input/output/cache fields are recognized. Metering depends on the provider sending a recognized usage frame; not every Responses event schema is covered. Do not infer a free request from absent usage or assume closing a client stream guarantees upstream cancellation.

Stream errors

Before streaming begins, failures use the normal JSON error envelope. Once HTTP 200 has been sent, a read failure produces this SSE event instead of changing the HTTP status:

event: error
data: {"error":{"message":"The provider returned an error.","type":"api_error","code":"upstream_error"}}

Handle provider error events too: they are forwarded as-is. Treat partial output as incomplete on failure. The gateway can fail over while opening a stream, but it does not resume or replay a stream after output starts. Retrying can create another metered request.