Streaming makes an AI feature feel responsive because the interface can show useful output before the model has finished. It also turns one request into a long-lived connection with cancellation, buffering, partial output, authentication, and moderation concerns. This tutorial shows a small team how to build the boundary deliberately: the browser consumes Server-Sent Events, the Node.js API owns authorization and provider details, and the model stream is treated as untrusted, incomplete input.
Choose the transport and define its contract
This example uses a POST request to start generation and a response with `Content-Type: text/event-stream`. Server-Sent Events are a one-way server-to-browser connection, which fits token or text deltas flowing in one direction while the original prompt is already on the request. MDN specifies UTF-8 event data, messages separated by a blank line, and the `data:` field for payloads.
Use a small event vocabulary rather than forwarding provider packets directly. For example: `start` with a request ID, `delta` with text, `done` with usage or finish metadata, and `error` with a safe public code. The client should be able to render a partial answer and know whether it completed, was cancelled, or failed. Keeping the provider format behind your API makes a model or vendor change a contained server-side change.
- Node.js API with an authenticated route
- A browser client that can use `fetch()` and `ReadableStream`
- A model SDK or HTTP client that exposes an async stream
- A request timeout, usage limit, and cancellation policy
- A test prompt that does not contain customer secrets
Build the Node.js streaming boundary
Authenticate and authorize before opening the stream. Validate the prompt size, tenant, model allowlist, and any output limits before making a provider call. Then set `Content-Type: text/event-stream; charset=utf-8`, `Cache-Control: no-cache`, and a connection policy appropriate to your proxy. Send each event as JSON followed by two newline characters, for example `event: delta\ndata: {"text":"hello"}\n\n`. Do not put API keys or internal provider errors into the event data.
In a Node HTTP handler, the core loop is conceptually `for await (const chunk of providerStream) { response.write(formatEvent("delta", extractText(chunk))); }`. The exact extraction depends on the SDK, so keep it in one adapter. Flush a small initial `start` event, not arbitrary padding, and add periodic comment heartbeats such as `: keep-alive\n\n` only when your infrastructure requires them. A heartbeat keeps an idle connection visible but is not a substitute for a timeout.
Wire the request abort signal to the provider request and stop consuming when the client disconnects. Otherwise a user who navigates away can leave a model call running and still incur cost. Make the handler close its resources in a `finally` block, and ensure the `done` event is emitted only for a normal completion—not after an error or cancellation.
Consume chunks safely in the browser
For a POST-based stream, use `fetch()` rather than `EventSource`, because the browser EventSource constructor is designed around a URL and does not provide a request body. MDN documents that a fetch response exposes a readable stream and that a reader returns `{ value, done }` chunks. Decode incrementally with one `TextDecoder`, append incoming text to a buffer, split complete SSE messages on the blank-line delimiter, and retain an incomplete final fragment for the next read.
Parse only complete events and JSON-decode the `data` field inside a guarded error path. On `delta`, append text to the visible answer. On `done`, mark the answer complete and record the request ID. On `error`, show a retry or fallback action without replacing already-rendered text with an invented answer. Use an `AbortController` for a Stop button; MDN also documents cancellation of a fetch stream with an abort signal.
- Keep one decoder for the lifetime of the response
- Do not assume one network chunk equals one SSE event
- Escape or text-render model output; do not inject it as HTML
- Disable duplicate submit actions while a request is active
- Show a partial-response state distinct from a completed answer
Handle backpressure, proxies, and limits
Streaming is not automatically faster. It improves time to first visible output, while total generation time, token cost, and correctness remain separate measures. A slow browser or proxy can buffer writes, so test through the same reverse proxy and CDN path used in production. Disable response buffering where your platform requires it, and verify that compression does not turn small events into long silent batches.
Put hard limits around the connection: maximum prompt bytes, maximum output tokens, maximum wall-clock duration, maximum concurrent generations per user or tenant, and an overall provider budget. A long-lived request consumes application resources even when few bytes are moving. Use a queue or background job for work that must finish reliably after the user leaves; do not pretend an interactive stream is durable execution.
SSE has operational trade-offs. MDN notes a low per-browser, per-domain connection limit when HTTP/2 is not used. That matters if one page opens many streams or a product has several live panels. Prefer one multiplexed application stream where appropriate, or use WebSockets when the product genuinely needs bidirectional messaging rather than one-way output.
Treat partial output as a safety and product problem
A streamed answer is visible before you have the complete output. OpenAI’s streaming guidance warns that moderation is more difficult because partial completions are harder to evaluate and moderation scores arrive after the full output. Decide whether your use case can show text immediately, needs a review buffer, or should stream only low-risk drafts. Never stream a proposed tool call directly into an irreversible action.
Keep model access server-side, redact sensitive context, and attach the authenticated tenant and request ID to logs. Store only what your retention policy allows. If the provider fails after several deltas, label the result as incomplete and offer retry; do not silently persist it as a final answer. If you need citations, structured fields, or a decision, wait for a validated final object or run a second validation step rather than treating the visible text as a reliable record.
Verify the failure modes before shipping
Test the complete route through a real HTTP client, not only a mocked provider function. Confirm the first `start` event arrives promptly, multiple deltas render in order, split delimiters and multi-byte characters decode correctly, and the client can cancel without leaving a provider request running. Test a provider timeout, malformed provider chunk, proxy buffering, browser refresh, duplicate submission, and a response that ends without `done`.
Monitor time to first event, total stream duration, completion and cancellation rates, disconnects, provider errors, output-limit terminations, tokens or cost, and concurrent streams by tenant. A useful production checklist is: authorization happens before generation; the provider key never reaches the browser; limits are enforced server-side; abort propagates; partial output is labeled; errors are safe; logs have correlation IDs; and a non-streaming fallback exists for clients or networks that cannot sustain the connection.
- Use `curl -N` or an equivalent client to confirm events are not buffered
- Assert event ordering and verify the final state for normal, error, and cancel paths
- Test a client disconnect while the provider is still generating
- Measure first-byte latency separately from total completion latency
- Red-team prompt and document inputs if the stream is part of an agent workflow
- Keep a feature flag so streaming can be disabled without a redeploy