What is streaming?
A model writes its reply one token at a time. Without streaming, you wait until the last token is written and then receive everything together.
With streaming, each token is forwarded as soon as it exists. The total time is the same, but the first words appear in a fraction of a second, so the product feels fast.
For this to work, every step between the model and the screen has to pass tokens along immediately. One step that waits for the full reply removes the effect.
Why does streaming matter?
A model writes a long answer over several seconds. If nothing appears until the last word is ready, the product feels broken. Streaming shows each piece the moment it is written, so the first words appear in a fraction of a second. The total time is unchanged. What changes is how long a person waits before something happens.
How does streaming work?
- Your server makes the call with streaming switched on and keeps the connection open.
- The provider sends each token, or small group of tokens, as soon as the model produces it.
- Your server passes each piece straight on to the person's screen, where it is added to the text.
- A final message says the reply is complete and gives the token counts.
When should you use streaming?
Use it when
- A person is watching and waiting for the reply
- Replies are longer than a sentence or two
- Another stage, such as a voice, can start work on the first sentence
Think twice when
- A program will read the whole reply, such as a filled-in form
- The reply must be checked in full before anyone may see it
Where does streaming sit in a real system?
Streaming runs along the whole path: provider to your server, server to the browser or phone. Every link has to pass pieces on without waiting. A single part that collects the full reply before forwarding it undoes the benefit.
What goes wrong with streaming?
- Showing text to a person before a safety check has seen it. Check as you go, a sentence behind, or do not stream.
- Trying to read structured data from a half-finished reply.
- Measuring only total time. For a streamed reply the number that matters is time to the first word.
What does streaming look like in practice?
A 150-word answer takes four seconds to write. Without streaming the customer watches a spinner for four seconds. With streaming the first word appears after 0.4 seconds and the text flows at reading speed. Both took four seconds. Only one felt slow.