StoryboardAI Engineering

A time budget

A fixed amount of time a reply may take, shared out between every step on the way.

What is a time budget?

In a text chat, a three-second wait is fine. On a phone call, a silence longer than about a second feels like the line has dropped. So a voice product has a time budget, and every step must fit inside its share.

Engineers write the budget down step by step: waiting for silence, speech to text, the model's first words, the first sound. If the shares add up to more than the total, something has to get faster or be cut.

The main trick is streaming. No step waits for the one before to finish. The speech model starts talking as soon as the first sentence exists, while the rest is still being written.

One second, divided between the stages that share it.1. Know they have finished: 0.3 s. 2. Speech to text: 0.1 s. 3. First sentence written: 0.4 s. 4. First sound of the voice: 0.2 s. 5. Total the caller waits: 1.0 s.Know they have finished0.3 sSpeech to text0.1 sFirst sentence written0.4 sFirst sound of the voice0.2 sTotal the caller waits1.0 s
One second, divided between the stages that share it.

Why does a time budget matter?

A reply passes through several stages, and each one adds delay. Without a target, every stage is a little slow and the total is unbearable. Deciding the acceptable total first, then giving each stage a share, turns speed from a hope into something that is designed and measured.

How does a time budget work?

  1. Decide the longest acceptable wait for the person, such as one second to the first sound.
  2. List every stage between their request and the first thing they see or hear.
  3. Give each stage a share of the total.
  4. Measure each stage on every request.
  5. When a stage goes over its share, that stage is where the work is.

When should you use a time budget?

Use it when

  • People wait for the reply in real time, especially by voice
  • A reply passes through several models or tools
  • Speed complaints are vague and you need to know where the time goes

Think twice when

  • Work runs in the background and nobody is waiting

Where does a time budget sit in a real system?

The budget is a design decision recorded with the system. The measurements come from the harness, which times each stage and stores the result with every request.

What goes wrong with a time budget?

  • Measuring only the average. The slowest one in twenty is what people remember.
  • Optimising a stage that was already inside its share.
  • Counting total time when the person only cares about the first word.

What does a time budget look like in practice?

A voice assistant has one second from the caller's last word to its first sound. The shares: 0.3 seconds to be sure the caller has finished, 0.1 for speech to text, 0.4 for the first sentence, 0.2 for the first sound. Each is measured on every turn.

Watch it happen

These stories show a time budget at work, one step at a time.