The voice on the phone: AI Engineering, story 26
HM Nayem calls the shop and a voice answers. Follow the words as they become text, an answer, and speech again in under a second.
Speech in, speech out, and why every stage has a time budget.
The story, step by step
A phone call
HM Nayem is driving and cannot type, so calls the shop instead. A voice answers at once. It is not a person. Let's follow one question and its answer through the call.
Sound is not text
A voice on a phone line is a wave: thousands of numbers every second describing air pressure. The models you have met read tokens of text. So the most common design is three specialists in a row.
Has HM Nayem finished speaking?
Before anything else, a surprisingly hard question. In a chat, pressing send marks the end of a turn. On a call, nothing does. So the system listens for a short silence. Too impatient, and it cuts people off. Too patient, and every reply feels slow.
Sound into words
The sound goes to a speech-to-text model, which writes down what it hears. It does not wait for the end. It transcribes while HM Nayem is still talking, so the words are ready the instant the silence comes.
The part you already know
From here it is what you have seen many times. The text goes to the language model with the shop's instructions. It asks for an order lookup, gets the result and writes a reply.
Words into sound
Now the reverse trip. A text-to-speech model reads the reply and produces the sound of a voice saying it. That sound travels down the line to HM Nayem's ear.
The real problem is time
In a text chat, waiting three seconds is fine. On a call, a silence of more than about a second makes people say “hello? are you there?” So the whole chain gets a budget of one second, and every step has to fit its share.
Do not wait for the whole answer
The fix is the idea from the very first story: streaming. As soon as the model has written its first few words, they go to the speech model, which starts talking while the rest is still being written. Nobody waits for anybody to finish.
HM Nayem interrupts
Halfway through the reply, HM Nayem talks over it: “Wait, can I change the address?” People do this constantly. The assistant has to stop speaking that instant and start listening again.
What gets lost in the words
Turning a voice into text throws things away. A sigh, a worried tone, a laugh: all gone. And mistakes creep in. Was that “order four eight two one” or “for eight to one”? Good voice assistants read important details back to be sure.
Another way: one model for speech
Newer models take speech in and give speech out directly, with no text in the middle. They are faster and keep the tone of voice. The price is that there is no written text halfway to check, filter or log.
Now at real speed
Sound in, words, a reply, sound out, with the first of it reaching HM Nayem's ear in under a second. A voice assistant is the chatbot you already understand, with a model on each end and a stopwatch running.
A voice assistant is a chatbot racing a one-second clock.
- Speech becomes text, a language model replies, and the reply becomes speech.
- The system has to judge when the caller has finished, by listening for silence.
- A pause over about a second feels broken, so every step has a time budget.
- Streaming lets the voice start before the reply is finished.
- When the caller interrupts, the assistant must stop at once.
Ideas in this story
- Voice assistants: Speech turned into text, answered by a model, and turned back into speech, fast enough to feel like talking.
- A time budget: A fixed amount of time a reply may take, shared out between every step on the way.
- Streaming: The reply is sent piece by piece as it is written, instead of all at once at the end.