AI Engineering, story 26The voice on the phone

A phone call

1 of 12

The voice on the phone: AI Engineering, story 26

HM Nayem calls the shop and a voice answers. Follow the words as they become text, an answer, and speech again in under a second.

Speech in, speech out, and why every stage has a time budget.

The story, step by step

  1. A phone call

    HM Nayem is driving and cannot type, so calls the shop instead. A voice answers at once. It is not a person. Let's follow one question and its answer through the call.

  2. Sound is not text

    A voice on a phone line is a wave: thousands of numbers every second describing air pressure. The models you have met read tokens of text. So the most common design is three specialists in a row.

  3. Has HM Nayem finished speaking?

    Before anything else, a surprisingly hard question. In a chat, pressing send marks the end of a turn. On a call, nothing does. So the system listens for a short silence. Too impatient, and it cuts people off. Too patient, and every reply feels slow.

  4. Sound into words

    The sound goes to a speech-to-text model, which writes down what it hears. It does not wait for the end. It transcribes while HM Nayem is still talking, so the words are ready the instant the silence comes.

  5. The part you already know

    From here it is what you have seen many times. The text goes to the language model with the shop's instructions. It asks for an order lookup, gets the result and writes a reply.

  6. Words into sound

    Now the reverse trip. A text-to-speech model reads the reply and produces the sound of a voice saying it. That sound travels down the line to HM Nayem's ear.

  7. The real problem is time

    In a text chat, waiting three seconds is fine. On a call, a silence of more than about a second makes people say “hello? are you there?” So the whole chain gets a budget of one second, and every step has to fit its share.

  8. Do not wait for the whole answer

    The fix is the idea from the very first story: streaming. As soon as the model has written its first few words, they go to the speech model, which starts talking while the rest is still being written. Nobody waits for anybody to finish.

  9. HM Nayem interrupts

    Halfway through the reply, HM Nayem talks over it: “Wait, can I change the address?” People do this constantly. The assistant has to stop speaking that instant and start listening again.

  10. What gets lost in the words

    Turning a voice into text throws things away. A sigh, a worried tone, a laugh: all gone. And mistakes creep in. Was that “order four eight two one” or “for eight to one”? Good voice assistants read important details back to be sure.

  11. Another way: one model for speech

    Newer models take speech in and give speech out directly, with no text in the middle. They are faster and keep the tone of voice. The price is that there is no written text halfway to check, filter or log.

  12. Now at real speed

    Sound in, words, a reply, sound out, with the first of it reaching HM Nayem's ear in under a second. A voice assistant is the chatbot you already understand, with a model on each end and a stopwatch running.

A voice assistant is a chatbot racing a one-second clock.

Ideas in this story