What is a voice assistant?
Most voice assistants are three models in a row. A speech-to-text model turns what the caller says into words. An ordinary language model writes the reply. A text-to-speech model turns that reply into sound.
Around them sit two harder problems. One is deciding when the caller has finished speaking, which is done by listening for a short silence. The other is being interrupted: when the caller speaks over the assistant, it has to stop at once and listen.
Newer models can take speech in and give speech out directly, without text in the middle. That is faster and keeps tone of voice, but leaves no written text to check or log along the way.
Why does a voice assistant matter?
Speaking is the most natural way to ask for something, and for many situations, such as a phone call, it is the only way. Voice is also the hardest form of AI product to make feel right, because conversation has a rhythm, and a delay that would go unnoticed in a chat sounds like a broken line.
How does a voice assistant work?
- Sound from the caller arrives in small packets.
- A speech-to-text model turns it into words as it is spoken.
- A detector decides when the caller has finished their turn.
- A language model writes a reply as text.
- A text-to-speech model turns the reply into sound, starting with the first sentence while the rest is still being written.
When should you use a voice assistant?
Use it when
- People reach you by phone
- Hands and eyes are busy
- Typing is a barrier for your users
Think twice when
- The answer is long, or includes links, tables or codes. Send those as text
- You cannot keep the delay near a second
Where does a voice assistant sit in a real system?
A voice assistant is a harness with a strict clock. It sits between a phone line or microphone and three models, passing sound and text between them and timing every stage.
What goes wrong with a voice assistant?
- Waiting for each stage to finish before starting the next.
- Writing replies for the eye, with symbols and lists that sound absurd aloud.
- Not stopping at once when the caller interrupts.
What does a voice assistant look like in practice?
A first version waited for silence, then transcribed, then wrote, then spoke: 4.2 seconds of dead air. With every stage streaming and a detector for the end of a turn, the first sound came 0.9 seconds after the caller's last word.