The whole machine: AI Engineering, story 35
HM Nayem calls about a damaged parcel, sends a photo and gets a refund. Every idea so far, in one request, and nothing in it is new to you.
One request through everything: a production multimodal agent, with nothing new to you.
The story, step by step
One last call
HM Nayem phones the shop. A jacket has arrived torn. In the next ninety seconds a voice will listen, look at a photo, check the rules and pay back $149. Follow it through. You already know every part.
Sound becomes words
A speech model writes down what HM Nayem says, as it is said. From here on it is text, the only thing a language model reads. You saw this in the story about the voice on the phone.
Who is calling?
Before the model is involved at all, the assistant looks up what the shop already knows about this caller. The model remembers nothing. The shop's memory does. That was the story about being remembered.
The prompt is put together
Now the assistant builds the prompt: the shop's instructions, a menu of tools, the relevant memories, and the caller's words with the phone number masked. Every piece comes from an earlier story. The model will see only this.
Two requests at once
The model does not answer. It asks for two things side by side: the order itself, and the shop's rules on damaged goods. It cannot fetch either. It asks, and the assistant does. Tool calling, in parallel.
The right page, found by meaning
The search returns the chunk that matters, out of thousands. It was found by meaning, re-sorted, and comes with its source. The order comes back too. Both are added to the record.
It asks for a photo
The rules say a photo is needed. So the model asks for one, and the assistant texts HM Nayem a link. A minute later a picture arrives. The model reads it the way it read the receipt: cut into squares, each one like a token.
The model asks to pay
Everything lines up: the order is real, it arrived two days ago, the damage is visible, the rules allow a refund. So the model asks for one. Notice the word. It asks.
The checkpoint
Money is about to move, so the request stops at a checkpoint written in ordinary code. Is this the caller's own order? Is the amount what was paid? Is it under the limit for refunds without a person? No model can argue past these.
The refund goes through
The assistant runs the real refund, stamped with a unique label so that a retry could never pay twice. Had it been over the limit, a person at the shop would have been asked first.
The answer, spoken
With the result in the record, the model writes its reply. The first words go to the speech model before the last are written, so HM Nayem hears a voice within a second. Streaming, from the very first story.
Something did fail, quietly
One thing went wrong on the way. The second call to the model was refused for a moment: too many requests. The gateway waited a second and tried again. Nobody on the call noticed. That was launch day.
Every step was written down
All of this was recorded as it happened, step by step, with times and costs. If HM Nayem had been told something wrong, an engineer could open this trace and read it backwards. Most of the prompt was read from the cache, so the whole call cost under two cents.
Saved for next time
The call ends. The assistant saves one new fact, so that next time the shop already knows. And the conversation joins the pile that tonight's batch job will label, at half price.
Nothing here was new
Look at the whole scene. A voice, a memory, a search, tools, a photo, a checkpoint, a retry, a trace. A production AI system. And not one piece of it was new to you. At the centre it is still what you saw in story one: text goes to a model, and the next token comes back.
Now at real speed
Here is the whole call, as fast as it happens. Watch it once more, knowing what every parcel is. This is what is going on behind an AI reply.
You now know how the whole machine works.
- At the centre is one thing: text goes to a model and the next token comes back.
- Everything else is ordinary code deciding what the model sees and what happens with its reply.
- That code gives it knowledge, tools, memory, eyes and a voice.
- And that code, not the model, keeps it safe, affordable and reliable.
Ideas in this story
- Voice assistants: Speech turned into text, answered by a model, and turned back into speech, fast enough to feel like talking.
- Long-term memory: Facts the app saves between conversations and puts back into the prompt when they are relevant.
- Retrieval-augmented generation: Retrieve the right text, add it to the prompt, then let the model answer from it. RAG for short.
- Tool calling: The model cannot do anything itself. It asks your code to, and your code answers.
- Images as input: A model that accepts pictures cuts each one into small squares and reads them much like tokens.
- Asking first: Pausing before an action that sends something out or cannot be undone, and waiting for a person to say yes.
- Retries and backoff: Trying a failed request again after a wait that grows each time, with some randomness.
- Tracing: Recording every step of a request, with what went in, what came out and how long it took.