AI Engineering, story 35The whole machine

One last call

1 of 16

The whole machine: AI Engineering, story 35

HM Nayem calls about a damaged parcel, sends a photo and gets a refund. Every idea so far, in one request, and nothing in it is new to you.

One request through everything: a production multimodal agent, with nothing new to you.

The story, step by step

  1. One last call

    HM Nayem phones the shop. A jacket has arrived torn. In the next ninety seconds a voice will listen, look at a photo, check the rules and pay back $149. Follow it through. You already know every part.

  2. Sound becomes words

    A speech model writes down what HM Nayem says, as it is said. From here on it is text, the only thing a language model reads. You saw this in the story about the voice on the phone.

  3. Who is calling?

    Before the model is involved at all, the assistant looks up what the shop already knows about this caller. The model remembers nothing. The shop's memory does. That was the story about being remembered.

  4. The prompt is put together

    Now the assistant builds the prompt: the shop's instructions, a menu of tools, the relevant memories, and the caller's words with the phone number masked. Every piece comes from an earlier story. The model will see only this.

  5. Two requests at once

    The model does not answer. It asks for two things side by side: the order itself, and the shop's rules on damaged goods. It cannot fetch either. It asks, and the assistant does. Tool calling, in parallel.

  6. The right page, found by meaning

    The search returns the chunk that matters, out of thousands. It was found by meaning, re-sorted, and comes with its source. The order comes back too. Both are added to the record.

  7. It asks for a photo

    The rules say a photo is needed. So the model asks for one, and the assistant texts HM Nayem a link. A minute later a picture arrives. The model reads it the way it read the receipt: cut into squares, each one like a token.

  8. The model asks to pay

    Everything lines up: the order is real, it arrived two days ago, the damage is visible, the rules allow a refund. So the model asks for one. Notice the word. It asks.

  9. The checkpoint

    Money is about to move, so the request stops at a checkpoint written in ordinary code. Is this the caller's own order? Is the amount what was paid? Is it under the limit for refunds without a person? No model can argue past these.

  10. The refund goes through

    The assistant runs the real refund, stamped with a unique label so that a retry could never pay twice. Had it been over the limit, a person at the shop would have been asked first.

  11. The answer, spoken

    With the result in the record, the model writes its reply. The first words go to the speech model before the last are written, so HM Nayem hears a voice within a second. Streaming, from the very first story.

  12. Something did fail, quietly

    One thing went wrong on the way. The second call to the model was refused for a moment: too many requests. The gateway waited a second and tried again. Nobody on the call noticed. That was launch day.

  13. Every step was written down

    All of this was recorded as it happened, step by step, with times and costs. If HM Nayem had been told something wrong, an engineer could open this trace and read it backwards. Most of the prompt was read from the cache, so the whole call cost under two cents.

  14. Saved for next time

    The call ends. The assistant saves one new fact, so that next time the shop already knows. And the conversation joins the pile that tonight's batch job will label, at half price.

  15. Nothing here was new

    Look at the whole scene. A voice, a memory, a search, tools, a photo, a checkpoint, a retry, a trace. A production AI system. And not one piece of it was new to you. At the centre it is still what you saw in story one: text goes to a model, and the next token comes back.

  16. Now at real speed

    Here is the whole call, as fast as it happens. Watch it once more, knowing what every parcel is. This is what is going on behind an AI reply.

You now know how the whole machine works.

Ideas in this story