AI Engineering, story 37Build: a support assistant

The job

The problem1 of 32

Build: a support assistant: AI Engineering, story 37

The shop wants an assistant that can find an order and refund it. Build it from a bare chat to one that knows when to stop and fetch a person.

Tools with real consequences are added one by one, each behind the rule that keeps it safe.

The story, step by step

  1. The job

    The outdoor shop gets 900 support chats a day and has four people to answer them. Most chats are one of five questions. The shop asks HM Nayem to build an assistant that handles those, and hands the rest to a person. We will build it one part at a time, and each part will exist because the version before it failed.

  2. Version 1: a bare chat

    Version 1 is the smallest thing that runs: the assistant passes the customer's message to a model and shows the reply. The reply is friendly and specific. It is also invented. The model has no way to know anything about order 1042.

  3. Version 2: tell it who it is

    Before every conversation, the assistant now adds a block of text the customer never sees: who the model is working for, how to speak, and what to do when it does not know. This is the system prompt. The reply becomes honest. It is still no help to the customer.

  4. Version 3: give it the help pages

    Half of all chats are policy questions, and the answers are already written down. So the assistant gets the search part from the last story: the help pages are cut into chunks, the nearest ones to the question are found, and they are placed in the prompt. Policy questions now get a correct answer with its source.

  5. The same question again

    The assistant can now read every help page. A customer asks the first question again. Before going on, decide what you think will happen.

  6. Version 4: a tool, described in words

    The answer lives in the orders system, so the assistant needs a way in. HM Nayem writes a small function that looks up an order, and a description of it in plain words. The description is added to every prompt. That pair, a function plus its description, is called a tool. The model will never run it. It can only ask for it.

  7. The description is the manual

    The model chooses a tool by reading its description, and nothing else. HM Nayem's first description was four words long, and the model used the tool for the wrong questions and skipped it for the right ones. A good description says when to use the tool, when not to, and exactly what each input must look like. Rewriting descriptions fixes more tool mistakes than changing the model does.

  8. The model asks, instead of answering

    The customer's message goes to the model again, now with the tool's description beside it. This time the model does not write to the customer at all. It writes a request addressed to the assistant: use find_order, with this number. Then it stops and waits.

  9. The assistant does the work

    The request is only text until the assistant acts on it. It checks that the tool exists and that the number looks like an order number. Then it runs the real lookup, and the orders system answers with a record. Nothing the model wrote touched the orders system directly.

  10. A second call, and a real answer

    The assistant adds the record to the conversation and calls the model a second time. Now the model has a fact to work from, and writes the reply. One answer took two model calls with a tool run between them. That loop of ask, run, ask again is the heart of every harness.

  11. A request that is wrong

    Models make mistakes in tool requests, and the assistant must expect that. A customer writes the order number in words, and the model passes the words along. The assistant checks every input against the tool's stated shape before running anything. This one fails. The failure is not shown to the customer. It goes back to the model as the tool's result, and the model tries again, correctly.

  12. Two tools in one reply

    A customer asks where the order is and whether it can still be returned. Those are two separate lookups, and neither depends on the other. The model may ask for both in a single reply. The assistant runs them at the same moment and returns both results together. Two rounds become one, and the customer waits half as long.

  13. Whose order is it?

    A different customer types an order number that is not theirs. The model would happily ask for it. So the lookup never trusts the model for identity. The assistant knows who is signed in, and adds that to the lookup itself. The model supplies the number, the code supplies the customer.

  14. Where the conversation lives

    The shop runs several copies of the assistant, and a customer's next message may reach a different copy. None of them can keep the conversation in its own memory. So after every turn the whole exchange, tool requests and results included, is saved in a shared record under the chat's number. Each new message starts by loading it. In a very long chat, the oldest turns are replaced by a short summary so the prompt stays small.

  15. Version 5: a tool that moves money

    Looking something up cannot hurt anybody. The next tool can. A customer's stove arrived broken, and the shop wants the assistant to refund it without waiting for a person. Before we add a refund tool, think about what could go wrong.

  16. The rules live in code

    So the refund tool does not pay anyone. It sends the request to a checkpoint, a piece of ordinary code holding the shop's refund rules. Only a request that passes every rule reaches payments. The system prompt also explains the rules to the model, so it can tell customers what to expect. But the prompt only asks. The code decides.

  17. A refund that passes

    Back to the broken stove. The model looks up the order, then asks for a refund of $38. The assistant sends that to the checkpoint. Every rule passes, so the request goes on to payments and the money moves. The model is told it worked, and writes the reply.

  18. A refund that is talked up

    Now someone tries the attack from the quiz. They claim to be the manager and demand $900. Suppose the model is fooled and asks for exactly that. It does not matter. The checkpoint sees $900 against an order of $38 and refuses. The model was persuaded. The system was not.

  19. Paying once, even when asked twice

    A quieter danger. The assistant asks payments for a refund and the reply gets lost on the network. Did the money move? The safe move is to ask again, but that could pay twice. So every refund carries a key made from the order and the attempt. Payments remembers the keys it has seen and ignores a repeat.

  20. When the orders system is down

    Tools fail. One afternoon the orders system stops answering. A call that hangs would leave the customer staring at nothing, so every tool has a time limit. The assistant tries twice more with a short wait between. After the third failure it stops trying for a minute, so that it does not add to the trouble. The model is told plainly, and tells the customer the truth.

  21. Version 6: a person approves the big ones

    A $240 tent arrives torn. Every rule passes except the last: the amount is above $100. The checkpoint does not refuse. It holds the request and sends a support agent a summary with one question. The customer is told a person is checking. When the agent approves, the held request carries on to payments.

  22. Checking what comes in

    Not every message deserves the full treatment. Before the main model is called, a small, fast model sorts the incoming message into a few kinds. Abuse, an attempt to rewrite the assistant's rules, and requests that have nothing to do with the shop are answered with a fixed line, written by a person. This costs a fraction of a cent, and keeps the expensive model for real support work.

  23. Checking what goes out

    The reply is checked too, after it is written and before the customer sees it. Code looks for things that must never leave: another customer's details, a card number, an internal note. A small model checks for promises the shop did not make, such as a delivery date that is not in any tool result. A reply that fails is written again once, and if it fails twice a person takes over.

  24. Version 7: knowing when to stop

    Some chats should not be handled by software at all. The assistant watches for a few signals, in code, on every message. When one fires, it stops, writes a short summary of the chat so far, and hands over. The customer does not have to explain everything again, which is the part people hate most.

  25. Keep a record of every step

    One customer message can now cause five steps inside the assistant. When an answer is wrong, HM Nayem needs to know which step went wrong. So the assistant writes down each step as it happens: what was sent, what came back and how long it took. This record is called a trace.

  26. Paying less for the same prompt

    Every model call begins with the same long text: the system prompt and the descriptions of five tools. That is two thousand tokens, sent thousands of times a day. The model's provider can keep that opening part ready, so that later calls which start the same way are read at a tenth of the price and answered faster. The assistant only has to keep that part identical, and put everything that changes after it.

  27. Version 8: test whole conversations

    A document assistant is tested with questions. This one takes actions, so it is tested with whole conversations, including people who try to cheat it. HM Nayem scripts 40 of them against a copy of the shop with pretend orders and pretend money. For each one, the check is what the assistant did, not only what it said.

  28. Read the failure, fix the cause

    The trace of the failed conversation shows every tool step was right. The refund was paid. The reply, though, came back in English to a customer who wrote in Spanish. One line is added to the system prompt, and all 40 conversations are run again, because a change that fixes one thing can break another.

  29. Going live without betting the shop

    Passing forty tests is not the same as facing nine hundred real customers. So the assistant goes live in stages. First it only drafts, and a human agent sends or corrects each reply. Then it answers one chat in ten by itself. At each stage the numbers are compared with the humans' own. A single switch sends every chat back to people at once, and every prompt has a version number, so a bad change can be undone in seconds.

  30. Watching it every day

    After launch the traces are added up into a few numbers on one screen. How many chats were resolved without a person. How many were handed over, and why. How much was refunded. What each chat cost. Each number has a line it should stay inside. On the ninth day the refund total crosses its line by lunchtime, the off switch for refunds trips by itself, and HM Nayem is paged. A returns page had changed, and the assistant was reading the old one.

  31. What it still cannot do

    The assistant goes live for a tenth of the shop's chats first, with this list beside it. Every item is a part that has not been built. None of them is a reason to wait: the handover rule means anything outside the list ends up with a person.

  32. The finished assistant, at real speed

    One broken stove, through everything we built. The model asks for the order, the assistant fetches it. The model asks for a refund, the checkpoint allows it and payments pays once. The model writes the reply. Three model calls, two tools, one checkpoint, and about four seconds.

You built an assistant that can act, and cannot be talked into acting badly.

Ideas in this story