Rolling it out: AI Engineering, story 34
A better prompt is ready. Follow it from HM Nayem's laptop to every customer, a few at a time, with a switch that takes it back in seconds.
Changes go out in stages: tests, a silent run, a few users, everyone. Each stage has numbers to watch and a way back.
The story, step by step
A better prompt, ready to go
HM Nayem has rewritten the part of the prompt that handles complaints. On the test set, it scores higher. The tempting next step is to switch every customer to it this afternoon. Tests only cover what he thought of. Nine hundred real customers a day will think of the rest. So the change goes out in stages.
Everything that shapes behaviour, under one number
First, what exactly is being released? An assistant's behaviour comes from its prompt, its model, its tools and its search settings together. Change any one and it is a different assistant. So all four are stored as one bundle with a version number. Releasing means pointing customers at a bundle. Going back means pointing at the last one.
Stage 1: the tests
The test set is the first gate, and version 13 is already through it. It runs automatically on every proposed version. A version that scores worse than the current one, by more than the usual wobble between runs, goes no further without someone reading the failures. This stage costs two dollars and ten minutes.
Stage 2: silent
For two days, version 13 answers every real message, and its answers are shown to nobody. Customers get version 12's reply as always. The two replies are saved side by side. Where they differ most, a person reads both. This finds problems with real questions before a single customer is exposed to them.
Stage 3: one customer in twenty
Now real people see it. The splitter sends one customer in twenty to version 13. The choice is made from the customer's account number, not at random each time, so nobody flips between versions in the middle of a conversation. The other nineteen are the comparison group.
What to watch
Each version writes the same few numbers for every chat, into its own column. How many chats were settled without a person. How many were handed over. How many customers pressed thumbs-down. What it cost, and how long the first word took. The number of times a guard stopped a reply. Each has a line it should stay inside.
Is the difference real?
Version 13 settled 67 chats in a hundred against 64. Is that the change, or luck? With forty-five customers a day on the new version, three points is well within what chance alone produces. HM Nayem works out beforehand how many chats are needed to tell a three-point difference from noise, and waits for them. Deciding early is how teams ship changes that did nothing.
The off switch
Before widening, HM Nayem tests the way back. The splitter's rule is a setting, not part of the program. Changing it takes effect within seconds and needs nothing to be rebuilt or restarted. He sets it to zero on a quiet morning, watches every customer return to version 12, and sets it back. A switch that has never been pressed cannot be relied on.
Day two: something is wrong
With half of customers on version 13, one number moves the wrong way. Handovers to a person are up, and only for customers writing in Spanish. Version 13's new complaints section was written in English, with English examples, and the model now replies to Spanish complaints in English. No test covered it. HM Nayem presses the switch.
Fix it, and begin again
Going back took four seconds and cost a day. Version 14 adds one line telling the model to reply in the customer's language, and twelve Spanish complaints join the test set. Then it starts again at stage one. It does not skip ahead because it is "only a small fix". Small fixes are how the last problem got in.
Change one thing at a time
Version 14 was nearly released together with a new search setting that was also ready. HM Nayem keeps them apart. If two things change at once and a number moves, there is no telling which one moved it, and both have to be withdrawn. One change per version makes every result readable. It is slower by days and faster by weeks.
A diary of behaviour
Every version leaves one line in a log: what changed, who changed it, what the numbers did, and whether it stayed. Six months on, when someone asks why the assistant stopped offering discount codes, the answer is a search away. Ordinary software keeps this history for code. An AI product needs it for behaviour.
Version 14, all the way out
Version 14's journey, at speed. Tests. Two silent days. One customer in twenty, then half, then everyone, with the numbers level or better at each step and a tested switch behind it. Eleven days from laptop to all customers. The stages are slow on purpose. They are what lets HM Nayem change the assistant every week without fearing it.
Release in stages, and keep a hand on the switch.
- Prompt, model, tools and search settings are released together under one version number.
- A change passes the tests, runs silently beside the current version, then meets a few customers.
- Each customer always gets the same version, and each version's numbers are kept apart.
- Decide how much traffic is needed before trusting a difference.
- Going back is a setting that takes seconds, and it is tested before it is needed.
- Change one thing per version, and keep a log of what each change did.
Ideas in this story
- Staged rollout: Releasing a change to an AI product a step at a time, with numbers to watch at each step and a quick way back.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.
- Tracing: Recording every step of a request, with what went in, what came out and how long it took.
- Guardrails: Checks placed before and after a model call, in code or small models, that stop certain messages going in and certain replies coming out.
- Fallbacks: A second model or provider that takes over when the first one is unavailable.