AI Engineering, story 34Rolling it out

A better prompt, ready to go

1 of 13

Rolling it out: AI Engineering, story 34

A better prompt is ready. Follow it from HM Nayem's laptop to every customer, a few at a time, with a switch that takes it back in seconds.

Changes go out in stages: tests, a silent run, a few users, everyone. Each stage has numbers to watch and a way back.

The story, step by step

  1. A better prompt, ready to go

    HM Nayem has rewritten the part of the prompt that handles complaints. On the test set, it scores higher. The tempting next step is to switch every customer to it this afternoon. Tests only cover what he thought of. Nine hundred real customers a day will think of the rest. So the change goes out in stages.

  2. Everything that shapes behaviour, under one number

    First, what exactly is being released? An assistant's behaviour comes from its prompt, its model, its tools and its search settings together. Change any one and it is a different assistant. So all four are stored as one bundle with a version number. Releasing means pointing customers at a bundle. Going back means pointing at the last one.

  3. Stage 1: the tests

    The test set is the first gate, and version 13 is already through it. It runs automatically on every proposed version. A version that scores worse than the current one, by more than the usual wobble between runs, goes no further without someone reading the failures. This stage costs two dollars and ten minutes.

  4. Stage 2: silent

    For two days, version 13 answers every real message, and its answers are shown to nobody. Customers get version 12's reply as always. The two replies are saved side by side. Where they differ most, a person reads both. This finds problems with real questions before a single customer is exposed to them.

  5. Stage 3: one customer in twenty

    Now real people see it. The splitter sends one customer in twenty to version 13. The choice is made from the customer's account number, not at random each time, so nobody flips between versions in the middle of a conversation. The other nineteen are the comparison group.

  6. What to watch

    Each version writes the same few numbers for every chat, into its own column. How many chats were settled without a person. How many were handed over. How many customers pressed thumbs-down. What it cost, and how long the first word took. The number of times a guard stopped a reply. Each has a line it should stay inside.

  7. Is the difference real?

    Version 13 settled 67 chats in a hundred against 64. Is that the change, or luck? With forty-five customers a day on the new version, three points is well within what chance alone produces. HM Nayem works out beforehand how many chats are needed to tell a three-point difference from noise, and waits for them. Deciding early is how teams ship changes that did nothing.

  8. The off switch

    Before widening, HM Nayem tests the way back. The splitter's rule is a setting, not part of the program. Changing it takes effect within seconds and needs nothing to be rebuilt or restarted. He sets it to zero on a quiet morning, watches every customer return to version 12, and sets it back. A switch that has never been pressed cannot be relied on.

  9. Day two: something is wrong

    With half of customers on version 13, one number moves the wrong way. Handovers to a person are up, and only for customers writing in Spanish. Version 13's new complaints section was written in English, with English examples, and the model now replies to Spanish complaints in English. No test covered it. HM Nayem presses the switch.

  10. Fix it, and begin again

    Going back took four seconds and cost a day. Version 14 adds one line telling the model to reply in the customer's language, and twelve Spanish complaints join the test set. Then it starts again at stage one. It does not skip ahead because it is "only a small fix". Small fixes are how the last problem got in.

  11. Change one thing at a time

    Version 14 was nearly released together with a new search setting that was also ready. HM Nayem keeps them apart. If two things change at once and a number moves, there is no telling which one moved it, and both have to be withdrawn. One change per version makes every result readable. It is slower by days and faster by weeks.

  12. A diary of behaviour

    Every version leaves one line in a log: what changed, who changed it, what the numbers did, and whether it stayed. Six months on, when someone asks why the assistant stopped offering discount codes, the answer is a search away. Ordinary software keeps this history for code. An AI product needs it for behaviour.

  13. Version 14, all the way out

    Version 14's journey, at speed. Tests. Two silent days. One customer in twenty, then half, then everyone, with the numbers level or better at each step and a tested switch behind it. Eleven days from laptop to all customers. The stages are slow on purpose. They are what lets HM Nayem change the assistant every week without fearing it.

Release in stages, and keep a hand on the switch.

Ideas in this story