A new model arrives: AI Engineering, story 33
The provider releases a better model and will switch the old one off. HM Nayem swaps it in, and three things quietly break.
A new model is tested like any other change: pinned versions, the test set, a silent trial, then a gradual switch.
The story, step by step
An email from the provider
A notice arrives. The provider has released version 5 of its model. It is faster, cheaper and scores higher on every public test. Version 4, which the shop's assistant has used for a year, will be switched off in six months. HM Nayem has to move. The question is how to do it without customers noticing.
Better on average is not better for you
A model's public scores are averages over thousands of general tasks. The shop's assistant is one particular task, built from prompts that were adjusted, line by line, to how version 4 happens to behave. A model that is better in general can be worse at exactly the things those prompts were leaning on.
Pin the version
The first rule protects against surprises. The assistant's settings name the model by its exact version. Some providers offer a name that always points at the newest model. Using that name means the model changes underneath the product on a day somebody else chooses. With a pinned version, nothing changes until HM Nayem changes it.
Run the whole test set on both
Before anyone forms an opinion, HM Nayem runs all three hundred test cases through both models, with the same prompts. Version 5 scores lower. Not by much, and the public tests said it would be higher. This is the normal result of a model swap, and the reason a team's own test set is the most valuable thing it owns.
Read the ones that broke
The total says something changed. The twenty-four cases say what. HM Nayem reads every one and sorts them. They fall into three piles, and none of them is the new model being less clever. It behaves differently, and the prompts were written for the old behaviour.
It does exactly what it is told
The instructions say "be thorough". Version 4 treated that loosely, and gave three sentences. Version 5 follows instructions more literally, and gives three paragraphs. The old prompt was shouting to be heard by a model that half listened. The new model hears every word. The fix is to say what was always meant.
The shape of the reply drifts
Seven failures are about shape. The assistant asks for a small form with fixed fields, and code reads it. Version 5 sometimes puts a friendly sentence before the form, and names one field differently. Nothing a person would mind. Code minds a great deal. The check that validates every form caught all seven, which is what it is for.
It reaches for tools differently
The last four are about tools. Version 5 is more eager. Where version 4 answered a simple question from the help page it had been given, version 5 looks the order up as well, to be sure. And it now asks for two tools in a single reply, which the assistant's code had never been sent before. Both are improvements the code was not ready for.
Cost and speed move too
The price per token went down. The bill might not. Version 5 counts tokens a little differently, writes longer replies until told otherwise, and makes more tool calls. HM Nayem measures the cost of the whole test set on each model, after fixing the prompts. Only that number says what the switch will do to the bill.
Tests pass. Real traffic next
With the prompts rewritten, version 5 passes more cases than version 4 ever did. A test set is still only what HM Nayem thought to test. So for a week, every real customer message is answered by both models. Customers see only version 4's reply. Version 5's is saved beside it, and the pairs that differ most are read.
Switch a little at a time
Now customers start to see version 5. One in twenty at first, each chosen by their account so that the same person always gets the same model. The assistant's live numbers are watched for each group. After two quiet days, half. Then everyone. Going back at any point means changing one line.
Keep the old one until it is proven
Version 4 stays in the settings, unused, for another month. The eleven cases where version 5 did worse, and the twenty-four from the first run, all go into the test set, so the next model is checked against them from the start. A new model will arrive again within the year. The second time takes HM Nayem two days.
One customer, across the switch
The same question, asked the day before and the day after. A different model, different prompts, the same answer, a little sooner. The customer saw nothing change. That is the whole aim. A model swap that customers notice was not finished.
A new model is a change like any other. Test it like one.
- Name the model by its exact version, so that it changes only when you decide.
- Public scores are averages. Only your own test set says what happens to your product.
- Read the cases that broke. Usually the model behaves differently, not worse.
- Prompts are tuned to a model. Expect to rewrite them.
- Check the format of replies, the use of tools, and the real cost.
- Run the new model silently on real traffic, then switch a few users at a time.
Ideas in this story
- Changing models: Moving a product from one model version to another without changing how it behaves for its users.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.
- The system prompt: Instructions the product's builders write, placed ahead of every user message.
- Structured output: Getting a reply in a fixed shape that code can read, checking it, and retrying when it is wrong.
- Tool calling: The model cannot do anything itself. It asks your code to, and your code answers.
- Staged rollout: Releasing a change to an AI product a step at a time, with numbers to watch at each step and a quick way back.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.