AI Engineering, story 33A new model arrives

An email from the provider

1 of 13

A new model arrives: AI Engineering, story 33

The provider releases a better model and will switch the old one off. HM Nayem swaps it in, and three things quietly break.

A new model is tested like any other change: pinned versions, the test set, a silent trial, then a gradual switch.

The story, step by step

  1. An email from the provider

    A notice arrives. The provider has released version 5 of its model. It is faster, cheaper and scores higher on every public test. Version 4, which the shop's assistant has used for a year, will be switched off in six months. HM Nayem has to move. The question is how to do it without customers noticing.

  2. Better on average is not better for you

    A model's public scores are averages over thousands of general tasks. The shop's assistant is one particular task, built from prompts that were adjusted, line by line, to how version 4 happens to behave. A model that is better in general can be worse at exactly the things those prompts were leaning on.

  3. Pin the version

    The first rule protects against surprises. The assistant's settings name the model by its exact version. Some providers offer a name that always points at the newest model. Using that name means the model changes underneath the product on a day somebody else chooses. With a pinned version, nothing changes until HM Nayem changes it.

  4. Run the whole test set on both

    Before anyone forms an opinion, HM Nayem runs all three hundred test cases through both models, with the same prompts. Version 5 scores lower. Not by much, and the public tests said it would be higher. This is the normal result of a model swap, and the reason a team's own test set is the most valuable thing it owns.

  5. Read the ones that broke

    The total says something changed. The twenty-four cases say what. HM Nayem reads every one and sorts them. They fall into three piles, and none of them is the new model being less clever. It behaves differently, and the prompts were written for the old behaviour.

  6. It does exactly what it is told

    The instructions say "be thorough". Version 4 treated that loosely, and gave three sentences. Version 5 follows instructions more literally, and gives three paragraphs. The old prompt was shouting to be heard by a model that half listened. The new model hears every word. The fix is to say what was always meant.

  7. The shape of the reply drifts

    Seven failures are about shape. The assistant asks for a small form with fixed fields, and code reads it. Version 5 sometimes puts a friendly sentence before the form, and names one field differently. Nothing a person would mind. Code minds a great deal. The check that validates every form caught all seven, which is what it is for.

  8. It reaches for tools differently

    The last four are about tools. Version 5 is more eager. Where version 4 answered a simple question from the help page it had been given, version 5 looks the order up as well, to be sure. And it now asks for two tools in a single reply, which the assistant's code had never been sent before. Both are improvements the code was not ready for.

  9. Cost and speed move too

    The price per token went down. The bill might not. Version 5 counts tokens a little differently, writes longer replies until told otherwise, and makes more tool calls. HM Nayem measures the cost of the whole test set on each model, after fixing the prompts. Only that number says what the switch will do to the bill.

  10. Tests pass. Real traffic next

    With the prompts rewritten, version 5 passes more cases than version 4 ever did. A test set is still only what HM Nayem thought to test. So for a week, every real customer message is answered by both models. Customers see only version 4's reply. Version 5's is saved beside it, and the pairs that differ most are read.

  11. Switch a little at a time

    Now customers start to see version 5. One in twenty at first, each chosen by their account so that the same person always gets the same model. The assistant's live numbers are watched for each group. After two quiet days, half. Then everyone. Going back at any point means changing one line.

  12. Keep the old one until it is proven

    Version 4 stays in the settings, unused, for another month. The eleven cases where version 5 did worse, and the twenty-four from the first run, all go into the test set, so the next model is checked against them from the start. A new model will arrive again within the year. The second time takes HM Nayem two days.

  13. One customer, across the switch

    The same question, asked the day before and the day after. A different model, different prompts, the same answer, a little sooner. The customer saw nothing change. That is the whole aim. A model swap that customers notice was not finished.

A new model is a change like any other. Test it like one.

Ideas in this story