StoryboardAI Engineering

Changing models

Moving a product from one model version to another without changing how it behaves for its users.

What is changing models?

Model providers release new versions and retire old ones. A new model may be better on average and still behave differently in the ways one product depends on: how literally it follows instructions, how eagerly it uses tools, how it formats its replies.

So the version is always named exactly, and never left to mean whichever is newest. A new version is treated as a change like any other. The whole test set is run on both, the differences are read, and the prompts are adjusted, because prompts are tuned to the model they were written for.

Before any user sees it, the new model answers real traffic silently beside the old one. Then it takes a small share of users, then more, with a single setting that sends everyone back.

A model swap, done as a tested change.Step 1: Pin the exact version. Step 2: Run the test set on both. Step 3: Fix prompts for the new one. Step 4: Run silently, then switch gradually.1Pin the exactversion2Run the test seton both3Fix prompts forthe new one4Run silently, thenswitch gradually
A model swap, done as a tested change.

Why does changing models matter?

Providers release new models several times a year and retire old ones. A product must move sooner or later. A new model is usually better on average and still different in ways that break things built for the old one. Teams that treat a model swap as a routine, tested change move quickly. Teams that do not are surprised in production.

How does changing models work?

  1. Name the model by its exact version, so it changes only when you decide.
  2. Run your whole test set on the old and the new model with the same prompts.
  3. Read the cases that got worse, and sort them by cause.
  4. Rewrite the prompts for the new model, and check formats, tool use and real cost.
  5. Run the new model silently on real traffic, then switch a few users at a time.

When should you use changing models?

Use it when

  • A provider announces a new version or a retirement date
  • A cheaper or faster model might do the same job
  • Adding a second provider as a fallback

Think twice when

  • You have no test set. Build that first, or you will not know what changed

Where does changing models sit in a real system?

The model's name is one line in your product's versioned settings. The work of a migration happens in your evals and your prompts, well before that line is changed for real users.

What goes wrong with changing models?

  • Using a name that always points to the newest model.
  • Trusting public test scores to predict your own product.
  • Changing the model and the prompts at once for everyone, with no way back.

What does changing models look like in practice?

A new version scored 261 of 300 where the old one scored 276. The 24 new failures were replies that were too long, a changed format and more eager tool use. After the prompts were rewritten it scored 291, cost 20% less, and customers were moved over in nine days.

Watch it happen

These stories show changing models at work, one step at a time.