What is a staged rollout?
A change that passes its tests can still fail with real people. So it goes out in stages. First it runs silently beside the current version. Then a small share of users get it, chosen so that each person always sees the same version. Then everyone.
Everything that shapes the product's behaviour is kept together under one version number: the prompt, the model, the tools and the search settings. Going back means changing one setting, not publishing new code.
Each stage has a few numbers to watch and a line each should stay inside. Enough traffic is collected to tell a real difference from chance. One thing is changed at a time, so that a change in the numbers has one possible cause.
Why does a staged rollout matter?
Tests only cover what someone thought to test. Real users will do the rest. Releasing a change to everyone at once means everyone meets its problems at the same moment. Releasing in stages limits how many people a mistake can reach, and gives time to see it in the numbers before it spreads.
How does a staged rollout work?
- Bundle everything that shapes behaviour, prompt, model, tools, settings, under one version number.
- The new version passes the test set.
- It runs silently beside the current one, answering real requests that nobody sees.
- A small share of users get it, each person always seeing the same version.
- The numbers are compared. If they hold, the share is widened. If not, one setting sends everyone back.
When should you use a staged rollout?
Use it when
- Any change to a product real people depend on
- Changing the model, the prompt or the tools
- The effect of a change is hard to predict
Think twice when
- The product has no users yet
- The change cannot affect behaviour
Where does a staged rollout sit in a real system?
A rollout is controlled by a setting that decides which version each user gets. It is separate from the program itself, so it can be changed in seconds without rebuilding anything.
What goes wrong with a staged rollout?
- Releasing two changes together, so nobody can tell which one moved the numbers.
- Declaring a winner before there is enough traffic to tell a real difference from chance.
- Never testing the way back until it is needed.
What does a staged rollout look like in practice?
A new complaints prompt went to half of customers. Handovers for Spanish speakers rose from 24% to 41%, because the model had started replying in English. It was withdrawn in four seconds, fixed with one line, and went through every stage again.