What is model routing?
Models come in sizes. Large ones handle difficult reasoning and cost more per token. Small ones are several times cheaper and faster, and are perfectly good at simple questions.
Routing puts a quick sorting step in front. Each request is judged easy or hard and sent to the model that fits. For most products, most requests are easy.
A common safety net is escalation: if the small model's answer fails a check, or it says it is unsure, the same request is tried again on the large one. Any routing change should be run through an eval set first, because cheaper must not quietly mean worse.
Why does model routing matter?
Models come in sizes. Small ones are fast and cheap. Large ones are slower, cost many times more, and handle harder problems. Most requests to a real product are easy. Sending all of them to the largest model wastes money and makes every reply slower than it needs to be.
How does model routing work?
- A quick first step looks at each request and judges how hard it is.
- Easy requests go to a small model.
- Harder ones go to a large model, and the hardest to a large model allowed to think first.
- A model that finds itself out of its depth can pass the request up.
- Each lane is measured separately, so savings never hide a drop in quality.
When should you use model routing?
Use it when
- Requests vary widely in difficulty
- Cost or speed is a real constraint
- A job contains simple steps, such as sorting, alongside hard ones
Think twice when
- Volume is low and the bill does not matter
- Every request is equally hard
Where does model routing sit in a real system?
The router is a step in your harness or gateway, before the main model call. The table of which model serves which lane is a setting, reviewed whenever prices or models change.
What goes wrong with model routing?
- Routing by guesswork without testing each lane's quality.
- Never revisiting the table as cheaper models improve.
- Giving the small model no way to say a request is too hard.
What does model routing look like in practice?
Of a million messages, 71% are simple, 20% need care and 9% are hard. Routed to a small model, a large one, and a large one with thinking, the bill falls from $9,400 to $4,090, and seven in ten customers see a first word in 0.2 seconds, not 1.2.