Which model for which job: AI Engineering, story 30
The shop sends every message to its biggest model. Most of them are easy. Watch a router send each one to the smallest model that can handle it.
Models come in sizes. A router matches each job to the cheapest model that does it well, and pays for thinking only when it helps.
The story, step by step
One model for everything
When the shop's assistant was built, every message was sent to the best model available. That was a sensible way to start. It also means "thanks, bye" is handled by the same model, at the same price, as a three-way dispute about a damaged tent. The bill and the waiting time are both higher than they need to be.
Models come in sizes
Every provider sells a family of models. The small ones answer quickly and cost little. The large ones are slower, cost many times more, and cope with harder problems. A third option lets a large model think before it answers. They are different tools, and a team may use all three in one product.
Most questions are easy
HM Nayem reads a thousand real messages and sorts them by how hard they are. Seven in ten are simple: a greeting, a policy question, a lookup. Two in ten need some care. One in ten is truly difficult. The assistant had been paying the top price for all ten.
A router decides
So a step is added in front of the models. It looks at each message and picks a lane. The step itself is a small model asked one question: how hard is this? That call takes a fraction of a second and a fraction of a cent. Simple messages go to the small model. The rest go up.
Send it up when unsure
The router will sometimes call a hard question simple. So there is a second chance. The small model is told that it may reply with a single word, "escalate", when a question is beyond it. The router then passes the message to the large model. The customer waits a little longer and never sees the handover.
Thinking, on demand
A customer wants to know which of three tents suits a windy trip, under a budget, for someone tall. Answering well means holding several facts and weighing them. With thinking switched on, the model first writes private working that the customer never sees, then answers. The working is paid for like any other text, and the answer is noticeably better.
When thinking does not help
Thinking is not a general improver. HM Nayem tries it on both kinds of question. On tangled ones it lifts the share of good answers a great deal. On simple ones it changes nothing, except that the customer waits five seconds longer and the shop pays eight times as much. It is switched on only in the difficult lane.
Different jobs inside one reply
Routing is not only about whole messages. One reply to a complaint involves several jobs. Deciding what kind of message it is. Writing the reply. Checking the reply promises nothing false. The first and last are small, sharp jobs for the small model. Only the middle one needs the large one.
The router can be wrong
Saving money is easy if quality is allowed to slip unnoticed. So each lane has its own test questions and its own score, and the lanes are compared with sending everything to the large model. The simple lane must not be worse. When a customer replies "that's not what I asked", the next message skips the small model altogether.
Speed is part of the product
The saving that customers notice is not money. Seven in ten of them now see the first word of a reply in a fifth of a second and not over a second. A reply that begins at once feels like a conversation. People are far more patient with a hard question taking six seconds than with "hello" taking two.
The bill, before and after
A month later the numbers are in. The same million messages. The simple lane handles most of them for almost nothing. The difficult lane costs more per message than before, because of the thinking, and there are few of them. The total is under half of the old bill, and the hardest questions are answered better than they were.
A table that needs looking after
Which model sits in which lane is a small table in the shop's settings. It is not a decision made once. Prices fall, new small models appear that handle what only a large one could, and the mix of questions shifts with the seasons. The table is tested and updated like the prompts. A later story follows what happens when a new model arrives.
Three messages, three lanes, at real speed
Three customers at once. A thank-you goes to the small model and is answered before the others have left the router. A complaint goes to the large one. A tangled comparison goes to the large one with time to think. Each got the least expensive model that could do it properly, which is all that routing is.
Use the smallest model that does the job well.
- Models come in sizes: small and fast, large and careful, and large with time to think.
- Most messages are easy. A router sends each to the cheapest lane that can handle it.
- A model that is out of its depth can pass the message up.
- Thinking helps tangled questions a lot and simple ones not at all.
- Give each lane its own test, so that savings never hide a drop in quality.
- The routing table is checked again whenever prices or models change.
Ideas in this story
- Model routing: Sending easy requests to a small, cheap model and only the hard ones to a large one.
- Thinking before answering: Letting the model work a problem through before it writes the reply.
- A time budget: A fixed amount of time a reply may take, shared out between every step on the way.
- A model as judge: Using a second model, with a written rubric, to grade answers that simple code cannot check.
- Fallbacks: A second model or provider that takes over when the first one is unavailable.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.