AI Engineering, story 30Which model for which job

One model for everything

1 of 13

Which model for which job: AI Engineering, story 30

The shop sends every message to its biggest model. Most of them are easy. Watch a router send each one to the smallest model that can handle it.

Models come in sizes. A router matches each job to the cheapest model that does it well, and pays for thinking only when it helps.

The story, step by step

  1. One model for everything

    When the shop's assistant was built, every message was sent to the best model available. That was a sensible way to start. It also means "thanks, bye" is handled by the same model, at the same price, as a three-way dispute about a damaged tent. The bill and the waiting time are both higher than they need to be.

  2. Models come in sizes

    Every provider sells a family of models. The small ones answer quickly and cost little. The large ones are slower, cost many times more, and cope with harder problems. A third option lets a large model think before it answers. They are different tools, and a team may use all three in one product.

  3. Most questions are easy

    HM Nayem reads a thousand real messages and sorts them by how hard they are. Seven in ten are simple: a greeting, a policy question, a lookup. Two in ten need some care. One in ten is truly difficult. The assistant had been paying the top price for all ten.

  4. A router decides

    So a step is added in front of the models. It looks at each message and picks a lane. The step itself is a small model asked one question: how hard is this? That call takes a fraction of a second and a fraction of a cent. Simple messages go to the small model. The rest go up.

  5. Send it up when unsure

    The router will sometimes call a hard question simple. So there is a second chance. The small model is told that it may reply with a single word, "escalate", when a question is beyond it. The router then passes the message to the large model. The customer waits a little longer and never sees the handover.

  6. Thinking, on demand

    A customer wants to know which of three tents suits a windy trip, under a budget, for someone tall. Answering well means holding several facts and weighing them. With thinking switched on, the model first writes private working that the customer never sees, then answers. The working is paid for like any other text, and the answer is noticeably better.

  7. When thinking does not help

    Thinking is not a general improver. HM Nayem tries it on both kinds of question. On tangled ones it lifts the share of good answers a great deal. On simple ones it changes nothing, except that the customer waits five seconds longer and the shop pays eight times as much. It is switched on only in the difficult lane.

  8. Different jobs inside one reply

    Routing is not only about whole messages. One reply to a complaint involves several jobs. Deciding what kind of message it is. Writing the reply. Checking the reply promises nothing false. The first and last are small, sharp jobs for the small model. Only the middle one needs the large one.

  9. The router can be wrong

    Saving money is easy if quality is allowed to slip unnoticed. So each lane has its own test questions and its own score, and the lanes are compared with sending everything to the large model. The simple lane must not be worse. When a customer replies "that's not what I asked", the next message skips the small model altogether.

  10. Speed is part of the product

    The saving that customers notice is not money. Seven in ten of them now see the first word of a reply in a fifth of a second and not over a second. A reply that begins at once feels like a conversation. People are far more patient with a hard question taking six seconds than with "hello" taking two.

  11. The bill, before and after

    A month later the numbers are in. The same million messages. The simple lane handles most of them for almost nothing. The difficult lane costs more per message than before, because of the thinking, and there are few of them. The total is under half of the old bill, and the hardest questions are answered better than they were.

  12. A table that needs looking after

    Which model sits in which lane is a small table in the shop's settings. It is not a decision made once. Prices fall, new small models appear that handle what only a large one could, and the mix of questions shifts with the seasons. The table is tested and updated like the prompts. A later story follows what happens when a new model arrives.

  13. Three messages, three lanes, at real speed

    Three customers at once. A thank-you goes to the small model and is answered before the others have left the router. A complaint goes to the large one. A tangled comparison goes to the large one with time to think. Each got the least expensive model that could do it properly, which is all that routing is.

Use the smallest model that does the job well.

Ideas in this story