The bill arrives: AI Engineering, story 28
One reply to HM Nayem costs about a cent. The shop sends a million a month. See where the money goes, and four moves that cut the bill to a fifth.
Where the money goes, and the moves that cut it: caching, smaller models for easy work, batching.
The story, step by step
One reply, about a cent
HM Nayem asks the shop's chat a question and gets an answer. In the very first story you saw that every token read and written is billed. Add this one up, and the reply cost the shop about one cent.
Times a million
A cent is nothing. But the shop answers 1.2 million messages a month. At the end of the first month, this arrives.
Where the money goes
Before cutting anything, look at what one request is made of. Most of it is not HM Nayem's question, or even the answer. It is the shop's own instructions and tool menu, the same 1,800 tokens, sent again with every message.
Move one: let the provider remember
Model providers offer prompt caching. They keep the already-processed start of a prompt for a few minutes. When the next request begins with exactly the same text, that part is read back at about a tenth of the price, and the reply starts sooner too.
The order of a prompt now matters
The cache matches from the very first token up to the first difference, and stops there. So the shop arranges every prompt the same way: what never changes at the top, what changes at the bottom.
One small mistake undoes it
A classic slip: someone adds the current time as the first line of the instructions. Now every request differs from the last in its very first tokens. Nothing matches, nothing is cached, and the bill quietly doubles.
What caching saved
With the prompts laid out properly, three quarters of every request is read from the cache at a tenth of the price. The bill halves, and nothing about the answers changed.
Move two: not everything needs the big model
Look at what customers actually ask. Most messages are simple. A small model answers those just as well, at about a third of the price and noticeably faster. Only some questions need the large one.
A sorter in front
So each message is sorted first by a very quick, very cheap step that judges how hard it is. Simple ones go to the small model. Hard ones go to the large. This is called routing. About seven in ten turn out to be simple.
When the small model is out of its depth
The sorter is not perfect. So there is a safety net. If the small model's answer fails a check, or it says it is not sure, the same message is sent again to the large model. The customer just gets a good answer, a moment later.
What routing saved
Seven in ten messages now cost a third as much. The bill nearly halves again.
Move three: work that can wait
Every night the shop labels the day's 40,000 conversations by topic, for its reports. Nobody is waiting on those. Providers offer a batch lane for exactly this: hand it all over at once, collect the results by morning, pay half.
Move four: say less
One more thing from the price list. A token the model writes costs about five times as much as one it reads. Replies that ramble are expensive. Asking for shorter answers, and capping their length, trims the bill a little further.
Did the answers get worse?
This is the question that matters, and guessing is not allowed. Every one of these moves was run through the shop's eval set before it went live. The score barely moved. Cheaper is only a win if it is still good.
The bill, a month later
Same customers, same number of messages, answers just as good. The bill is about a fifth of what it was. None of it needed a cleverer model. It needed someone to look at where the tokens were going.
Most of the bill is waste you can measure.
- Most of each request is the same instructions, sent again every time.
- Prompt caching reads that unchanging start at about a tenth of the price.
- Routing sends simple messages to a small model and hard ones to a large one.
- Work nobody is waiting for can go in a batch at half price.
- Every saving is checked against the eval set before it ships.
Ideas in this story
- Prompt caching: The provider remembers the unchanging start of a prompt, so repeats of it cost about a tenth and arrive faster.
- Model routing: Sending easy requests to a small, cheap model and only the hard ones to a large one.
- Batch jobs: Sending work that nobody is waiting for all at once, for results within hours at about half the price.
- Evals: A fixed set of test questions with known good answers, run after every change to see if things got better or worse.