Never work it out twice: AI Engineering, story 29
Four hundred people a day ask the shop when it opens, and each answer is paid for. See four places an answer can be kept, and how each one goes wrong.
Answers, prompt openings and tool results can each be kept and reused. Every cache needs the right key and a way to go stale safely.
The story, step by step
The same question, 400 times a day
Four hundred people a day ask the shop's assistant when it opens on Saturday. Each time, the question goes to the model with the full prompt, and the model works out the same answer, at the same price, in the same second and a half. The four hundredth answer cost as much as the first. That is the waste this story removes.
Layer 1: keep the whole answer
The simplest fix is to keep the answer. The first time a question is asked, the reply is filed under the question's text. The next time the same text arrives, the reply is handed back from the file. No model is called. The answer arrives in a few thousandths of a second and costs nothing.
A key is more than the question
The label an answer is filed under is called its key, and choosing it is where caches go wrong. The same question can deserve different answers: in another language, for a member with a discount, under a newer version of the instructions. Everything that could change the answer must be part of the key. Leave one out, and someone gets an answer meant for somebody else.
Answers go stale
In December the shop opens at ten. The kept answer still says nine. Every cached answer needs two ways to die. A time limit, after which it is thrown away and worked out afresh. And a link to the page it came from, so that when someone edits the opening hours, every answer built on that page is cleared at once.
Never keep what is personal
"Where is my order?" is asked just as often as the opening hours. Its answer must never be kept and shared, because it belongs to one person. The rule is set in code, by kind of question. Anything that used a tool about a particular customer is not filed at all.
Layer 2: match by meaning
Exact matching catches less than you would hope. People ask the same thing forty different ways. So each kept question is also stored by its meaning, using an embedding, as the search stories showed. A new question is compared with those. If one is close enough, its answer is used, though not one word matches.
Too close is dangerous
Two questions can be nearly the same in meaning and need opposite answers. "Can I return boots?" Yes. "Can I return worn boots?" Only if worn indoors. Their embeddings almost touch. A loose match here gives a wrong answer, instantly and with confidence. So the bar is set high, and a borderline match is confirmed by a small model before it is used.
Layer 3: the provider keeps the opening ready
Some questions are new, and the model must be called. Even then, most of what it reads is not new. Every prompt opens with the same instructions and tool list. The provider can keep that opening ready after the first call, and read it again for about a tenth of the price, and faster. This is prompt caching, and it happens on the provider's side.
What breaks it
The provider matches the opening from the first character. Change anything early on and the match fails, for everything after it. HM Nayem's first prompt began with the current time. It changed every second, so nothing was ever reused. Parts that never change go first. Anything that varies goes after them.
It is kept for minutes
The provider does not keep an opening for ever. If nobody uses it for a few minutes, it is dropped, and the next call pays full price and a little extra to store it again. For a busy assistant that never happens. For a quiet one at night it happens on almost every call. Prompt caching rewards traffic, and is worth measuring before it is counted on.
Layer 4: keep what the tools said
The last layer sits below the model. The assistant looks up the same popular products hundreds of times an hour, and the catalogue changes a few times a day. Each lookup is kept for ten minutes. The same goes for embeddings: a page's embedding is worked out once and kept until the page's text changes.
Count the hits
A cache nobody measures is a cache nobody can trust. For each layer, HM Nayem counts how often a request was answered from the kept copy. That number is the hit rate. Beside it sit the money saved and, most important, the count of answers later found to be stale or wrong.
When not to keep an answer
Caching is wrong for some things, and it is better to say so than to find out. An answer that should differ each time, such as a product description written to order. Facts that move by the minute, such as stock on a sale day. Anything where a stale answer does harm. For those, the cost of working it out again is the right price.
A question, through every layer, at real speed
A new wording of an old question. No exact match. A match by meaning, well above the bar. The kept answer goes back in three hundredths of a second, and the model is never called. For a truly new question the same path ends at the model, with the opening of the prompt already waiting.
The cheapest answer is one you already have.
- Keep whole answers to common questions, and hand them back without calling the model.
- The key must hold everything that could change the answer.
- Every kept answer needs a time limit and a way to be cleared when its source changes.
- Never keep or share anything personal.
- Matching by meaning saves more and needs a high bar and a second check.
- Put the unchanging part of a prompt first, so the provider can keep it ready.
Ideas in this story
- Response caching: Keeping an answer that was already worked out, and giving it again when the same question, or one that means the same, comes back.
- Prompt caching: The provider remembers the unchanging start of a prompt, so repeats of it cost about a tenth and arrive faster.
- Embeddings: Turning text into a list of numbers so that similar meanings end up close together.
- Access control: Deciding in code, not in the prompt, whose data a request is allowed to touch.
- Monitoring: Watching speed, cost, errors and quality across all requests, with alarms when something shifts.