AI Engineering, story 29Never work it out twice

The same question, 400 times a day

1 of 14

Never work it out twice: AI Engineering, story 29

Four hundred people a day ask the shop when it opens, and each answer is paid for. See four places an answer can be kept, and how each one goes wrong.

Answers, prompt openings and tool results can each be kept and reused. Every cache needs the right key and a way to go stale safely.

The story, step by step

  1. The same question, 400 times a day

    Four hundred people a day ask the shop's assistant when it opens on Saturday. Each time, the question goes to the model with the full prompt, and the model works out the same answer, at the same price, in the same second and a half. The four hundredth answer cost as much as the first. That is the waste this story removes.

  2. Layer 1: keep the whole answer

    The simplest fix is to keep the answer. The first time a question is asked, the reply is filed under the question's text. The next time the same text arrives, the reply is handed back from the file. No model is called. The answer arrives in a few thousandths of a second and costs nothing.

  3. A key is more than the question

    The label an answer is filed under is called its key, and choosing it is where caches go wrong. The same question can deserve different answers: in another language, for a member with a discount, under a newer version of the instructions. Everything that could change the answer must be part of the key. Leave one out, and someone gets an answer meant for somebody else.

  4. Answers go stale

    In December the shop opens at ten. The kept answer still says nine. Every cached answer needs two ways to die. A time limit, after which it is thrown away and worked out afresh. And a link to the page it came from, so that when someone edits the opening hours, every answer built on that page is cleared at once.

  5. Never keep what is personal

    "Where is my order?" is asked just as often as the opening hours. Its answer must never be kept and shared, because it belongs to one person. The rule is set in code, by kind of question. Anything that used a tool about a particular customer is not filed at all.

  6. Layer 2: match by meaning

    Exact matching catches less than you would hope. People ask the same thing forty different ways. So each kept question is also stored by its meaning, using an embedding, as the search stories showed. A new question is compared with those. If one is close enough, its answer is used, though not one word matches.

  7. Too close is dangerous

    Two questions can be nearly the same in meaning and need opposite answers. "Can I return boots?" Yes. "Can I return worn boots?" Only if worn indoors. Their embeddings almost touch. A loose match here gives a wrong answer, instantly and with confidence. So the bar is set high, and a borderline match is confirmed by a small model before it is used.

  8. Layer 3: the provider keeps the opening ready

    Some questions are new, and the model must be called. Even then, most of what it reads is not new. Every prompt opens with the same instructions and tool list. The provider can keep that opening ready after the first call, and read it again for about a tenth of the price, and faster. This is prompt caching, and it happens on the provider's side.

  9. What breaks it

    The provider matches the opening from the first character. Change anything early on and the match fails, for everything after it. HM Nayem's first prompt began with the current time. It changed every second, so nothing was ever reused. Parts that never change go first. Anything that varies goes after them.

  10. It is kept for minutes

    The provider does not keep an opening for ever. If nobody uses it for a few minutes, it is dropped, and the next call pays full price and a little extra to store it again. For a busy assistant that never happens. For a quiet one at night it happens on almost every call. Prompt caching rewards traffic, and is worth measuring before it is counted on.

  11. Layer 4: keep what the tools said

    The last layer sits below the model. The assistant looks up the same popular products hundreds of times an hour, and the catalogue changes a few times a day. Each lookup is kept for ten minutes. The same goes for embeddings: a page's embedding is worked out once and kept until the page's text changes.

  12. Count the hits

    A cache nobody measures is a cache nobody can trust. For each layer, HM Nayem counts how often a request was answered from the kept copy. That number is the hit rate. Beside it sit the money saved and, most important, the count of answers later found to be stale or wrong.

  13. When not to keep an answer

    Caching is wrong for some things, and it is better to say so than to find out. An answer that should differ each time, such as a product description written to order. Facts that move by the minute, such as stock on a sale day. Anything where a stale answer does harm. For those, the cost of working it out again is the right price.

  14. A question, through every layer, at real speed

    A new wording of an old question. No exact match. A match by meaning, well above the bar. The kept answer goes back in three hundredths of a second, and the model is never called. For a truly new question the same path ends at the model, with the opening of the prompt already waiting.

The cheapest answer is one you already have.

Ideas in this story