What is response caching?
Many products are asked the same things over and over. A response cache keeps the finished answer under a key, and returns it at once the next time, with no model call at all.
The key must contain everything that could change the answer: the question, the language, who is asking and what they may see, and the version of the prompt. A key that leaves one of those out will sooner or later show one person an answer meant for another.
A stricter cache matches only identical questions. A looser one matches by meaning, using embeddings, so that differently worded questions share an answer. That saves more and is riskier, because two questions can be close in meaning and need different answers.
Every cached answer needs a way to go stale: a time limit, and a way to clear it when the source it came from changes. Personal answers and fast-moving facts are usually not cached at all.
Why does response caching matter?
Products are asked the same things over and over. Working out the same answer each time costs money and makes people wait. Keeping an answer and handing it back when the question returns is the cheapest speed-up there is, provided it is done carefully enough never to give the wrong person the wrong answer.
How does response caching work?
- When an answer is worked out, it is filed under a key.
- The key holds everything that could change the answer: the question, the language, who may see it, the prompt version.
- When a request arrives, the key is looked up first.
- If it is found and still fresh, the kept answer is returned and no model is called.
- Kept answers expire after a set time, and are cleared when their source changes.
When should you use response caching?
Use it when
- The same questions recur
- The answer does not depend on who is asking
- The source changes rarely
Think twice when
- The answer is personal
- The facts move by the minute
- The reply should be different each time
Where does response caching sit in a real system?
The cache sits in your own system, in front of the model call. A stricter form matches identical questions. A looser one matches by meaning, using embeddings, and needs a high bar and a second check.
What goes wrong with response caching?
- Leaving the user's permissions out of the key, so one person sees another's answer.
- No way to clear answers when the underlying page changes.
- Matching by meaning too loosely, so a near question gets the wrong answer.
What does response caching look like in practice?
"What time do you open on Saturday?" is asked 400 times a day. The first answer is kept for 24 hours. The other 399 are returned in 4 thousandths of a second at no cost. When the opening-hours page is edited, every answer built on it is cleared at once.