What is prompt caching?
Most products send the same long instructions with every request. Normally that text is processed, and paid for, in full each time. With prompt caching, the provider stores the processed form of the start of the prompt for a few minutes.
When the next request begins with exactly the same text, that part is read from the store at roughly a tenth of the price, and the reply starts sooner. Only the new part at the end is processed in full.
The match runs from the first token to the first difference. So the order of a prompt matters: put what never changes first and what changes last. One changing detail near the top, such as the current time, undoes the whole saving.
Why does prompt caching matter?
Most of what a product sends to a model is the same on every call: the instructions, the tool descriptions, earlier turns of the conversation. Paying full price to have that read again each time is waste. Prompt caching lets the provider keep the repeated opening ready, cutting its cost to about a tenth and making replies start sooner.
How does prompt caching work?
- Your prompt is arranged so that everything that never changes comes first.
- On the first call, the provider processes that opening and keeps the result for a few minutes.
- On later calls that start with exactly the same text, it reuses the kept result.
- Only the new part at the end is processed, and charged, in full.
When should you use prompt caching?
Use it when
- Prompts share a long, fixed opening
- An agent makes many calls in one job
- The same instructions are applied to thousands of documents
Think twice when
- Every prompt is different from its first words
- Calls are so rare that the kept opening expires between them
Where does prompt caching sit in a real system?
The keeping happens at the provider. Your side of it is the order of the prompt, decided by the prompt builder: fixed parts first, anything that varies after them.
What goes wrong with prompt caching?
- Putting the current time or a user's name at the very start, so nothing ever matches.
- Reordering the tool list between calls.
- Counting on the saving for a product with little traffic, where the kept opening has usually expired.
What does prompt caching look like in practice?
Every call to a support assistant begins with 2,000 tokens of instructions and tools. With the opening kept ready, those are charged at a tenth. The cost of one chat falls from 3.1 cents to 1.2. A timestamp on the first line had been preventing any saving at all.