Predictable prompt caching is an optimization technique used by LLM providers where the model reuses computation for the unchanged prefix of a prompt instead of processing it from scratch every time.
The idea is simple:
If the beginning of your prompt stays exactly the same across requests, the provider can cache the internal representations (KV cache or equivalent).
On the next request, only the new or changed part needs to be processed.
This reduces latency, cost (for providers that discount cached tokens), and sometimes GPU usage.
Example
Instead of sending completely different prompts:
Request 1
You are a financial analyst.
Follow these 25 rules.
Company policy...
Database schema...
Previous conversation...
Question:
What was revenue in Q1?
Request 2
You are a travel planner.
Suggest places in Japan.
Nothing is reusable.
Now imagine:
STATIC PART
----------------------------------
You are a financial analyst.
Follow these 25 rules.
Company policy...
Database schema...
Available tools...
Previous context...
----------------------------------
Question:
What was revenue in Q1?
Later:
STATIC PART
----------------------------------
You are a financial analyst.
Follow these 25 rules.
Company policy...
Database schema...
Available tools...
Previous context...
----------------------------------
Question:
Compare Q1 and Q2 revenue.
Everything above Question: is identical.
Only the final question changes.
The provider can reuse the cached prefix.
Why "predictable"?
It means you intentionally design prompts so that the reusable portion is always in the same place.
Instead of :
Question --> Instructions --> Database schema -- > you always do
Instructions --> Schema --> Examples Conversation --> User Question
The cache becomes predictable because the prefix rarely changes.
What is usually cached?
Typically:
System prompt
↓
Developer instructions
↓
Knowledge base
↓
Tool descriptions
↓
Few-shot examples
↓
Conversation history
--------------------------
NEW USER MESSAGE
Everything before the latest user message may be reusable.
Visual representation
Request 1
+--------------------------------------+
| System Prompt |
| Company Policy |
| SQL Schema |
| Few-shot Examples |
| Conversation |
+--------------------------------------+
| New Question |
+--------------------------------------+
↓ Cached
Request 2
+--------------------------------------+
| System Prompt | ✓
| Company Policy | ✓
| SQL Schema | ✓
| Few-shot Examples | ✓
| Conversation | ✓
+--------------------------------------+
| Different Question |
+--------------------------------------+
Only the bottom portion needs fresh processing.
In AI Agents
Suppose you're building an SQL assistant.
Without caching:
Every query sends
- 3000 token system prompt
- 5000 token schema
- 2000 token examples
- user query
Total:
10,200 tokens
every request.
With predictable caching:
First request:
10,200 tokens
Later requests:
cached 10,000 tokens
new 200 tokens
Much faster and potentially cheaper.
Benefits
| Benefit | Explanation |
|---|---|
| Faster responses | Less repeated computation on unchanged prompt prefixes |
| Lower cost | Some providers bill cached input tokens at a reduced rate |
| Lower GPU load | Fewer computations for repeated context |
| Better scalability | Useful for long system prompts and RAG applications |
Best practices
1. Put static instructions first
Good:
System prompt
Rules
Examples
Schema
User question
Bad:
Question
Rules
Schema
Examples
2. Keep the prefix identical
Even small edits can prevent a cache hit.
Changing
Answer briefly.
to
Answer very briefly.
may invalidate the cached prefix from that point onward.
3. Append new information
Instead of rewriting the whole prompt:
Conversation
New message
avoid
New conversation summary
Different ordering
Stable ordering improves cache reuse.
4. Avoid timestamps in the cached section
Bad:
Today's time:
12:01:43
That changes every request.
Instead:
System instructions
...
User question:
Current time is ...When it matters most
Predictable prompt caching provides the biggest gains for applications that repeatedly send large amounts of identical context, such as:
AI chatbots with long system prompts
RAG (Retrieval-Augmented Generation) systems
Coding assistants with large codebases
SQL assistants with database schemas
Customer support bots with policy documents
Multi-agent workflows where each agent shares the same instructions
For short prompts, the benefit is usually minimal because there isn't much repeated computation to reuse. Designing prompts with a stable, unchanging prefix and appending new information at the end is the key principle behind predictable prompt caching.
Join the Conversation