Prompt Caching

Every time you send a request to an LLM, the model processes every token in your prompt from scratch — including the system prompt, any injected documents, and few-shot examples that are identical across all requests. For enterprise applications with long system prompts (compliance policies, persona instructions, formatting guides) or document-grounded QA (the same contract or manual referenced in hundreds of questions), this redundant computation is pure waste.

Provider-level prompt caching (supported by Anthropic Claude, Google Gemini, and increasingly OpenAI) works by storing the transformer's key-value (KV) cache state for a defined prefix of the prompt. When a subsequent request shares that same prefix, the model skips recomputing those tokens and starts processing from the cache hit. Anthropic's implementation charges cached tokens at roughly 10% of the normal input token price — making it a 90% discount on repeated context. Application-level semantic caching (via tools like GPTCache and Zep) is a complementary strategy that caches entire prompt-response pairs based on semantic similarity, serving identical or near-identical responses without hitting the model at all.

The business case for prompt caching is straightforward math. An enterprise support chatbot with a 4,000-token system prompt and an average of 50 user messages per day will spend more than 95% of its token budget re-processing that system prompt repeatedly. Enabling provider-level caching on the static prefix alone, at scale across thousands of users, converts that spend into 90% savings while simultaneously reducing response latency — since cached prefix processing is significantly faster than full computation.