Strategy 2: Semantic Caching

Semantic caching matches requests by meaning, not exact text. "How do I reset my password?" and "I forgot my password, how do I fix it?" would both hit the same cache entry because they're semantically equivalent.

How it works

1. Generate an embedding vector for each request using a cheap embedding model
2. Store the embedding + response in a vector database
3. On new requests, search for the nearest embedding (cosine similarity above a threshold)
4. On hit: return cached response; On miss: call API, store embedding + response

import numpy as np
from openai import OpenAI

client = OpenAI()

def get_embedding(text):
    response = client.embeddings.create(input=text, model="text-embedding-3-small")
    return response.data[0].embedding

def semantic_cached_completion(messages, model="gpt-4o-mini",
                                similarity_threshold=0.92, **kwargs):
    # Combine messages into a single query string for embedding
    query_text = " ".join(m["content"] for m in messages)
    query_embedding = get_embedding(query_text)

    # Search vector DB for similar cached queries
    similar = vector_db.search(query_embedding, top_k=1,
                                threshold=similarity_threshold)

    if similar:
        return similar[0]["response"]

    # Cache miss
    response = client.chat.completions.create(
        messages=messages, model=model, **kwargs
    )
    result = response.model_dump()

    # Store embedding + response
    vector_db.insert(query_embedding, {
        "response": result,
        "query": query_text,
        "model": model
    })
    return result

Semantic caching trade-offs

Tuning the similarity threshold

The threshold controls how "similar" queries need to be to count as a cache hit. Too low and you'll return wrong answers; too high and you'll miss real matches.

• 0.95+ — Very conservative. Almost exact meaning. Low risk of wrong answers.
• 0.90-0.95 — Balanced. Good hit rates with minimal quality risk.
• 0.85-0.90 — Aggressive. Higher hit rates but risk of semantically different queries matching.

Start at 0.92 and adjust based on your quality metrics. For support bots where accuracy matters, stay above 0.93.

Strategy 3: Provider Prompt Caching

Both OpenAI and Anthropic now offer built-in prompt caching at the API level. This automatically caches the prefix of your prompts and gives you a discount on subsequent requests that share the same prefix.

OpenAI Prompt Caching

• Automatically caches prompts longer than 1,024 tokens (GPT-4o) or 2,048 tokens (GPT-4o mini)
• 90% discount on cached input tokens (you pay only 10% of the input price)
• Cache entries expire after 5-10 minutes of inactivity
• No code changes needed — it's automatic for supported models

Anthropic Prompt Caching

• Cache prefixes up to 4,096 tokens with explicit caching markers
• 90% discount on cached input tokens
• Cache TTL is 5 minutes (extended on each hit)
• Requires adding cache_control parameter to mark the cache breakpoint

# Anthropic prompt caching example
response = client.messages.create(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a helpful assistant for Acme Corp. [long system prompt...]",
            "cache_control": {"type": "ephemeral"}  # Cache this prefix
        }
    ],
    messages=[{"role": "user", "content": "What's the return policy?"}]
)

When provider caching helps most

• Long system prompts — if your system prompt is 500+ tokens, every request after the first pays 10% instead of 100%
• Multi-turn conversations — conversation history is the prefix; only new messages are full price
• RAG applications — large context documents in the prompt get cached automatically

Combining Strategies for Maximum Savings

The best results come from layering multiple caching approaches. Here's a production architecture that combines all three:

1. Layer 1 — Exact-match cache (Redis). Fast, zero-cost lookups. Catches duplicate requests.
2. Layer 2 — Semantic cache (Vector DB). Catches paraphrased versions of cached queries.
3. Layer 3 — Provider prompt caching (Automatic). Even on cache misses, the system prompt prefix is cached by the provider at 90% off.

On a cache miss at Layer 1, you check Layer 2. If both miss, the API call still benefits from provider caching on the prompt prefix. The result: you rarely pay full price for any request.

Cache Invalidation: The Hard Part

Caching is easy. Invalidating caches correctly is where most teams struggle. Strategies:

• Time-based expiration (TTL) — simplest approach. Set 1-24 hour TTLs based on how stale your data can be. Pricing data? 24 hours is fine. Real-time support? 5-15 minutes.
• Event-driven invalidation — when your data changes (new pricing, updated docs), invalidate affected cache entries. More complex but more precise.
• Version-prefixed keys — include a version number in cache keys: v2:hash(prompt). Bump the version to invalidate everything at once.
• Write-through caching — update cache and DB simultaneously on writes. Ensures consistency but adds write latency.

Rule of thumb: Use TTL for most cases. Only build event-driven invalidation if stale responses would cause user-facing issues.

Measuring Cache Performance

Track these metrics to know if your caching is working:

— See if you're overpaying for AI APIs