Semantic Caching for LLM Apps: Savings Without Cross-Tenant Risk

Exact and semantic caches solve different problems

An exact cache hashes a normalized request and returns a stored response only for the same key. A semantic cache embeds the request and searches for a nearby prior request. The semantic version can catch paraphrases, but similarity is not equivalence.

Start with exact matches on stable, deterministic routes. Add semantic matching only where the cost of a false hit is understood and an evaluation set exists.

Partition before comparing similarity

Never search one global vector index for every customer and route. The lookup key should include organization, team or application, route family, system prompt version, tool availability, relevant model settings, locale, and a cache-policy version. Sensitive or user-specific routes should bypass entirely.

This is both a privacy control and a correctness control. Two prompts can look similar while relying on different hidden instructions, entitlements, or current data.

Choose thresholds with false-hit tests

A cosine threshold such as 0.97 can be a conservative starting hypothesis, not a permanent truth. Build pairs of requests that should match and pairs that look close but must not match. Measure false positives by route and language before relaxing the threshold.

Store the matched key, similarity score, age, namespace, and policy reason in the request ledger. That makes a suspicious answer traceable to its source.

• Use TTLs that reflect source freshness.
• Invalidate on system-prompt or tool-schema changes.
• Bypass personalized and transactional routes.
• Redact or avoid storing prompt previews.
• Track hit rate and false-hit rate separately.

Calculate effective savings

Count embedding calls, vector storage, lookup latency, invalidations, and false-hit remediation. A cache with a high hit rate can still be harmful if it returns stale or incorrectly scoped answers. The valuable metric is verified cost avoided on eligible traffic.