GPTCache Alternatives for Production Semantic Caching

What GPTCache is, and where its README says it stops

GPTCache is an MIT-licensed Python library that embeds a query, searches a vector store for a close match, and returns the stored response on a hit. Its README lists modular pieces for embeddings, cache storage, vector stores, eviction, and similarity evaluation, and it supports SQLite, PostgreSQL, and other stores for responses alongside FAISS, Milvus, PGVector, Chroma, Qdrant, and Weaviate for vectors.

The README also sets limits worth reading before you build on it. It says that because large model APIs keep changing shape, the project no longer adds support for new APIs or models, and it points users to the get and set API instead. Its adapter checklist leaves the line for other LLMs, naming Hugging Face Hub, Bard, and Anthropic, unchecked. It states that in-memory eviction decisions rely solely on the number of lines, which can cause out-of-memory errors, and that not every combination of modules is compatible. The repository still shows a commit on 2026-09-22, so it is active, but the README itself was last changed in July 2025.

Option 1: a semantic cache inside the gateway

LiteLLM's proxy supports three semantic backends: Redis, Valkey, and Qdrant. You set the cache type, an embedding model defined in your model list, and a similarity_threshold in config.yaml, and a cached response carries an x-litellm-semantic-similarity header. Because the cache sits in the proxy, every client that calls the gateway gets it without code changes.

The documentation is direct about two risks. First, the semantic key is built from the whole messages array, so on multi-turn or agent traffic consecutive turns are about 0.99 similar and the client replays a stale response, typically an agent repeating the same tool call. Raising the threshold does not reliably fix this. The recommendation is to keep semantic caching for single-shot traffic and exclude agent keys, for example with a per-key no-cache setting. Second, by default every end user behind one virtual key shares one semantic bucket, so a response generated for one of them can be served to another. Setting semantic_cache_scope to end_user isolates buckets per end user.

There are operational limits too. The embedding call runs inline before the request can proceed and is capped at 5 seconds by default, after which the request falls through as a miss. On AWS ElastiCache, vector search for Valkey needs a node-based 8.2+ cluster, and ElastiCache Serverless does not support vector search.

Option 2: a hosted vector index with a small client

Upstash Semantic Cache is an MIT-licensed Node.js package that stores entries in an Upstash Vector database. You create the index with a pre-made embedding model, then call set and get with a minProximity between 0 and 1. Its README gives 0.95 as very high similarity and 0.75 as low similarity, and it supports namespaces so you can partition entries by user or tenant.

This fits a TypeScript service that wants a cache without running Redis or Qdrant. The tradeoff is that the library is a client for one vendor's index, the cache logic lives in your application instead of a shared gateway, and the repository's last commit was 2024-11-21. Treat it as a small building block you own, not a managed control plane.

Worked example: net savings after embedding and storage

The numbers below are hypothetical and exist to show the arithmetic. Assume 100,000 requests a month at $0.0165 each, which is $1,650.00. Assume half the traffic is eligible single-shot traffic and 30 percent of eligible requests hit the cache, so 15 percent of all requests are served from cache.

• Avoided model cost: 15,000 requests x $0.0165 = $247.50.
• Embedding cost: the lookup embeds the whole messages array on every eligible request. At 3,000 tokens and a hypothetical $0.02 per million tokens, 50,000 lookups x 3,000 = 150 million tokens, or $3.00.
• Vector storage and hosting, hypothetical: $50.00 a month.
• Net monthly saving: $247.50 - $3.00 - $50.00 = $194.50, about 11.8 percent of the $1,650.00 bill.
• False hits: if 2 percent of the 15,000 hits are wrong, that is 300 incorrect answers a month. Count remediation time before declaring a win.

How to choose

Choose the gateway cache when several services share one proxy and you need per-key and per-end-user controls. Choose a thin client over a hosted index when you have one Node.js service and want the smallest moving parts. Keep GPTCache when you are in Python, already run one of its supported stores, and accept that you own provider adapters through its get and set API.

In every case, start with an exact-match cache, exclude agent and personalized routes, and log the similarity score and matched key on each hit so a bad answer can be traced. Our guide to semantic caching without tenant risk covers the partitioning checklist, and the prompt caching comparison covers the provider-side layer.