Semantic Caching for LLM Applications in Java: How It Works and Where It Breaks

Semantic caching stores LLM answers and serves one again when a new prompt means the same thing as an earlier one, judged by how close their embeddings are. A hit costs an embedding call and a vector search instead of a model call. A false hit returns the answer to a different question.

This guide covers what decides whether a semantic cache is safe to run: which scale your similarity threshold is on, how to calibrate it, how to drop answers when the documents behind them change, what happens when you switch embedding models, and which conversations it cannot help. It is written for Java teams running Redis or Valkey. We build Redisson, so the code samples use its RSemanticCache, which is set up in the Redisson section, and the alternatives are listed near the end.

How Semantic Caching Works

The flow has four steps:

1. Embed the incoming prompt with an embedding model.

2. Search the stored prompts for the nearest neighbour by cosine similarity.

3. If the similarity clears a threshold, return the answer stored with that prompt.

4. If it does not, call the model, return its answer, and store the prompt's vector with the answer for next time.

This is cache-aside with a different key. The application checks the cache, calls the expensive service on a miss and writes the result back, but the lookup asks "what have I already answered that means this?" An exact-key cache, such as the one behind Spring's @Cacheable or a plain key-value distributed cache, can only ask "have I seen this exact string?" That one change alters how the cache fails:

"How do refunds work?" and "What is the refund policy?" share no exact key but sit close together in embedding space, so the second one can be a hit. So can "Can I get a refund after 30 days?", which may deserve a different answer. Most of this guide is about telling those apart.

Semantic Caching vs Prompt Caching (and KV Caching)

Three different mechanisms share the word "cache," and search results mix them up:

They are not alternatives. A semantic cache miss still goes to the model, where the provider's prompt caching can discount a repeated prefix, such as a long system prompt or a block of retrieved documents. Anthropic's prompt cache is opt-in through cache_control and has a five-minute default lifetime; OpenAI's is on by default for prompts of 1,024 tokens or more on current models. Both bill cache reads at a fraction of the normal input rate. Anthropic charges more than normal input for writing a prefix to the cache, and so does OpenAI on its newest models, so there a prefix used only once costs extra. Check the current terms in Anthropic's prompt caching documentation and OpenAI's prompt caching guide before you estimate savings.

Where It Pays and Where It Fails

A semantic cache earns its keep where many people ask the same few questions in different words, and where the right answer does not depend on who is asking or when:

An error code or a product SKU that appears in the prompt can also be stored as a filterable field on each entry and required on each lookup, so that "what does error E1234 mean?" can only ever match entries for that code. The isolation section shows how.

Choosing a Similarity Threshold

The two errors are not symmetric

A false miss costs one model call: some latency and some tokens. A false hit answers a question nobody asked, and neither the user nor your code can tell. Tune for precision first and accept a lower hit rate.

Know which scale your threshold is on

Similarity scores are not reported on one scale. Some tools report raw cosine similarity, which runs from -1 to 1. Others report cosine distance, 1 - cos, which runs from 0 to 2 and where lower means closer: the Redis Query Engine does, and so does the distance_threshold of RedisVL for Python. Redisson's vector store and semantic cache report a normalized similarity, (1 + cos) / 2, which maps the range onto 0 to 1 and gives the same value on both vector engines; its lower-level RSearch API returns the Query Engine's raw distance. The same pair of prompts therefore gets three different numbers:

Threshold advice you read elsewhere ("0.85 to 0.95") rarely says which scale it means. Treat any copied similarity as raw cosine until shown otherwise, so 0.85 raw becomes 0.925 normalized, and check whether a threshold is really a distance, where a lower number is stricter. Redisson's default of 0.9 normalized is a raw cosine of 0.80, and its documentation calls that default a placeholder.

Calibrate on labelled prompts

Collect a few hundred real prompts from your logs and label each with the stored entry it should hit, or with "none." Then look at what each candidate threshold would serve. lookupAll returns up to k candidates (five by default), each with its similarity, and moves no hit or miss counter, so it is safe to run against a live cache:

List<CacheHit> candidates = cache.lookupAll(CacheQuery.of(prompt).threshold(0).k(5));
for (CacheHit c : candidates) {
    System.out.printf("%.4f  %s%n", c.similarity().getAsDouble(), c.prompt());
}

Choose the lowest threshold at which no labelled prompt is served a wrong entry, then add a margin. On a first deployment, start at a normalized 0.999 (raw cosine 0.998), which serves only prompts seen verbatim, and lower it as the labelled set grows. A normalized threshold(1.0) is not "exact match": it never hits, because a prompt compared with itself comes back from a float32 search at about 0.9999999. A single call can also raise the bar for itself with CacheQuery.of(prompt).threshold(...), without changing the whole cache.

Keeping Answers Fresh: TTL and Invalidation

An answer generated from retrieved documents goes stale when one of those documents changes, even though no prompt has changed. A TTL alone handles this badly: set it short and the cache barely hits, set it long and it serves last week's policy. The precise fix is to record which documents each answer came from and to remove exactly those answers when a document changes. This is tag-based invalidation, with the document id as the tag:

// Only the retrieval step knows which documents this answer used, so record them now.
cache.put(CacheEntry.of(prompt, answer).dependencies(retrievedDocumentIds));

// policy-7 was edited. Listing first shows what the invalidation will take out.
List<CacheHit> affected = cache.dependents("policy-7", 50);
affected.forEach(h -> log.info("invalidating answer to '{}'", h.prompt()));
InvalidationResult removed = cache.invalidateByDependency(List.of("policy-7"));

All the ways to remove entries:

dependents(id, limit) lists the answers an invalidation by dependency would take, and preview(filter) counts what a filter would match, without removing anything.

For entries written without dependencies, the TTL is the backstop. Leave refreshTtlOnHit off for retrieval-augmented answers: refreshing the TTL on every hit would keep a popular stale answer alive forever. Turn it on for answers that cannot go stale, such as classification over a fixed corpus.

Expiry is logical: lookups skip expired entries, but the store never sets a Redis TTL on them, so only sweepExpired() frees their memory, and an entry with no TTL is never swept. Schedule the sweep and catch its failures; on a server without enumeration (see the Valkey section) it always throws.

ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(() -> {
    try {
        cache.sweepExpired();
    } catch (RuntimeException e) {
        // Letting this escape would cancel the schedule, and expired entries
        // would quietly stop being freed.
        log.warn("semantic cache sweep failed", e);
    }
}, 10, 10, TimeUnit.MINUTES);

When the Embedding Model Changes

Switching to another embedding model, or another version of the same one, changes every vector. A new prompt's embedding is no longer comparable with the stored ones, and a naive cache keeps comparing them anyway, producing scores that mean nothing.

Redisson's RSemanticCache stores a fingerprint with every entry: a record of how the embedding model embeds a fixed probe text. Every lookup filters on the current fingerprint. After a model change, the old entries stop matching, so the change shows up as a run of misses rather than as wrong answers. Expect the hit rate to drop sharply and recover as the cache refills. The old entries remain removable with invalidateAll(), invalidateByDependency() and sweepExpired(), but not with invalidate(prompt), because the entry id includes the fingerprint.

That covers a new model with the same vector width. A model with a different width cannot reopen the existing store at all, so it needs a new store. And on Valkey, or on vector sets before Redis 8.4, where none of those three methods can run, a model change leaves every old entry in memory with no way to remove it through the cache.

Keeping Tenants, System Prompts and Model Settings Apart

A semantic cache shares answers across everyone who asks a similar question, so isolation has to be designed in. Start from how entries are identified: an entry's id is derived from the cache name, the model fingerprint and the normalized prompt, and storing the same prompt again in the same cache replaces the earlier entry.

• Separate caches by name. The cache name scopes every lookup and invalidation, and several caches can share one store, so a cache per tenant, assistant or product costs no extra index. If a cache name carries a tenant id, set meterName to a fixed value so that the cache's meters are not tagged once per tenant.

• Filters narrow lookups, but entries still collide. When two tenants store the same prompt in one cache, the second put replaces the first, field values included, and one tenant's invalidate(prompt) removes the other's entry. Use filterable fields for values that appear in the prompt, such as an error code, or that you invalidate by, such as a prompt-template version. Entries written by the Spring AI advisor or the LangChain4j decorator carry no filterable fields, so filters apply to entries you write through the core API.

• System prompts. The framework integrations include the system prompt in the key, so answers produced under different system prompts do not mix. With the core API the key is only the prompt you pass, so use a cache per system prompt.

• Model settings are not in the key. Requests that differ only in model, temperature, tools or response format share entries. Give each setting its own cache.

• Security boundaries. If tenants must never see each other's answers, give each tenant its own cache, or its own store so that a bug in how you name caches cannot reach another tenant's entries.

Map<String, FieldType> schema = new HashMap<>(SemanticCacheOptions.REQUIRED_SCHEMA);
schema.put("code", FieldType.TAG);
// ... build the store with .filterableSchema(schema) ...

cache.put(CacheEntry.of("what does error E1234 mean?", answer).filterable("code", "E1234"));

// "E1235" barely moves the embedding, so without the filter it could hit the E1234 answer.
Optional<CacheHit> hit = cache.lookup(CacheQuery.of(prompt)
        .filter(VectorFilter.eq("code", errorCodeFrom(prompt))));

A lookup filter can only narrow the cache's own scope, which is this cache, the current model fingerprint and unexpired entries. It matches only entries that were written with the field.

Building a Redis Semantic Cache in Java With Redisson

Redisson 4.8.0, released in October 2026, adds RSemanticCache as part of Redisson AI, a set of modules in Redisson PRO. It needs Java 17 or later. Add the core module and an embedding provider:

<dependency>
    <groupId>pro.redisson</groupId>
    <artifactId>redisson-ai-core</artifactId>
    <version>4.8.0</version>
</dependency>
<dependency>
    <groupId>pro.redisson</groupId>
    <artifactId>redisson-embedding-openai</artifactId>
    <version>4.8.0</version>
</dependency>

The cache lives in an ordinary Redisson vector store, whose schema must declare the cache's required fields and whose metric must be cosine. The store below uses the Query Engine backend, which on Valkey runs over valkey-search with the limits described below; build it with VectorSetStoreOptions instead to use Redis vector sets. Imports are omitted, and the code follows the Redisson reference guide for 4.8.0.

REmbeddingModel model = OpenAiEmbeddingModel.create(
        OpenAiEmbeddingOptions.model("text-embedding-3-small")
                .apiKey(System.getenv("OPENAI_API_KEY"))
                .nativeDimensions(1536),
        RedissonAI.of(redisson).getMeterRegistry());

RVectorStore<CachePayload> store = RedissonAI.of(redisson).getVectorStore(
        RediSearchStoreOptions.hnsw("answers")
                .payloadType(CachePayload.class)
                .payloadCodec(new JacksonCodec<>(CachePayload.class))
                .embeddingModel(model)
                .filterableSchema(SemanticCacheOptions.REQUIRED_SCHEMA));

RSemanticCache cache = RedissonAI.of(redisson).getSemanticCache(
        SemanticCacheOptions.name("support")
                .store(store)
                // Verbatim repeats only (raw cosine 0.998). Lower it once labelled
                // prompts show where false hits start.
                .threshold(0.999)
                .ttl(Duration.ofHours(6)));

Construction checks the store's schema and metric and embeds one fixed probe text to fingerprint the model, so it costs one embedding call. Create the cache once and share it: it is thread-safe.

Looking up and storing

Optional<CacheHit> hit = cache.lookup(prompt);
String answer;
if (hit.isPresent()) {
    CacheHit h = hit.get();
    // The hit answers a stored prompt, not this one. Logging both prompts and the
    // score is what lets you trace a wrong answer back to the threshold.
    log.debug("cache hit {} for '{}', stored as '{}'",
            h.similarity().getAsDouble(), h.askedPrompt(), h.prompt());
    answer = h.response();
} else {
    List<String> sourceIds = retrieveDocumentIds(prompt);   // your retrieval step
    answer = generateAnswer(prompt, sourceIds);               // your model call
    // An empty, truncated or refused answer would otherwise be served to every paraphrase.
    if (isCacheable(answer)) {
        cache.put(CacheEntry.of(prompt, answer).dependencies(sourceIds));
    }
}

Storing the same prompt again overwrites the earlier entry, and a retried put leaves one entry, because the entry id is derived from the cache name, the model fingerprint and the normalized prompt. Redisson caches an empty answer like any other and does no checking of its own.

There is no stampede protection. Callers that miss on the same new prompt at the same moment all call the model, and one of their answers is kept. That wastes model calls but never produces a wrong answer. If a burst of identical new questions is likely, for example right after a launch, take a short distributed lock keyed by the normalized prompt, look the cache up again once you hold it, and call the model only if it still misses: the same single-flight fix as for any thundering herd. Paraphrases that arrive together still each call the model.

Spring AI

Redisson ships two Spring AI semantic cache advisors, both named SemanticCacheAdvisor, so check which artifact an import comes from. The Redisson AI advisor, from redisson-store-spring-ai, puts an RSemanticCache into a ChatClient advisor chain. It is built for Spring AI 2.0.0 and does not bring Spring AI with it, so declare spring-ai-vector-store, spring-ai-model and spring-ai-client-chat yourself. Then add the advisor to a ChatClient:

ChatClient client = ChatClient.builder(chatModel)
        .defaultAdvisors(
                MessageChatMemoryAdvisor.builder(chatMemory).build(),
                new SemanticCacheAdvisor(cache))
        .build();

ChatClientResponse response = client.prompt()
        .user("what is the refund policy?")
        .advisors(a -> a.param(Advisors.DEPENDENCIES_CONTEXT_KEY, List.of("policy-3", "policy-7")))
        .call()
        .chatClientResponse();

CacheHit hit = (CacheHit) response.context().get(Advisors.HIT_CONTEXT_KEY);   // null on a miss

Dependencies come from the request context under Advisors.DEPENDENCIES_CONTEXT_KEY, set per call as above or by your own retrieval advisor ordered before this one. Spring AI's own retrieval advisors do not set it. The other advisor comes from Redisson's older semantic cache module for Spring AI, and the two behave differently in ways that matter in production:

Pick the older module's advisor for chat that must keep working when Redis is down and for cached streaming. Its documentation describes the key as the question and says nothing about conversation history, so test follow-ups such as "and for annual plans?" before you use it in multi-turn chat. Pick the Redisson AI advisor for invalidation by source document (on Redis 8.4+ vector sets or the Redis Query Engine), protection against model changes, and one cache shared with LangChain4j. It also runs on Valkey, but there it can invalidate neither by source document nor all at once, and expired entries are never freed. Either way, calibrate its threshold on its own: the older module's documentation describes its threshold as running from 0 to 1 without giving the formula, so a value tuned for one advisor is no evidence for the other. Both are covered in the documentation for the Redisson AI advisor and the older module.

LangChain4j

LangChain4j has no hook that can answer instead of the model, so redisson-store-langchain4j, which runs on LangChain4j 1.18.x, provides the cache as a ChatModel decorator:

ChatModel cached = new CachingChatModel(chatModel, cache,
        () -> currentRetrieval().stream().map(Match::id).toList());

String answer = cached.chat("what is the refund policy?");
// lastHit() is per thread, across every CachingChatModel: read it on the thread that made the call.
Optional<CacheHit> hit = ((CachingChatModel) cached).lastHit();

On a hit the delegate is never called, so its listeners, retries and observability do not run. The key is the whole conversation in the request. A request that contains an image or other media, or a tool-call message anywhere in its history, bypasses the cache entirely, which in a tool-using agent means every request after the first tool call. Like the Redisson AI advisor, the decorator fails the chat call when the cache or the embedding provider fails.

Running a Semantic Cache on Valkey

Valkey has no vector sets, so a semantic cache on Valkey uses the Query Engine backend over the valkey-search module. Lookups and stores work, and so does removing an entry by its exact prompt. Every other removal, plus preview and dependents, walks the store, which needs a capability called enumeration, and valkey-search does not provide it:

Where the answer is "No," the call throws UnsupportedByBackendException. Answers built from documents that change then have only their TTL to protect them. And the TTL hides expired entries without ever freeing them, because sweepExpired() cannot run, so memory grows with every distinct prompt you store. The same holds for a vector-set store on Redis 8.0 to 8.3.

On Valkey, then, use a semantic cache for answers that do not go stale, and size the instance for everything you will store. Where answers must follow their documents, run the cache on Redis 8 or later with the Query Engine backend, or on Redis 8.4 or later with vector sets, even if the rest of your data stays on Valkey. Valkey vs Redis covers the wider differences between the two engines.

Measuring It

stats() returns the counters of the cache object it is called on: lookups, hits, misses, near misses, expired skips, puts, overwrites, invalidations and swept entries. Most of the same events are reported as Micrometer meters, redisson.cache.hit, .miss, .nearMiss, .expired, .corrupt and .invalidated, plus a redisson.cache.lookup timer, tagged with the cache name and the model fingerprint. Two of them need watching:

• Near misses are misses where candidates existed but scored below the threshold; an ordinary cache miss found no candidate at all. A high near-miss rate is the signal to tune: run lookupAll on a sample of near-missed prompts and check them against your labelled set before you lower the threshold.

• Corrupt counts matching records that came back without a payload, because something other than the cache wrote into its store, or a payload read failed or could not be decoded, for example after a codec change. Those lookups report a miss, not an error, so this counter, with redisson.vectorstore.payload.errors to tell the two causes apart, is the only sign.

For savings, compute your own figure:

net saving ≈ hits × average cost of a model call
           - embedded texts × average cost of embedding one
           - cost of running the cache

Count embedded texts from the redisson.embedding.provider.texts meter rather than assuming one per request: every lookup embeds the prompt, and storing an answer can embed it again. The meter is tagged by provider and model only, so if the same model also embeds your retrieval queries, those texts are in the count too, and you need to subtract them. Redisson's gen_ai.client.token.usage counts the tokens of those embedding calls; read its input and total values separately, because they overlap. Take the chat model's tokens, for the first line of the formula, from your chat client's own metrics. For collecting and alerting on these meters alongside your other Redis metrics, see what to monitor in Redis from Java.

Frequently Asked Questions

What Is Semantic Caching?

Semantic caching stores the answers a language model gives and serves one again when a new prompt means the same thing as an earlier one. It compares prompts by the similarity of their embeddings rather than by their exact text, so paraphrases can hit the cache. A hit replaces a model call with an embedding call and a vector search; a false hit returns the answer to a different question.

What Is the Difference Between Semantic Caching and Prompt Caching?

Semantic caching runs in your own store and returns a whole stored answer when a new prompt is similar to an old one. Prompt caching runs at the LLM provider and reuses the computation for an identical prompt prefix, so the model still generates a new answer, at a lower input cost. Only semantic caching skips the model call, and only semantic caching can return a wrong answer. Two related terms are different again: semantic search retrieves documents by meaning, and RAG passes those documents to the model to write an answer.

What Similarity Threshold Should a Semantic Cache Use?

There is no universal value, and the scale matters as much as the number. On Redisson's normalized scale, (1 + cos) / 2, the default of 0.9 equals a raw cosine similarity of 0.80, and the documentation calls it a placeholder. On a first deployment, start at 0.999, which serves verbatim repeats only. Then calibrate on a few hundred labelled prompts: choose the lowest threshold that serves no wrong answer, and add a margin.

How Do I Stop a Semantic Cache From Serving Stale Answers?

Record which source documents each answer was built from when you store it, and remove those answers when a document changes. In Redisson that is invalidateByDependency. Use a TTL as the backstop for answers stored without dependencies, leave TTL refresh on hits turned off for retrieval-augmented answers, and schedule a sweep, because expired entries are only hidden until they are swept.

Can Semantic Caching Work With Streaming Responses?

It depends on the integration. The advisor in Redisson's older Spring AI semantic cache module (redisson-spring-ai-semantic-cache-20) caches a streamed response once the stream completes. The redisson-store-spring-ai advisor is a call advisor and passes streamed requests through uncached. The LangChain4j CachingChatModel decorates ChatModel only, so a StreamingChatModel is not cached. With the core RSemanticCache API you decide, by storing the assembled answer after the stream ends.

Does Semantic Caching Work on Valkey?

Yes, through the valkey-search module, with limits. With Redisson, lookups, stores and removal by exact prompt work on valkey-search, but invalidation by source document or filter, and the sweep that frees expired entries, need enumeration, which valkey-search does not provide. Use it there for answers that do not go stale, or run the cache on Redis 8 or later with the Query Engine backend, or Redis 8.4 or later with vector sets, where answers must follow their documents.

Next Steps

Start by collecting real prompts from your logs, before any code. A few hundred of them, grouped by which ones should share an answer, tell you whether your workload repeats itself enough to be worth caching. Once the cache runs at 0.999, the same set shows how far the threshold can move. Then decide how answers will leave the cache: by dependency where they come from documents, by TTL where they do not.

The vector store underneath is covered in Redis vector search in Java, and the wider question of when Redis makes sense for vector workloads in Redis as a vector database. The full API is in the Redisson AI documentation.

Similar articles

• Redis Vector Search in Java: Vector Sets, the Query Engine, and What to Use When
• Redis Vector Database: Vector Search and RAG in Java
• What is Cache Invalidation?
• What is the cache-aside pattern?
• What is the thundering herd problem?
• Spring Boot Redis Cache: A Complete Guide to @Cacheable