Semantic Caching

What it is

A cache that matches on meaning.

An ordinary cache needs the same string twice. A semantic cache keys on meaning instead: you embed the incoming request, and if a past request is semantically close, you return its cached answer. "Reset my password" and "how do I change my password" become one entry. You implement it with GPTCache, a Redis vector index, or a gateway's built-in cache (Portkey and LiteLLM both ship one), so it is as much a pattern as a product.

The one job

Stop paying for the same question twice.

It stops you paying for the same question asked in different words. When traffic repeats, that is the difference between billing every rephrasing and billing only the genuinely new ones.

Reach for it when

• You have repetitive or overlapping queries: FAQs, support, common lookups.
• Your workload is read-heavy and answers are safe to reuse.
• You want a real cost cut: the 40–70% redundant-call reduction is realistic.
• You already run an embedding model and a vector store you can lean on.

Skip it when

• Answers are personalized or stateful, where a near-match is wrong and dangerous.
• Traffic is low-repeat, so the cache rarely hits and adds only overhead.
• You have strict freshness needs and a stale answer is unacceptable.

A minimal look

Embed, look up, answer if close.

// embed the request and find the nearest past question
const q = await embed(request);
const hit = await store.nearest(q);   // nearest-neighbour over cached questions

// only trust the cache above a similarity threshold
if (hit && hit.score > 0.92) {
  return hit.answer;             // the call you never make
}

const answer = await callModel(request);
await store.put(q, answer);        // remember it for next time

The principle it teaches

Where it fits your labs

This is a Week 3 tool with a Week 5 tail. Lab 3 has you add a semantic-cache step and measure the call reduction; the same lab is where you learn to guard the threshold so cheap answers do not become wrong ones.