Multi-Layer Semantic Caching for Production LLM Systems

Introduction In a previous article , I described building an agentic search framework in Go. While that architecture handled the functional requirements well, operating it at scale revealed significant cost and latency challenges. At millions of queries per month, LLM API costs, and P95 latency approached 5 seconds. This article presents the semantic caching architecture we implemented to address these issues. The system reduced LLM costs by 45-50% and improved P95 latency to under 2 seconds, while maintaining response freshness guarantees. The key insight: caching at multiple granularities within the agentic pipeline provides better results than end-to-end response caching alone. Specifically, caching the agent's planning decisions—which are deterministic and independent of result freshness—achieved a 50% hit rate even with conservative similarity thresholds. Problem Analysis Cost Structure A single query in an agentic search system involves multiple LLM calls: Planning/Tool Selection (~8,500 input tokens): Agent reads tool definitions and decides which tools to invoke Tool Execution (minimal cost): External API calls Summarization (~24,000 tokens): LLM formats tool outputs into natural language And couple of smaller models for rewriting the query, and selecting the tools based on the query, and tool responses. Why Traditional Caching Fails String-based caching provides minimal hit rates for natural language queries: "weather in san francisco" → cache key: hash_1 "what's the weather in sf" → cache key: hash_2 (miss) "san francisco weather" → cache key: hash_3 (miss) "tell me about san francisco weather"→ cache key: hash_4 (miss) These queries are semantically identical but produce different cache keys. Our initial implementation with exact string matching achieved only ~15% hit rate. The Freshness Challenge Semantic similarity matching introduces a new problem: distinguishing between queries that should produce identical responses versus those requiring fresh data. Consider: "who invented the telephone" (answer never changes) "what's the weather today" (answer changes daily) Both queries might have similar embeddings to cached entries, but only the first should retrieve cached results. This requires query classification and validation beyond pure similarity matching. Architecture Design Multi-Layer Caching Strategy Rather than caching only the final output, we implemented caching at three distinct layers in the processing pipeline: Query Input ↓ ┌─────────────────────────────┐ │ Agent Planning & Tool │ ← Layer 1: Planner Cache (50% hit rate) │ Selection │ Cache tool calls, not tool results └─────────────────────────────┘ ↓ ┌─────────────────────────────┐ │ Tool Execution │ No caching (requires fresh data) └─────────────────────────────┘ ↓ ┌─────────────────────────────┐ │ Response Generation/ │ ← Layer 2: Summarization Cache (35% hit rate) │ Summarization │ Cache only when tools used static data └─────────────────────────────┘ ↓ Final Response ← Layer 3: End-to-End Cache (3% hit rate) Each layer targets different characteristics: Layer 1 (Planning) : High hit rate, deterministic outputs, no freshness concerns Layer 2 (Summarization) : Medium hit rate, requires freshness validation Layer 3 (End-to-End) : Low hit rate but zero computation when hits occur Core Components The semantic cache system consists of: Embedding Generator : Converts queries to dense vector representations (768-dimensional) Vector Database (ANN) : Stores embeddings with approximate nearest neighbor search Key-Value Store : Stores response payloads separately from embeddings Query Classifier : Categorizes queries as evergreen vs time-sensitive Implementation Layer 1: Agent Planning Cache The agent planning step is particularly well-suited for caching because: Tool selection depends only on query intent, not result freshness Planning outputs are deterministic for semantically similar queries This is the most expensive operation (6,500+ tokens per call) type PlannerCache struct { embeddingClient EmbeddingClient vectorDB VectorDB kvStore KVStore } type CacheContext struct { ModelVersion string ToolVersions []string // sorted Locale string EmbeddingVersion string Temperature float32 } func (pc *PlannerCache) Get(query string, ctx CacheContext) (*ToolCalls, bool) { // Generate embedding embedding := pc.embeddingClient.Embed(query) // Create cache key from context contextHash := hashContext(ctx) // Search with conservative threshold candidates := pc.vectorDB.Search(VectorSearchRequest{ Embedding: embedding, Tags: contextHash, Threshold: 0.98, // Near-exact matching Limit: 10, }) // Verify context match and retrieve from KV store for _, candidate := range candidates { if candidate.ContextHash == contextHash { if toolCalls := pc.kvStore.Get(candidate.Key); toolCalls != nil { return toolCalls, true } } } return nil, false } func (pc *PlannerCache) Set(query string, ctx CacheContext, toolCalls *ToolCalls, ttl time.Duration) { embedding := pc.embeddingClient.Embed(query) contextHash := hashContext(ctx) key := generateKey(embedding, contextHash) // Store in both vector DB (for similarity search) and KV store (for retrieval) pc.vectorDB.Insert(embedding, key, contextHash) pc.kvStore.Set(key, toolCalls, ttl) } Cache Context Importance : Including model version, tool versions, and other configuration parameters in the cache key prevents serving stale plans after system updates. When tool definitions change, the context hash changes, effectively invalidating old cache entries. Layer 2: Summarization Cache with Freshness Validation The summarization cache requires additional logic to prevent serving stale responses: func shouldCacheSummarization(query Query, toolResults []ToolResult) bool { // Multi-turn queries have context dependencies if query.IsFollowUp { return false } // Check tool result freshness for _, result := range toolResults { switch result.Source { case "web_fresh", "web_daily": // Results from frequently-updated sources return false case "static_index": // Results from weekly-updated index are cacheable continue } } // Additional constraints return query.IsSimpleQuery && query.IsSingleStep && len(toolResults) == 1 } This conservative approach accepts a lower hit rate (35%) to ensure response freshness. Only queries that exclusively use static data sources are cached. Layer 3: End-to-End Cache The end-to-end cache serves as a catch-all for repeated identical queries: func (sc *SemanticCache) GetEndToEnd(query string, ctx CacheContext) (*Response, bool) { // First try exact match exactKey := md5Hash(normalize(query)) if resp := sc.kvStore.Get(exactKey); resp != nil { return resp, true } // Fall back to semantic search with high threshold embedding := sc.embeddingClient.Embed(query) candidates := sc.vectorDB.Search(VectorSearchRequest{ Embedding: embedding, Tags: hashContext(ctx), Threshold: 0.98, Limit: 5, }) for _, candidate := range candidates { if resp := sc.kvStore.Get(candidate.Key); resp != nil { return resp, true } } return nil, false } Freshness Control: Two-Stage Gating To prevent serving stale responses, we implement a two-stage freshness check: Stage 1: Query Classification A BERT-based classifier categorizes queries as evergreen (static answers) or time-sensitive (dynamic answers): type QueryClassifier struct { model BERTClassifier } func (qc *QueryClassifier) Predict(query string) (isEvergreen bool, confidence float64) { features := qc.extractFeatures(query) logits := qc.model.Forward(features) isEvergreen = logits[0] > logits[1] // [evergreen_logit, time_sensitive_logit] confidence = softmax(logits)[0] if isEvergreen else softmax(logits)[1] return isEvergreen, confidence } The classifier is trained with 95%+ precision at the cost of recall. This conservative tuning ensures we rarely cache time-sensitive queries incorrectly. Stage 2: Post-Execution Validation Even if the classifier isn't confident, we can still cache if we verify that only static data sources were used: func (sc *SemanticCache) GetOrCompute( query string, ctx CacheContext, compute func() (Response, error), ) (Response, error) { // Stage 1: Query classification isEvergreen, confidence := sc.classifier.Predict(query) if isEvergreen && confidence > 0.95 { // High confidence evergreen - try cache if cached, found := sc.Get(query, ctx); found { return cached, nil } } // Execute computation response, err := compute() if err != nil { return nil, err } // Stage 2: Validate based on execution results shouldCache := false if isEvergreen && confidence > 0.95 { shouldCache = true } else if response.OnlyUsedStaticSources() { shouldCache = true } if shouldCache && response.IsCacheable() { sc.Set(query, ctx, response, 7*24*time.Hour) } return response, nil } Embedding Generation We use a 768-dimensional embedding model (similar to BERT base) for semantic similarity: type EmbeddingClient struct { endpoint string model string } func (ec *EmbeddingClient) Embed(text string) []float32 { // Normalize text normalized := strings.ToLower(strings.TrimSpace(text)) // Call embedding service resp := ec.callEmbeddingAPI(EmbeddingRequest{ Text: normalized, Model: ec.model, }) // L2 normalize for cosine similarity return l2Normalize(resp.Embedding) } func l2Normalize(vec []float32) []float32 { var norm float32 for _, v := range vec { norm += v * v } norm = sqrt(norm) normalized := make([]float32, len(vec)) for i, v := range vec { normalized[i] = v / norm } return normalized } Latency: ~15-20ms per embedding generation call. Vector Search with ANN For efficient similarity search at scale, we use an approximate nearest neighbor (ANN) index: type VectorDB struct { index HNSWIndex // Hierarchical Navigable Small World graph metadata map[string]Metadata } func NewVectorDB(dimension int) *VectorDB { return &VectorDB{ index: NewHNSWIndex(HNSWConfig{ Dimension: dimension, M: 16, // connections per node EfConstruction: 200, // search quality during construction Metric: "cosine", }), metadata: make(map[string]Metadata), } } func (vdb *VectorDB) Search(req VectorSearchRequest) []Candidate { // ANN search returns approximate nearest neighbors neighbors := vdb.index.Search(req.Embedding, req.Limit*2) var candidates []Candidate for _, neighbor := range neighbors { // Filter by tags and threshold meta := vdb.metadata[neighbor.ID] if meta.Tags == req.Tags && neighbor.Similarity >= req.Threshold { candidates = append(candidates, Candidate{ Key: neighbor.ID, Similarity: neighbor.Similarity, ContextHash: meta.Tags, }) } if len(candidates) >= req.Limit { break } } return candidates } Latency: ~10-15ms for search across millions of vectors. Production Results After deploying the multi-layer semantic cache to production (serving 10M+ queries/month), we observed: Hit Rates by Layer Layer Hit Rate Avg Latency Saved Planner Cache 44% 380ms Summarization Cache 35% 950ms End-to-End Cache 18% 1,850ms Aggregate Impact Cost Reduction : 48% reduction Latency Improvement : P95 latency from 3.2s to 1.9s (41% reduction) Freshness : Zero incidents of stale responses since implementing two-stage gating Layer Contribution to Savings The planner cache (Layer 1) contributes disproportionately to total savings: Planner cache: ~56% of cost savings Summarization cache: ~30% of cost savings End-to-end cache: ~10% of cost savings This validates the strategy of caching deterministic intermediate computations rather than focusing solely on final outputs. Design Considerations and Trade-offs Similarity Threshold Selection We experimented with thresholds from 0.85 to 0.99: Threshold < 0.90 : Unacceptable false positive rate. Example: "weather in Seattle" matched "weather in San Francisco" Threshold 0.90-0.95 : Better hit rate but occasional semantic mismatches Threshold ≥ 0.98 : Near-exact matching, very low false positive rate We chose 0.98 as the default. This conservative approach sacrifices some hit rate for quality guarantees. TTL Strategy Cache TTLs vary by layer: Planner cache: 7 days (tool definitions change infrequently) Summarization cache: 2 days (more conservative due to potential freshness issues) End-to-end cache: 1 days Shorter TTLs provide additional freshness guarantees at the cost of reduced hit rates. Cache Context Granularity Including too many parameters in cache context reduces hit rate. Including too few causes incorrect cache hits after configuration changes. Our context includes: Model name and version Tool definitions and versions (sorted for consistency) User locale Embedding model version Temperature and top-p parameters We exclude: Request timestamp User ID Session ID Embedding Model Selection We evaluated several embedding models: BERT-base (768-dim): Baseline performance Sentence-BERT (768-dim): Similar performance to BERT-base MiniLM (384-dim): Faster but slightly lower quality Larger models (1024-dim+): Marginal improvement, significant latency cost We selected a 768-dimensional model as the best performance/latency trade-off. Operational Considerations Monitoring Critical metrics tracked in production: type CacheMetrics struct { HitRate float64 // by layer MissRate float64 // by layer LatencySaved time.Duration CostSaved float64 FalsePositiveRate float64 // classifier accuracy CacheSize int64 EvictionRate float64 } We alert on: Hit rate drops >10% week-over-week False positive rate >5% for evergreen classifier P99 cache lookup latency >100ms Cache Invalidation Beyond TTL-based expiry, we implement targeted invalidation: func (sc *SemanticCache) InvalidateByContext(ctx CacheContext) { contextHash := hashContext(ctx) // Find all cache entries with this context keys := sc.vectorDB.GetKeysByTag(contextHash) // Delete from both vector DB and KV store for _, key := range keys { sc.vectorDB.Delete(key) sc.kvStore.Delete(key) } } This allows immediate invalidation when tool definitions or models are updated, rather than waiting for TTL expiry. Storage Requirements Approximate storage per 1M cached queries: Vector DB: ~3GB (768-dim floats + metadata) KV Store: ~5GB (compressed response payloads) Total: ~8GB per 1M cache entries With 7-day TTLs and 10M queries/month, steady-state storage is approximately 50-60GB. Conclusion Multi-layer semantic caching is a critical component for production LLM systems operating at scale. By caching at multiple granularities—particularly at the agent planning layer—we achieved significant cost reduction (48%) and latency improvement (41%) while maintaining response quality and freshness. The key architectural insight is that deterministic intermediate computations (agent planning, tool selection) are more cacheable than final outputs, which depend on fresh data. This inverts the typical caching strategy and provides better hit rates where they matter most. For teams building agentic systems, the planner cache should be the first caching layer implemented. It provides the highest return on investment and requires no freshness validation. References Previous article: Building a Production-Ready Agentic Search Framework