Caching › Key Parameters
Michael BrenndoerferFebruary 11, 20261 min read
Language AI HandbookAll chapters Part 65 of 67Production Systems
Caching › Key ParametersOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the semantic caching system are:
- threshold: The minimum cosine similarity score required for a cache hit. Higher values (0.90-0.95) give precision; lower values (0.75-0.85) give recall. Domain-specific empirical tuning is necessary.
- ttl_seconds: How long a cached entry remains valid before expiration. Set based on how quickly the underlying information changes, not on system load.
- max_size: The maximum number of entries the cache stores before LRU eviction begins. Tuned against the Zipfian hit-rate curve for your workload.
- embedding_model: The model used to encode queries into vectors. Faster models reduce cache lookup latency but may produce lower-quality similarity estimates.