Cost Management › Cost Optimization
Language AI HandbookAll chapters Cost Management › Cost OptimizationOne section · Sections in this chapter3 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Cost optimization for LLM deployments draws on two categories of techniques: those that reduce the compute required per token (efficiency techniques) and those that avoid computation entirely (caching and routing techniques). The inference optimization chapters earlier in this handbook covered many of the underlying mechanisms; this section focuses on how to select and combine them for maximum cost impact.
Model Selection and Right-Sizing
The most impactful cost decision is model selection. Larger models are not always better for a given task, and the cost difference between model sizes can be enormous. A 7B parameter model costs roughly one-tenth as much to run as a 70B parameter model, while often achieving comparable quality for many practical tasks.
The right approach is to establish a quality baseline for your specific task, then find the smallest model that meets it. This process involves:
-
Define quality metrics: Establish task-specific evaluation criteria. For a customer support application, this might be response accuracy on a held-out test set of real queries. For code generation, it might be pass rate on a unit test suite.
-
Benchmark across model sizes: Test a range of models from small (3-7B) to large (70B+), including quantized variants of larger models. Quantization, covered in the inference optimization part of this handbook, can allow a 4-bit quantized 70B model to run on the same hardware as a 16-bit 13B model while achieving better quality.
-
Include latency in the trade-off: A smaller model that answers in 200ms may be preferable to a larger one that answers in 800ms even if the larger model is marginally more accurate, because users tolerate latency poorly.
-
Consider task-specific fine-tuning: A fine-tuned 7B model often outperforms a general-purpose 70B model on a specific narrow task. Fine-tuning adds upfront cost but can dramatically reduce ongoing serving cost.
Quantization for Cost Reduction
Quantization reduces model weight precision from 16-bit floating point to lower-precision formats (INT8, INT4), shrinking memory footprint and enabling higher throughput. The inference optimization chapters provide detailed coverage of quantization mechanics; here we focus on the cost impact.
The cost impact of quantization comes through two channels:
Increased concurrency: Smaller model weights leave more GPU memory for KV caches, allowing more simultaneous requests. If quantization reduces model size by 50%, the memory saved can potentially double the number of concurrent requests, roughly halving the cost per request.
Higher throughput on memory-bandwidth-bound workloads: The decode phase is memory-bandwidth-bound, meaning speed depends on how fast weights can be read from memory. INT4 weights are read four times faster than FP16 weights, directly translating to higher token throughput.
The trade-off is quality degradation. INT8 quantization typically preserves quality well, with less than 1% performance degradation on most benchmarks. INT4 quantization introduces more noticeable degradation, particularly for tasks requiring precise numerical reasoning or context-sensitive language understanding. Methods like GPTQ and AWQ, covered in the inference optimization chapters, minimize quality loss through calibration-aware quantization.
Batching and Request Scheduling
Continuous batching, covered in depth in the inference optimization part of this handbook, is one of the highest-impact cost optimizations available. By keeping the GPU occupied with multiple requests simultaneously, it dramatically improves utilization and reduces cost per token.
The key insight is that batching amortizes the cost of reading model weights across multiple requests. During a single forward pass, the GPU reads each weight matrix once but applies it to all requests in the batch. A batch of 16 requests reads weights once and processes 16 requests, while 16 sequential requests each read the full weights, performing 16 times more memory reads for the same work.
Effective batching requires careful configuration:
- Maximum batch size: Larger batches improve throughput but increase individual request latency (requests must wait to form a batch). The right setting depends on your latency budget.
- Batch timeout: How long to wait for a full batch before processing a partial one. Shorter timeouts improve latency at the cost of throughput efficiency.
- Token budget per batch: Total tokens (prompt + generated) that can be in flight simultaneously. This determines KV cache memory requirements.
Caching Strategies
Caching is among the most powerful cost optimization techniques because it eliminates computation entirely for repeated content.
Prompt prefix caching exploits the fact that many requests share a common prefix: a system prompt, few-shot examples, or a document being analyzed. If the KV cache computed for the prefix is stored and reused across requests, the prefill cost for that shared prefix is paid only once. For applications where a long system prompt is sent with every request, prefix caching can eliminate 30-80% of prefill compute.
Semantic caching goes further by caching complete responses and retrieving them for semantically similar queries. Rather than exact string matching, a small embedding model encodes each request and the cache is searched by cosine similarity. When a new request is within a similarity threshold of a cached request, the cached response is returned without any model inference.
Semantic caching is most effective for FAQ-style applications where users ask variations of the same questions. It is less effective for creative or open-ended tasks where similar questions warrant different responses. The similarity threshold requires careful calibration: too low and you return wrong answers for similar-but-distinct questions; too high and cache hit rates drop to negligible levels.
Output caching stores complete responses for exact or near-exact duplicate requests. This is appropriate for batch processing pipelines where the same document or query may appear multiple times, or for read-heavy knowledge base applications.
# Model the cost savings from caching
def compute_caching_savings(
daily_requests,
avg_input_tokens,
avg_output_tokens,
cost_per_input_1k,
cost_per_output_1k,
prefix_length_tokens,
prefix_cache_hit_rate,
semantic_cache_hit_rate,
):
"""Calculate cost savings from prefix and semantic caching."""
# Baseline cost without any caching
baseline_cost = daily_requests * (
(avg_input_tokens / 1000) * cost_per_input_1k
+ (avg_output_tokens / 1000) * cost_per_output_1k
)
# With semantic caching: cached requests cost only lookup, not inference
# Uncached requests have normal cost; cached requests save both input and output cost
semantic_uncached_fraction = 1 - semantic_cache_hit_rate
semantic_cost = daily_requests * (
semantic_uncached_fraction
* (
(avg_input_tokens / 1000) * cost_per_input_1k
+ (avg_output_tokens / 1000) * cost_per_output_1k
)
# Cached requests have negligible compute cost (just embedding lookup)
)
# With prefix caching: prefix portion of input is only processed once
# Non-prefix portion still processed for every request
non_prefix_input = avg_input_tokens - prefix_length_tokens
prefix_cache_cost = daily_requests * (
# Non-prefix input always costs full price
(non_prefix_input / 1000) * cost_per_input_1k
+
# Prefix is amortized (only paid for cache misses, once per unique prefix session)
(prefix_length_tokens / 1000)
* cost_per_input_1k
* (1 - prefix_cache_hit_rate)
+
# Output always costs full price
(avg_output_tokens / 1000) * cost_per_output_1k
)
return {
"baseline": baseline_cost,
"semantic_caching": semantic_cost,
"prefix_cache_cost": prefix_cache_cost,
"semantic_savings_pct": (1 - semantic_cost / baseline_cost) * 100,
"prefix_savings_pct": (1 - prefix_cache_cost / baseline_cost) * 100,
}
results = compute_caching_savings(
daily_requests=150_000,
avg_input_tokens=512,
avg_output_tokens=256,
cost_per_input_1k=0.0015,
cost_per_output_1k=0.005,
prefix_length_tokens=300, # 300-token system prompt
prefix_cache_hit_rate=0.90, # 90% of requests reuse prefix
semantic_cache_hit_rate=0.25, # 25% of requests are semantically cached
)Daily Cost Comparison (150K requests): Baseline (no caching): $307.20 Semantic caching (25% HR): $230.40 (saves 25.0%) Prefix caching (90% HR): $246.45 (saves 19.8%)
The savings from caching compound with other optimizations. A deployment with both semantic caching and prefix caching, combined with quantization and efficient batching, can achieve costs that are a fraction of a naive deployment.
Speculative Decoding
Speculative decoding, covered in the inference optimization chapters, uses a small draft model to propose token sequences that the target model verifies in parallel. When the draft model's predictions are accepted, multiple tokens are generated in a single forward pass of the target model, increasing effective throughput without degrading output quality.
The cost benefit of speculative decoding is conditional on the acceptance rate of draft tokens. Under the illustrative model below, which charges each drafted token 5% of a target-model forward pass, an 80% acceptance rate with drafts of length 4 yields about 2.80 times the baseline throughput. Exact gains depend on the draft model, hardware, and serving runtime.
Speculative decoding works best for tasks where output is predictable and the draft model can anticipate the target model's choices. It performs well on code generation and structured output tasks but provides smaller benefits for highly creative or open-ended generation where the target model's choices are less predictable.
Hardware Selection and Cloud Strategy
Infrastructure cost includes GPU type and the full spectrum of purchasing decisions.
GPU selection should be matched to model requirements. Memory-intensive operations during the prefill phase benefit from GPUs with high HBM bandwidth, while the memory-bandwidth-bound decode phase benefits from GPUs with high memory bandwidth relative to VRAM. Newer GPU generations (H100 vs. A100) offer significantly better price-performance for inference due to improved memory bandwidth and support for FP8 computation.
Spot instances offer 60-80% cost reductions for workloads that can tolerate interruption. This is practical for batch processing pipelines where jobs can be checkpointed and resumed, but requires careful architecture for latency-sensitive serving because a spot interruption would take an instance offline mid-request.
Reserved instances provide 40-70% discounts in exchange for one- or three-year commitments. A common strategy is to reserve capacity for baseline load and use on-demand instances for peak traffic, a pattern sometimes called "base plus burst."
Multi-cloud and spot diversification reduces exposure to any single provider's spot availability. Running across multiple availability zones or providers increases the likelihood that at least some spot capacity is available during high-demand periods.