Cost Management › Cost Modeling

Understanding what drives LLM serving costs is the foundation of effective cost management. Without a clear cost model, optimization efforts are guesswork and budget forecasting is unreliable. A good cost model makes visible the relationship between workload characteristics, infrastructure choices, and spending.

LLM inference costs are fundamentally driven by three resources: compute, memory, and network. Of these, compute and memory are the primary cost drivers for most deployments.

Compute is consumed during the forward pass of the model. For each token generated, the model must perform a matrix multiplication through every layer. The cost is roughly proportional to the number of parameters in the model and the number of tokens processed. This is why larger models cost more to run and why longer conversations are more expensive than short ones.

Memory is consumed by two distinct components. The first is static: the model weights. A 7 billion parameter model in 16-bit precision requires approximately 14 GB of GPU memory just to store the weights, before any request is processed. This fixed overhead means that memory cost must be amortized across requests. The second component is dynamic: the KV cache. As sequences grow longer, the KV cache grows proportionally, consuming memory that could otherwise accommodate additional concurrent requests.

Network costs include both data transfer fees (egress charges from cloud providers) and the cost of communication in distributed multi-GPU deployments. For most deployments, network costs are secondary to compute and memory, but they become significant for high-volume streaming applications or systems with frequent model updates.

The cost of a single inference request can be decomposed into a prefill component and a decode component. The prefill phase, which processes the input prompt, is compute-bound: it performs dense matrix multiplications across all prompt tokens simultaneously. The decode phase, which generates output tokens one at a time, is memory-bandwidth-bound: it reads the full model weights and KV cache for each generated token. This distinction matters for cost modeling because the two phases have different cost characteristics and respond to different optimization strategies.

The most useful framework for thinking about LLM costs is a token-based model. Rather than reasoning about GPU-hours directly, we can express cost in terms of input tokens and output tokens, since these are the quantities that applications control.

The cost per token depends on the model architecture, the hardware it runs on, and the infrastructure configuration. We can derive this from first principles:

$$C_{\text{per\_token}} = \frac{C_{\text{hourly}}}{T_{\text{hourly}}}$$

where:

• $C_{\text{per\_token}}$: cost per output token in dollars
• $C_{\text{hourly}}$: hourly infrastructure cost in dollars (GPU instance cost, networking, storage)
• $T_{\text{hourly}}$: tokens generated per hour at the observed throughput

This formula highlights the fundamental lever in cost optimization: to reduce cost per token, you must either reduce $C_{\text{hourly}}$ (cheaper infrastructure) or increase $T_{\text{hourly}}$ (higher throughput). Every optimization technique in the inference stack, from quantization to continuous batching, aims to increase throughput for a given hardware budget.

For a production deployment, we can expand this into a total cost model that accounts for the mixed nature of real workloads:

$$C_{\text{total}} = N_{\text{requests}} \cdot \left( C_{\text{input}} \cdot L_{\text{input}} + C_{\text{output}} \cdot L_{\text{output}} \right)$$

where:

• $N_{\text{requests}}$: total number of requests
• $C_{\text{input}}$: cost per input token
• $C_{\text{output}}$: cost per output token
• $L_{\text{input}}$: average input token length per request
• $L_{\text{output}}$: average output token length per request

Output tokens are typically more expensive than input tokens because they require sequential decode steps while input tokens can be processed in parallel. Commercial API providers often charge two to four times more per output token than per input token, showing this asymmetry.

When running your own infrastructure rather than using API providers, the cost model must account for the full stack of infrastructure costs:

• GPU instances: The dominant cost driver, typically accounting for 70-85% of total infrastructure spend. Cloud GPU pricing varies widely by GPU generation, region, and commitment type.
• Reserved vs. on-demand pricing: Reserved instances typically offer 40-70% discounts over on-demand pricing but require upfront commitment and reduce flexibility.
• Spot/preemptible instances: Can reduce costs by 60-80% but introduce interruption risk that must be managed through redundancy and checkpointing.
• Storage costs: Model weights, logs, and cached outputs. At a few cents per GB-month, storage is usually minor unless you store large volumes of request/response data.
• Networking costs: Egress charges from cloud providers for streaming responses to users. At typical token throughputs, this can become meaningful for high-volume deployments.
• Operational overhead: Engineering time for provisioning, monitoring, and maintaining the serving infrastructure. This is often underestimated and can exceed infrastructure costs for small teams.

Let's build a concrete cost model for a hypothetical production deployment to see how these pieces fit together.

The cost breakdown reveals that output tokens dominate the request cost even though the input is twice as long. This is the key insight from the prefill/decode asymmetry: generating text is more expensive than reading it. Applications that can reduce output length, or that use output caching to serve repeated responses, achieve outsized cost savings.

A critical concept in LLM cost modeling is GPU utilization. Unlike CPU-based systems where utilization correlates directly with cost efficiency, GPU utilization for LLM inference requires careful interpretation.

Compute utilization measures what fraction of the GPU's arithmetic units are active. During the prefill phase, utilization can approach 80-90% as the GPU performs dense matrix multiplications. During the decode phase, utilization often drops to 10-30% because token-by-token generation is memory-bandwidth-bound, not compute-bound. A low compute utilization during decode is not necessarily inefficient; it reflects the fundamental nature of autoregressive generation.

Memory utilization measures GPU memory occupancy. High memory utilization is generally desirable because it means more concurrent requests can be served. However, utilization above 95% risks out-of-memory errors during sudden load spikes. A target of 85-90% memory utilization is typical for production systems.

Effective GPU utilization for cost purposes is better measured as request throughput divided by the theoretical maximum throughput. This captures the revenue-generating work the GPU is doing relative to its capacity.