Inference Scaling › Deployment Cost Modeling
Language AI HandbookAll chapters Inference Scaling › Deployment Cost ModelingOne section · Sections in this chapter6 / 11 ↓
Real deployment decisions involve more than FLOP counting. Hardware costs, memory constraints, latency requirements, and energy consumption all factor into the total cost of ownership. Moving from theoretical compute analysis to practical cost modeling requires understanding how FLOPs translate to dollars and how hardware constraints shape achievable throughput.
Components of Deployment Cost
A complete cost model includes:
- Hardware amortization: The cost of training and inference hardware amortized over useful life
- Compute costs: Electricity and cooling, proportional to FLOPs executed
- Memory costs: Larger models require more expensive hardware configurations
- Latency penalties: Slower inference may reduce user engagement or transaction value
- Opportunity costs: Resources committed to one model can't serve others
The Cost-Per-Token Model
For a concrete cost analysis, we model the cost to generate one inference token. This simple economic model divides fixed hourly costs by throughput: fewer tokens per hour means each token must bear a larger share of the infrastructure cost:
This formula reveals the basic economic structure of inference: you're paying for GPU time, and the question is how many tokens you can squeeze out of each hour of that expensive hardware. Higher throughput means lower per-token costs, which is why inference optimization focuses so heavily on maximizing tokens per second.
Tokens per hour depends on model throughput, which is constrained by either compute or memory bandwidth:
where:
- : tokens generated per second
- : the GPU's floating-point operations per second
- : FLOPs required per token (two operations per parameter for the forward pass)
- : bytes per second the GPU can read from memory
- : memory footprint per parameter (e.g., 2 bytes for FP16)
The minimum captures a basic hardware constraint: generation speed is limited by whichever resource (compute or memory bandwidth) is exhausted first. This bottleneck analysis is important for understanding real-world inference performance.
The first term () is the theoretical maximum if compute were the only constraint, dividing total available operations by operations needed per token. This would be the limit if weights could be loaded from memory instantaneously.
The second term captures the memory bottleneck: each token generation requires loading all model weights from memory, so throughput cannot exceed memory bandwidth divided by model size. The autoregressive generation process must stream the entire weight matrix through the GPU's memory bus for every single token produced.
For large models, memory bandwidth typically dominates, making smaller models even more advantageous than pure FLOP analysis suggests. Modern GPUs have tremendous compute capacity but relatively limited memory bandwidth. An A100 can perform 312 trillion floating-point operations per second but can only move 2 trillion bytes per second from memory. For large models, the weights cannot be loaded fast enough to keep the compute units busy.
Implementing a Cost Calculator
The following code implements a deployment cost model incorporating these factors:
from dataclasses import dataclass
from typing import Optional
@dataclass
class HardwareSpec:
"""Specifications for inference hardware."""
name: str
flops: float # FP16 FLOPS
memory_bandwidth: float # bytes/second
cost_per_hour: float # USD
power_watts: float # for energy cost calculation
@dataclass
class ModelSpec:
"""Model specifications for cost calculation."""
name: str
parameters: float # number of parameters
bytes_per_param: float = 2.0 # FP16 = 2 bytes
@property
def memory_bytes(self) -> float:
return self.parameters * self.bytes_per_param
@dataclass
class DeploymentScenario:
"""Expected deployment characteristics."""
tokens_per_request: float
requests_per_month: float
target_latency_ms: Optional[float] = Nonedef calculate_throughput(model: ModelSpec, hardware: HardwareSpec) -> float:
"""Calculate max tokens per second (batch size 1)."""
# Compute-bound throughput
flops_per_token = 2 * model.parameters
compute_throughput = hardware.flops / flops_per_token
# Memory-bound throughput
bytes_per_token = model.memory_bytes # Load all weights per token
memory_throughput = hardware.memory_bandwidth / bytes_per_token
# Actual throughput is minimum of both
return min(compute_throughput, memory_throughput)
def calculate_cost_per_million_tokens(
model: ModelSpec, hardware: HardwareSpec, electricity_cost_kwh: float = 0.10
) -> dict:
"""Calculate cost breakdown per million output tokens."""
throughput = calculate_throughput(model, hardware)
tokens_per_hour = throughput * 3600
# Hardware cost per million tokens
hardware_cost = (hardware.cost_per_hour / tokens_per_hour) * 1e6
# Energy cost per million tokens
hours_per_million = 1e6 / tokens_per_hour
kwh_per_million = (hardware.power_watts / 1000) * hours_per_million
energy_cost = kwh_per_million * electricity_cost_kwh
return {
"throughput_tokens_sec": throughput,
"hardware_cost_per_M": hardware_cost,
"energy_cost_per_M": energy_cost,
"total_cost_per_M": hardware_cost + energy_cost,
"bottleneck": "memory"
if throughput == hardware.memory_bandwidth / model.memory_bytes
else "compute",
}Inference Cost Analysis (A100-80GB)
======================================================================
Model Throughput Hardware Energy Total Bottleneck
(tok/s) ($/M tok) ($/M tok) ($/M tok)
----------------------------------------------------------------------
7B 143 $ 7.7778 $ 0.0778 $ 7.8556 memory
13B 77 $ 14.4444 $ 0.1444 $ 14.5889 memory
34B 29 $ 37.7778 $ 0.3778 $ 38.1556 memory
70B 14 $ 77.7778 $ 0.7778 $ 78.5556 memoryThe analysis shows that inference costs scale with model size under this simple batch-one, memory-bandwidth-limited estimate. The 7B model reaches roughly 143 tokens per second with a total hardware cost around $7.78 per million tokens, while the 70B model drops to about 14 tokens per second at approximately $77.78 per million tokens. All models in this analysis are memory-bound rather than compute-bound, meaning the limiting factor is how fast weights can be loaded from GPU memory. Production batching and optimized kernels can improve effective throughput, but the relative advantage of smaller models remains. This reinforces the value of smaller, over-trained models for production deployments.
Monthly Deployment Cost Estimation
For capacity planning, we can project monthly costs based on expected request volume:
def estimate_monthly_cost(
model: ModelSpec,
hardware: HardwareSpec,
requests_per_month: float,
tokens_per_request: float,
electricity_cost_kwh: float = 0.10,
) -> dict:
"""Estimate monthly deployment costs."""
costs = calculate_cost_per_million_tokens(
model, hardware, electricity_cost_kwh
)
total_tokens = requests_per_month * tokens_per_request
tokens_in_millions = total_tokens / 1e6
throughput = costs["throughput_tokens_sec"]
hours_needed = total_tokens / (throughput * 3600)
gpus_needed = hours_needed / (24 * 30) # GPUs for continuous service
return {
"monthly_tokens_M": tokens_in_millions,
"monthly_cost": tokens_in_millions * costs["total_cost_per_M"],
"gpu_hours_needed": hours_needed,
"gpus_for_coverage": max(1, int(gpus_needed) + 1),
}Monthly Cost Projection (10M requests, 500 tokens each) ======================================================= Model Tokens (M) Monthly Cost GPUs Needed ------------------------------------------------------- 7B 5,000 $ 39,277.78 14 13B 5,000 $ 72,944.44 26 34B 5,000 $ 190,777.78 66 70B 5,000 $ 392,777.78 136
The cost differential between model sizes is substantial. Under the same batch-one, bandwidth-limited assumptions, 10 million requests generating 5 billion tokens monthly cost approximately $38,900 for a 7B model and $388,900 for a 70B model, a 10× difference. The smaller model also leaves far more capacity headroom for traffic spikes. Choosing a 7B over-trained model instead of a 70B compute-optimal model can reduce the modeled inference cost by 90% while maintaining comparable quality for many use cases.
