Efficiency Frontiers › Inference Efficiency

Training a model is a one-time cost. Inference is a continuous cost that scales with usage. For a large language model serving millions of users, inference efficiency directly determines whether a product is economically viable.

Autoregressive generation is inherently sequential: to generate token $t$, the model needs all preceding tokens $1, \ldots, t-1$. Without optimization, this requires recomputing the attention keys and values for all previous tokens at each step, making generation $O(n^2)$ in the total number of generated tokens.

The KV-cache solves this by storing the key and value tensors for all processed tokens and reusing them at each step. Token $t$ only needs to compute its own queries and attend to the cached keys and values:

$$\text{Attention}(q_t, [K_1, \ldots, K_t], [V_1, \ldots, V_t])$$

where $K_1, \ldots, K_{t-1}$ and $V_1, \ldots, V_{t-1}$ are retrieved from cache. This reduces generation from $O(n^2)$ to $O(n)$ per token.

The problem is that the KV-cache is large. For a 7B-parameter model with 32 layers, 32 heads, and 128 head dimension, each token's KV-cache occupies:

$$2 \times 32 \text{ layers} \times 2 \times 32 \text{ heads} \times 128 \text{ head dim} \times 2 \text{ bytes (BF16)} = 524{,}288 \text{ bytes} \approx 512 \text{ KB}$$

For a context of 8,192 tokens, that is 4 GB just for the KV-cache. Serving 100 concurrent users requires 400 GB of GPU memory just for KV-caches, far exceeding what a single GPU holds. Managing KV-cache memory efficiently is one of the central challenges in LLM serving.

Grouped Query Attention (GQA) reduces KV-cache size by sharing key and value heads across multiple query heads. Instead of $H$ separate KV heads for $H$ query heads, GQA uses $G$ KV groups where $G < H$. With $G = 1$, this becomes Multi-Query Attention (MQA), where all query heads share a single KV head. The memory reduction is $H/G$-fold. Llama 3 uses GQA with 8 key-value heads for 32 query heads, reducing KV-cache by 4x with minimal quality loss.

Post-training quantization (PTQ) reduces model weights and/or activations to lower bit-widths without retraining. For inference on edge devices or in memory-constrained serving environments, this can make the difference between a model fitting or not.

GPTQ is an influential PTQ method for weight-only quantization to INT4 or INT8. It uses second-order gradient information (the inverse Hessian of the weight perturbation) to optimally round each weight:

$$\delta W^* = \arg\min_{\delta W} \| X \delta W \|_F^2 \quad \text{subject to } W + \delta W \in \mathcal{Q}$$

where $\mathcal{Q}$ is the set of quantized weight matrices, and $X$ is a calibration input. By solving this layer-by-layer, GPTQ can quantize 175B-parameter models to INT4 in a few hours on a single GPU, with perplexity degradation of 1-3% compared to FP16.

AWQ (Activation-Aware Weight Quantization) makes a key observation: weights that are multiplied by large-magnitude activations are more important and should receive higher precision. By identifying the "salient" weights based on activation statistics and applying a per-channel scaling before quantization, AWQ achieves similar or better results than GPTQ with a simpler algorithm.

GGUF and llama.cpp have made quantized inference accessible on consumer hardware. A 7B model in 4-bit quantization occupies about 4 GB, allowing it to run on a GPU with 8 GB of VRAM or even on CPU with sufficient RAM. The throughput on CPU is low (a few tokens per second) but the accessibility has enabled a large open-source inference ecosystem.

Speculative decoding is a clever approach to accelerating autoregressive generation by exploiting the observation that a small "draft" model can propose tokens that a large "verifier" model can evaluate in parallel.

The algorithm works as follows. A small draft model $M_q$ generates $k$ draft tokens autoregressively:

$$\tilde{x}_{t+1}, \tilde{x}_{t+2}, \ldots, \tilde{x}_{t+k} \sim M_q(\cdot \mid x_1, \ldots, x_t)$$

The large verifier model $M_p$ then evaluates all $k$ draft tokens simultaneously in a single forward pass (since the verifier is not autoregressive at this point, just evaluating given tokens):

$$p_{t+j} = M_p(x_{t+j} \mid x_1, \ldots, x_{t+j-1}) \quad \text{for } j = 1, \ldots, k$$

Each draft token is accepted with probability $\min(1, p_{t+j}(\tilde{x}_{t+j}) / q_{t+j}(\tilde{x}_{t+j}))$, where $q_{t+j}$ is the draft model's probability. This rejection sampling scheme guarantees that the accepted tokens follow exactly the verifier model's distribution, giving no quality degradation.

When many draft tokens are accepted, the effective throughput of the large model increases because one verifier forward pass generates multiple tokens. With a 7B draft model and a 70B verifier, acceptance rates of 70-80% on typical text produce about 2-3x speedup compared to running the 70B model alone.

The speedup depends on two factors: the acceptance rate and the relative latency of draft versus verifier. If the draft model is much smaller than the verifier (say, 10x smaller in parameters), its contribution to total latency is small, and high acceptance rates translate directly to throughput gains.

Traditional serving systems process one request at a time or batch requests of the same length together. Both are wasteful. A single-request system leaves GPU compute idle whenever the model waits for memory. Fixed-length batching requires padding all sequences to the longest one in the batch, wasting compute on padding tokens.

Continuous batching (also called iteration-level batching) processes requests at the granularity of individual decoding steps. At each step, the serving system looks at all in-flight requests, determines which are currently generating a new token, and batches those tokens together for the forward pass. New requests can be inserted and completed requests can be removed at each step, keeping the GPU fully utilized.

PagedAttention addresses the KV-cache fragmentation problem. The KV-cache for each request grows as tokens are generated, but traditional systems pre-allocate a maximum-length block of contiguous memory for each request. This leads to fragmentation: a request that generates 100 tokens wastes the memory reserved for 1900 more (if the max length is 2000). PagedAttention stores KV-cache in fixed-size pages (similar to virtual memory paging in operating systems), allocating pages as needed and maintaining a page table mapping logical token positions to physical memory locations. This reduces memory waste to near zero and allows the system to serve 2-4x more concurrent requests.

vLLM is the most widely used implementation of these ideas. It combines continuous batching with PagedAttention and has become the de facto standard for high-throughput LLM serving.