Efficiency Frontiers › Summary

This chapter surveyed the major fronts of efficiency research in language AI:

Efficient architectures reduce the computational cost inherent in the transformer design. Sparse and linear attention address the quadratic scaling of standard attention, either by restricting the attention pattern (local windows, strided patterns) or by approximating softmax with a kernel function. Mixture-of-experts architectures decouple parameter count from compute cost by routing each token to a small subset of specialized sub-networks. Sub-quadratic models like Mamba process sequences in linear time using structured state spaces, trading some expressiveness for dramatically lower complexity.

Hardware co-design aligns algorithms with the realities of GPU memory hierarchies. The key insight is that most transformer operations are memory-bandwidth-bound rather than compute-bound, which is why FlashAttention produces such large speedups despite computing exactly the same result as standard attention. Understanding arithmetic intensity helps predict which operations will benefit from optimization and which are already near the hardware limit.

Inference efficiency makes deployment economically viable. The KV-cache converts quadratic generation into linear generation, but its memory footprint is a new bottleneck. Grouped Query Attention reduces KV-cache by sharing key-value heads. Post-training quantization shrinks model weights to INT4 or INT8, enabling models to run on consumer hardware. Speculative decoding uses a small draft model to propose tokens that a large verifier accepts in parallel, achieving 2-3x throughput gains. Continuous batching and PagedAttention maximize GPU utilization in serving systems.

Training efficiency reduces the cost of building new models. Mixed-precision training cuts memory usage and increases throughput. Gradient checkpointing trades compute for memory, enabling larger batch sizes. Chinchilla scaling laws revealed that optimal training requires matching compute allocation between model size and training tokens. Compiler-based kernel fusion, implemented in torch.compile and XLA, automatically merges small operations to minimize memory round-trips.

These efficiency advances are not independent. An MoE model benefits from FlashAttention in its attention layers, quantization in its weights, and speculative decoding at inference time. The cumulative effect of stacking multiple efficiency techniques is multiplicative: a model that is 2x faster from architecture changes, 2x faster from quantization, and 2x faster from speculative decoding runs 8x faster than the naive baseline. This stacking is how modern systems achieve inference on large models at costs that would have seemed implausible just a few years ago.