Efficiency Frontiers › Efficiency Frontiers

Modern language models have achieved remarkable capabilities, but they come with a price tag that makes deployment difficult for most organizations. GPT-3 reportedly cost over $4 million to train a single time. Serving a 175B-parameter model requires dozens of high-end GPUs just to handle inference. A smartphone cannot run these models at all. These constraints have pushed researchers to ask a question that turns out to be surprisingly deep: can we get the same capabilities for a fraction of the compute?

The answer, increasingly, is yes. Efficiency research has become one of the most active areas in language AI, and it does not just mean shrinking models. It means redesigning architectures to avoid unnecessary computation, co-designing software and hardware so neither wastes the other's capacity, and rethinking the entire training and inference pipeline from first principles.

This chapter covers four interconnected fronts. First, we examine efficient architectures: changes to the transformer design that reduce the quadratic cost of attention, eliminate redundant computation, and pack more capability into fewer parameters. Second, we look at hardware co-design: how modern AI accelerators work, what their bottlenecks are, and how architectures can be shaped to match the machine. Third, we explore inference efficiency: the techniques that make deployment faster and cheaper once a model is trained, from quantization to speculative decoding. Finally, we look at training efficiency: how recent advances in data quality, mixed-precision arithmetic, and gradient checkpointing have dramatically reduced the cost of training large models from scratch.

As we discussed in the chapter on Model Compression, techniques like knowledge distillation and pruning can reduce a model's size after training. This chapter complements that work by focusing on efficiency baked into the architecture and training process itself. The next chapter, Capability Frontiers, explores whether these efficiency gains are enabling qualitatively new capabilities in small models.