Sparse Models › Sparse Models

The scaling laws we explored in Part XXIII revealed a basic insight: model performance improves predictably with more compute, more data, and more parameters. This created an arms race toward ever-larger models, with GPT-3's 175 billion parameters quickly becoming a stepping stone rather than a ceiling. But this scaling approach faces a harsh economic reality: every parameter must be processed for every token, making the cost of inference grow linearly with model size.

What if we could have the capacity of a large model while only paying the computational cost of a much smaller one? This is the promise of sparse models. They break the assumption that bigger models must be proportionally more expensive to run. Instead of activating all parameters for every input, sparse models selectively engage different subsets of their parameters based on what the input requires.

This chapter introduces the foundational concepts behind sparse computation in neural networks. We'll examine why dense models hit efficiency walls, how conditional computation offers an escape, and what challenges arise when we abandon the all-parameters-all-the-time paradigm.