Sparse Models › Summary

Sparse models stand for a basic shift in how we think about neural network scaling. Rather than accepting that more parameters always means more compute, sparse architectures decouple these quantities through conditional computation.

The core concepts we've covered lay the groundwork for understanding mixture of experts:

• Dense models activate all parameters for every input, creating a tight coupling between capacity and cost
• Conditional computation activates different parameter subsets based on the input, breaking this coupling
• Sparse efficiency comes from having many experts (high capacity) while using few per token (low compute)
• Feed-forward networks are natural targets for sparsification due to their per-token independence and large parameter share

The challenges of sparse models, including load imbalance, communication overhead, training instability, and inference complexity, have driven a rich body of research into routing mechanisms and training techniques. The next chapter introduces expert networks in detail, followed by chapters on gating mechanisms, load balancing strategies, and landmark architectures that have made sparse models practical.

Sparse models do not replace dense models entirely. They occupy a new point on the Pareto frontier of capability versus compute, particularly when you need more capacity than you can afford to run densely. Understanding when and how to use sparsity is increasingly important as models continue to scale.