Sparse Models › The Historical Arc
Language AI HandbookAll chapters Sparse Models › The Historical ArcOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The idea of conditional computation predates the transformer era. Early work in the 1990s explored mixture of experts for various machine learning tasks. However, these early systems used small numbers of experts and faced challenges with training stability and load balancing.
The key enabler for modern sparse models was scale. Small models don't benefit much from sparsity because the routing overhead dominates. But as models grew to billions of parameters, the economics shifted. Google's research on sparse transformers, including the Switch Transformer, demonstrated that sparse models could match dense model quality while using a fraction of the compute.
Recent architectures like Mixtral, which we'll examine later in this part, have brought sparse models into the mainstream. By carefully engineering the routing mechanism and balancing losses, these models reach quality-to-compute ratios that make them practical for both training and deployment.