Mixture of Experts at Scale (2024) › The Problem

The traditional approach to scaling language models had relied on dense architectures where all parameters were used for every forward pass through the model. This approach, while effective for smaller models, became increasingly inefficient as model sizes grew. The computational cost of dense models scaled quadratically with the number of parameters, meaning that doubling the number of parameters required roughly four times the computational resources. This scaling pattern made it economically and practically difficult to train and deploy models beyond a certain size.

The memory requirements for storing and processing these large dense models also became prohibitive. Every parameter needed to be stored in memory, and every parameter needed to be accessed and computed during each forward pass. For models with hundreds of billions of parameters, this meant that even storing the model weights required substantial hardware resources. When combined with the computational requirements for training and inference, dense models were becoming too expensive for all but the largest organizations with massive computational resources.

Dense models also suffered from diminishing returns as they grew larger. Each additional parameter provided less improvement in performance than the previous ones, meaning that the cost of scaling up dense models was increasing faster than the benefits. Researchers found that beyond a certain point, simply adding more parameters to dense models no longer provided meaningful improvements in capability or performance.

The fundamental limitation was that dense architectures treated all inputs equally, using the full model capacity for every computation regardless of whether that capacity was needed. A simple input that required only basic language understanding would activate all parameters, just like a complex input that required sophisticated reasoning. This uniform activation pattern was wasteful, consuming computational resources for tasks that didn't need them while simultaneously limiting the total model size that could be practically deployed.

Additionally, dense models struggled with tasks that required specialized knowledge across different domains. A model trained on general text data might perform well on common tasks but struggle with specialized domains like scientific literature, legal documents, or technical code. Adding specialized capabilities typically required training entirely new models or fine-tuning, both of which were computationally expensive and inefficient.