Mixture-of-Experts at Scale (2021) › The Problem

As language models grew larger throughout 2020 and early 2021, researchers encountered a fundamental scaling problem. Dense neural network architectures, where every parameter is activated for every forward pass, became increasingly inefficient as model sizes approached hundreds of billions of parameters. GPT-3, released in 2020 with 175 billion parameters, required activating all 175 billion parameters even for simple tasks that might only need a fraction of the model's total capacity. This uniform activation pattern meant that computational cost scaled linearly with model size, making it economically difficult to scale models beyond a certain point.

The memory requirements for dense models were also becoming prohibitive. Every parameter needed to be stored in memory, and every forward pass required loading and computing with all parameters. For models with hundreds of billions of parameters, this meant that even storing the model weights required substantial hardware resources. Training these models required expensive GPU clusters with hundreds or thousands of devices, putting them out of reach for most research organizations. The computational and memory constraints were creating a practical ceiling on how large dense models could become.

Dense architectures also suffered from diminishing returns as models grew larger. Researchers found that adding more parameters to dense models provided less improvement per parameter than earlier scaling had demonstrated. Each additional parameter contributed less to model capability, meaning that the cost of scaling up was increasing faster than the benefits. Beyond a certain point, simply adding more parameters to dense models no longer provided meaningful improvements, suggesting that architectural changes might be necessary to continue improving model performance.

Another fundamental limitation was that dense models treated all inputs equally, using the full model capacity for every computation regardless of complexity. A simple question requiring basic factual recall would activate all parameters, just like a complex reasoning task requiring sophisticated analysis. This uniform activation was wasteful, consuming computational resources for tasks that didn't need them while simultaneously limiting the total model size that could be practically deployed. The inability to selectively activate only relevant parameters based on input characteristics represented a fundamental inefficiency in dense architectures.

The training dynamics of very large dense models also presented challenges. As models grew to hundreds of billions of parameters, training became increasingly unstable and expensive. The memory requirements for storing gradients and optimizer states during training scaled with model size, making it difficult to use effective batch sizes or learning rate schedules. These training challenges further limited the practical scalability of dense architectures, creating additional pressure to find alternative approaches that could achieve similar or better performance with more efficient resource usage.