Scaling Frontiers › Architectural Innovations for Scale

When the standard approach reaches its limits, the response is often to change what you are scaling. The history of deep learning includes several architectural transitions that improved efficiency and opened qualitatively new scaling regimes. Understanding what drove those transitions helps identify where the next ones might come from.

The Mixture of Experts (MoE) architecture offers a way to increase model capacity without proportionally increasing the compute required per forward pass. We covered this in detail in Part K, but it is worth revisiting in the context of scaling limits, because MoE represents a fundamentally different scaling axis.

In a dense transformer, every parameter participates in processing every token. In an MoE model, parameters are divided into groups called experts, and a learned routing function determines which small subset of experts handles each token. A model might have 100 billion total parameters but route each token through only 10 billion, maintaining the computational cost of the smaller model while having the capacity of the larger one.

This creates what researchers sometimes call conditional compute: the model's effective capacity grows with total parameters, but its computational cost per token does not. The relationship between parameters and FLOPs becomes decoupled.

The scaling law under MoE therefore changes. Instead of the standard:

$$C \approx 6ND$$

where $C$ is total compute, $N$ is parameters, and $D$ is training tokens, MoE allows:

$$C \approx 6 \cdot k \cdot N_{\text{active}} \cdot D$$

where $k$ is the fraction of parameters activated per token and $N_{\text{active}}$ is total parameters. For $k = 0.1$, you can train a model ten times larger at the same compute cost, at least in terms of FLOPs.

In practice, MoE introduces new engineering challenges. Expert load balancing, where all experts are utilized roughly equally, requires auxiliary loss terms that add complexity to training. Communication costs increase in distributed training because tokens may need to be routed to experts on different devices, creating network bandwidth pressure. And the relationship between parameters and downstream task performance is less straightforward than for dense models, because different experts develop different specializations.

Despite these challenges, leading frontier models including GPT-4 (reportedly), Gemini 1.5, and Mixtral use MoE architectures. The performance-per-FLOP benefits are large enough that the engineering complexity is considered worthwhile.

One of the most significant recent shifts in the scaling conversation has been the recognition that inference-time compute is its own scalable resource. Training-time scaling dominates much of the literature because it is where the upfront cost is paid, but inference-time computation, the compute used when the model generates each response, can be traded for quality in ways that are only beginning to be understood.

The clearest demonstration came from chain-of-thought reasoning, which we explored in Part P on alignment. By prompting models to generate explicit intermediate reasoning steps before answering, performance on complex reasoning tasks improved dramatically, without any change to model weights. This extra generated text represents additional inference-time compute. The reasoning chain is a form of additional computation the model performs at test time.

More recent work has formalized this into explicit reasoning models. OpenAI's o1 and o3 series and DeepSeek-R1 are trained to perform extended internal reasoning before producing answers. These models allocate variable amounts of inference compute depending on the difficulty of the problem. Easy questions receive short reasoning chains; hard mathematical or coding problems receive much longer ones.

This introduces a qualitatively new scaling law. Performance scales with training compute and with the amount of compute spent at inference:

$$L(C_{\text{train}}, C_{\text{infer}}) = f(C_{\text{train}}) - g(C_{\text{infer}})$$

where $f$ captures the training-time scaling and $g$ captures the inference-time boost. The exact functional forms of $f$ and $g$ are active research topics, but empirically, inference scaling shows power law returns analogous to training scaling, at least for reasoning-intensive tasks.

The practical implication is significant: compute-limited organizations that cannot afford frontier training runs can compensate at inference time. A smaller model that reasons for longer can sometimes match or exceed the performance of a larger model on specific tasks.

Transformers scale well, but they have a quadratic attention problem. The standard self-attention mechanism we covered in Part J has complexity $O(n^2)$ in sequence length $n$, which means that doubling the context window quadruples the attention compute. This puts fundamental pressure on the ability to scale context length, which is increasingly important for tasks that require reasoning over long documents.

Several architectural alternatives have emerged that offer linear or near-linear attention complexity. State Space Models (SSMs), particularly the S4 and Mamba architectures, reformulate sequence modeling as a learned linear dynamical system. Instead of attending to all previous tokens, SSMs maintain a compact state vector that is updated as new tokens arrive. This is analogous to RNNs, which we studied in Part H, but with structured state transition matrices that can be computed efficiently in parallel during training.

The key mathematical idea is that a linear SSM can be written as:

$$h_t = A h_{t-1} + B x_t$$ $$y_t = C h_t$$

where:

• $h_t$: the hidden state at time step $t$, a compact summary of processed history
• $x_t$: the input at step $t$
• $y_t$: the output at step $t$
• $A$: the state transition matrix, which determines how history is preserved or forgotten
• $B$: the input projection matrix
• $C$: the output projection matrix

The critical property is that $A$, $B$, and $C$ are fixed (or slowly varying) matrices, making the recurrence parallelizable via convolutions during training while remaining sequential (and therefore memory-efficient) during inference. Mamba extends this with selective state spaces where $B$, $C$, and a selection mechanism depend on the input, allowing the model to selectively ignore irrelevant context.

These architectures scale differently from transformers. They do not have the same attention bottleneck, so they can handle much longer sequences at lower cost. However, they also do not have the same in-context retrieval ability as full attention: transformers can precisely recall information from anywhere in a long context, while SSMs must compress history into a fixed-size state, which can cause information loss.

Hybrid architectures that combine attention layers with SSM layers represent an active area of research. The intuition is that attention handles precise retrieval when needed, while SSM layers handle bulk sequential processing more efficiently.

Traditional neural network scaling increases both depth (number of layers) and width (hidden dimension) together. But these two dimensions of scale have different properties, and optimizing their ratio for specific objectives is an ongoing area of research.

Wider networks have more capacity per layer but transmit information across many independent channels in parallel. This makes them better at storing many independent patterns and at distributed computation where different parts of the network handle different subtasks simultaneously.

Deeper networks have more sequential computation, which allows them to build up more complex representations step by step. Each layer refines the representation from the previous one, enabling the hierarchical composition of features that makes deep learning powerful.

For language modeling at scale, the optimal depth-to-width ratio is not constant. Early scaling laws research suggested that depth and width should scale roughly proportionally. But more recent work has found that at very large scales, the marginal value of additional depth may exceed that of additional width, because reasoning tasks benefit from more sequential computation steps. This insight influences architecture choices in modern models, which tend to be relatively deeper and narrower than architectures from the early scaling era.

Scaling context length is one of the most practically valuable dimensions for language AI applications. Many real-world tasks, including document analysis, code understanding, multi-turn conversation, and reasoning over databases, require processing far more tokens than early transformer models could handle. GPT-2 had a 1,024-token context window. Modern models like Gemini 1.5 Pro handle 1 million tokens.

This scaling has been achieved through several advances. Position encoding improvements, particularly Rotary Positional Embeddings (RoPE) and its extensions, allow models to generalize to longer sequences than seen during training. Attention approximations like Flash Attention rewrite the attention computation to be more memory-efficient without changing the results, enabling longer sequences at the same hardware budget.

Flash Attention deserves particular attention here because it illustrates an important principle: algorithmic improvement can enable scaling that hardware improvement alone cannot. The standard attention implementation requires materializing the full $n \times n$ attention matrix in GPU memory, which scales quadratically. Flash Attention computes attention in tiles that fit in the faster on-chip SRAM, avoiding the quadratic memory footprint while producing exactly the same output. This makes long-context training feasible on hardware that would otherwise be memory-limited.

The impact of efficient attention on scaling is not merely quantitative. Long context changes what models can do qualitatively. A model that can hold an entire codebase in context can reason about global code structure in ways that a short-context model cannot. A model that can process entire book-length documents can synthesize information across hundreds of pages. These qualitative shifts represent a different kind of scaling than simply adding more parameters.