Expert Networks › Expert Placement in Transformer

Not every layer in a transformer needs to be an MoE layer. The placement of expert layers significantly impacts model behavior and efficiency. This section examines the design patterns that have proven effective in practice and explains the reasoning behind the architectural choices that major models have converged on.

The transformer architecture has a regular repeating structure: each layer consists of a self-attention module followed by a feed-forward network, with layer normalization and residual connections wrapping each component. When introducing MoE, the key question is which components to replace with expert versions and in which layers. The answer is not purely a matter of taste: it reflects deep considerations about what each component does and whether sparse routing is compatible with that function.

Think of the choice of where to place MoE layers as deciding which departments in a company should have specialists and which should have generalists. In most companies, the legal and finance departments are highly specialized (only certain people can do that work), while the IT helpdesk and facilities management are shared across the whole company. Similarly, certain layers of a transformer may benefit from specialization while others are better off shared. The empirical pattern that has emerged is that the FFN component of the transformer block is the right place for expert routing, and that alternating between dense and MoE layers strikes a useful balance.

The placement decision also has consequences for the routing mechanism. If MoE layers appear at every depth, the router at each layer must learn to distribute tokens appropriately for that layer's function. Early layers tend to handle lower-level linguistic structure (morphology, basic syntax), while deeper layers handle higher-level semantics and task-specific reasoning. A model where every layer is MoE must learn routing policies that account for this depth-dependent specialization. Alternating patterns allow dense layers to create shared representations that subsequently MoE layers can then specialize from, potentially making the routing problem simpler.

The standard approach replaces some or all FFN layers with MoE layers while keeping attention layers unchanged. This follows naturally from the position-independence property discussed earlier.

In a transformer block, the computation flows through a sequence of operations. The input first passes through layer normalization, then through the attention mechanism where tokens exchange information. After a residual connection adds the input back to the attention output, another layer normalization prepares the representation for the FFN (or MoE) layer. Finally, another residual connection completes the block. The FFN position, coming after attention has already enabled cross-position communication, becomes the natural location for the MoE layer. At this point in the computation, each token has already had the opportunity to incorporate information from its context, and the remaining job is to transform that enriched representation in a token-specific way.

The FFN position becomes the MoE position. Attention layers remain dense. This keeps all positions can still interact before expert-specialized processing. This design principle reflects the distinct roles of attention and FFN layers in transformers: attention handles information routing between positions, while FFN layers turn the representation at each position independently. MoE exploits this independence by running different transformations for different tokens, all without interfering with the cross-position communication that attention handles.

Several patterns have been explored for alternating dense and MoE layers, each which reflects different assumptions about how much expert processing is beneficial relative to the overhead it introduces:

• Every layer MoE: All FFN layers become MoE layers. This maximizes expert utilization but increases routing overhead and may not improve quality proportionally. Training is also more complex, since every layer introduces the routing instability and load-balancing challenges associated with MoE.

• Alternating (every 2nd layer): A common pattern in practice. Every other FFN layer is replaced with MoE. Mixtral uses this approach. Dense layers provide regular checkpoints of shared processing between expert layers, which may help stabilize the representation and prevent expert drift across layers.

• Periodic (every nth layer): More aggressive interleaving, such as MoE at every 4th layer. This reduces router overhead significantly but limits expert specialization opportunities, since tokens pass through fewer expert layers overall.

Some architectures vary the number of experts by layer depth. Early layers might use fewer experts because input patterns are more uniform and syntax-driven, while later layers use more experts where task-specific and domain-specific specialization becomes most valuable. This pattern appears in certain custom architectures, though major publicly released models typically use uniform expert counts for simplicity. Token features become more diverse in deeper layers, so later layers may gain more from fine-grained specialization than early layers.

An emerging technique shares expert weights across multiple layers while maintaining layer-specific routers. This reduces total parameters while preserving adaptive routing. Each layer can route differently even if the underlying expert computations are shared. The key insight is that the specialization in MoE comes partly from routing decisions (which expert gets which token) and partly from the expert's weight matrices (what transformation that expert applies). Cross-layer sharing retains the routing flexibility while reducing the parameter count by reusing expert weights. This is an active area of research, and we may see more production models adopt it as the field develops better understanding of how routing and weight specialization interact.