Gating Networks › Router Learned Behavior

What patterns emerge in trained routers? Understanding these patterns gives insight into how MoE models organize their computation and what specializations develop naturally during training. MoE routers learn real specialization, though patterns depend on the architecture, training data, and task.

Studies of trained MoE language models show that experts often specialize by semantic domain. One expert might preferentially handle scientific text, another legal documents, another conversational content. This specialization emerges naturally from end-to-end training without any explicit labels about document types. The reason is that experts that specialize can model domain-specific patterns more precisely: an expert focused on legal language can learn the specific vocabulary, sentence structures, and logical patterns common in legal text.

To analyze this semantic specialization, we examine routing decisions on held-out data through a systematic process:

1. Route many examples through the trained model
2. Record which experts are selected for each token
3. Aggregate by document type or topic
4. Look for experts with concentrated domain preferences

These analyses reveal that domain specialization often develops in deeper layers, while earlier layers show more syntactic patterns. This makes sense from an information processing perspective: earlier layers establish basic linguistic structure, while later layers can afford to specialize on content-specific patterns.

Early MoE layers often route based on syntactic features rather than semantic content. Before the model can specialize on meaning, it must establish grammatical structure. Patterns observed in early layers include:

• Part-of-speech clustering: Experts that preferentially handle verbs, nouns, or punctuation
• Position effects: Some experts specialize in sentence-initial or sentence-final tokens
• Subword patterns: Different experts for complete words versus subword fragments

These syntactic patterns make sense given the information processing hierarchy: early layers need to handle positional and structural processing that applies broadly across domains. A verb requires similar processing whether it appears in a legal document or a casual conversation.

High-frequency tokens (function words, common subwords) often route to a small subset of experts, while rare tokens distribute more broadly across the expert pool. This asymmetry likely shows that common tokens have simpler, more predictable patterns that a few specialized experts can handle efficiently. Function words like "the," "is," and "of" behave consistently across contexts, so dedicating experts to them makes sense.

Rare tokens, on the other hand, appear in diverse contexts and may benefit from accessing different experts depending on their specific usage. A rare technical term might need different processing when appearing in a definition versus an application.

Routing patterns differ across layers in the model, showing a progression from general to specific processing. A common pattern observed in trained models:

• Early layers: More uniform routing, syntactic specialization
• Middle layers: Emerging domain specialization, more concentrated routing
• Later layers: Strong specialization, decisive routing (lower entropy in routing distribution)

This progression mirrors the general finding that transformer representations become more abstract and task-specific in deeper layers. Early layers establish shared representations useful for all types of content, while later layers can afford to specialize because they build on this shared foundation.