Expert Networks › What Is an Expert?

Before the implementation details, define what an expert is and its role in the architecture. The term "expert" suggests specialization, and MoE experts often develop distinct functional roles during training. Some experts might specialize in processing mathematical expressions, others in handling conversational text, and still others in domain-specific terminology. This specialization emerges from the routing mechanism rather than being explicitly programmed: gradient descent makes certain experts more effective for particular input types, and the routing mechanism learns to use that structure. No engineer hand-labels which expert should handle which kind of input; the division of labor arises from optimizing the training objective.

An expert is a small neural-network component, typically the size of a standard feed-forward block, rather than a separate model or reasoning system. It receives a token representation, applies a learned non-linear transformation, and returns a representation of the same dimensionality. The expert's output is weighted by the router's confidence score and added to the residual stream, as a standard FFN output would be. To the surrounding transformer, an MoE layer with sparse expert routing resembles a dense FFN layer: a vector goes in and a transformed vector comes out. The difference is that a dense layer uses one shared FFN, whereas an MoE layer uses one of $N$ specialized FFNs.

Experts address a representational as well as a computational problem. A single dense FFN must learn one set of weight matrices that works reasonably well for all possible tokens. That generalist weight set may be adequate for common patterns but suboptimal for rare or specialized ones. An MoE layer can maintain weight sets tuned to different subsets of the token distribution. Each expert can overfit, in a productive sense, to transformations useful for its assigned token types.

An expert is a transformation function that maps an input vector to an output vector within the same space. A token representation therefore emerges transformed but compatible with the rest of the model's architecture:

where:

• $E_i$: the function representing the $i$-th expert
• $\mathbb{R}^d$: the $d$-dimensional real vector space in which token representations live
• $d$: the model's hidden dimension, which determines both input and output size

This dimensionality constraint is not incidental; it is architecturally needed. The expert's output must integrate smoothly with residual connections and subsequent layers in the transformer. If the expert produced vectors of a different size, the residual addition would be impossible. The input and output spaces are identical by design, so the expert slot can substitute for a standard FFN without any other changes to the surrounding block.

For a set of $N$ experts $\{E_1, E_2, \ldots, E_N\}$, each processes its assigned tokens independently. The key insight is that different tokens can be routed to different experts, letting the model to apply specialized processing based on input characteristics. Consider a sentence containing both technical terminology and conversational phrases: the technical terms might be routed to one expert that has learned relevant transformations for such vocabulary, while the conversational phrases flow to a different expert optimized for casual language patterns. This routing happens dynamically for each forward pass, adapting the computation to the specific characteristics of each input.