Feed-Forward Networks › Limitations and Impact

The feed-forward network is conceptually simple: two linear layers with a nonlinearity between them. Yet this simplicity masks significant computational cost, and understanding those costs is essential for anyone working with large language models in practice.

The FFN's most significant limitation is its memory footprint. Because FFN weights are static after training, they must reside in memory at inference time. For a model like LLaMA-70B with a hidden dimension of 8192 and an FFN dimension of 28672 (approximately 3.5x), each FFN layer holds about 2 $\times$ 8192 $\times$ 28672 $\approx$ 470 million parameters. Across 80 layers, the FFN alone stores roughly 37 billion parameters, each requiring at least 2 bytes in FP16 format, meaning over 74 GB of memory just for FFN weights. This is why running large models requires high-memory GPUs or specialized hardware.

Weight quantization is the most common approach to this memory problem. By representing weights in lower precision (INT8, INT4, or even INT2), the memory footprint shrinks proportionally. A model quantized to INT4 requires one-quarter the memory of its FP16 counterpart, enabling deployment on consumer hardware. The challenge is maintaining model quality: aggressive quantization can degrade generation quality, particularly for reasoning tasks that require precise numerical computations in the FFN's hidden layers. Research into quantization-aware training and calibration has made significant progress, but the tradeoff between compression and quality remains an active area of work.

Sparsity offers another path forward. The ReLU activation naturally creates sparse hidden representations, as negative pre-activations become zero. Researchers have exploited this by identifying which hidden dimensions will be active for a given input and computing only those, skipping computation for dimensions that would be zeroed anyway. This requires predicting active neurons before the full computation, which introduces overhead, but the net savings can be substantial for highly sparse models. The GELU and SiLU activations that have replaced ReLU in many modern models are smoother alternatives that do not produce exact zeros, trading the sparsity benefit for better gradient flow during training.

The Mixture of Experts (MoE) architecture takes the sparsity idea further. Instead of a single FFN per layer, an MoE layer has multiple "expert" FFNs and a router that directs each token to only a small subset (typically 2 of 8, or 2 of 64). The model scales parameters without proportionally scaling compute, because only a fraction of experts process each token. Models like Mixtral-8x7B and GPT-4 (reportedly) use MoE to achieve parameter counts that would otherwise be computationally prohibitive. The key challenge is load balancing: ensuring that all experts are used approximately equally across the training data, preventing some experts from specializing in everything while others specialize in nothing.

Despite its computational weight, the FFN's role is needed and irreplaceable. It provides the nonlinearity that maps attention's linear weighted averages into expressive function approximation. It is the model's "memory," storing factual associations in its weights that attention retrieves based on context. Its position-wise nature enables massive parallelization that makes transformer training tractable at the scale of billions of parameters and trillions of training tokens.

The interplay between attention and FFN defines transformer expressiveness. Attention routes information between positions, creating context-dependent representations. The FFN transforms those representations position-by-position, adding computational depth and nonlinear capacity. Together, they enable the hierarchical, compositional language understanding that powers modern NLP. Neither component alone would be sufficient: attention without the FFN is a purely linear contextual aggregation; the FFN without attention is a powerful token-level classifier with no awareness of surrounding context.