Gated Linear Units › Hidden Dimension Choices in Practice
Language AI HandbookAll chapters Gated Linear Units › Hidden Dimension Choices in PracticeOne section · Sections in this chapter8 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Different models make different choices about how to handle the parameter trade-off. Let's examine the configurations used by several prominent models:
# Real-world model configurations
model_configs = [
("GPT-2 (standard FFN)", 768, 3072, "standard"),
("BERT-Base (standard FFN)", 768, 3072, "standard"),
("LLaMA-7B (SwiGLU)", 4096, 11008, "swiglu"),
("LLaMA-13B (SwiGLU)", 5120, 13824, "swiglu"),
("Mistral-7B (SwiGLU)", 4096, 14336, "swiglu"),
("PaLM-8B (SwiGLU)", 4096, 16384, "swiglu"),
]Real-world model FFN configurations: Model d_model d_ff Ratio Params ---------------------------------------------------------------------- GPT-2 (standard FFN) 768 3072 4.00x 4.7M BERT-Base (standard FFN) 768 3072 4.00x 4.7M LLaMA-7B (SwiGLU) 4096 11008 2.69x 135.3M LLaMA-13B (SwiGLU) 5120 13824 2.70x 212.3M Mistral-7B (SwiGLU) 4096 14336 3.50x 176.2M PaLM-8B (SwiGLU) 4096 16384 4.00x 201.3M
The table reveals interesting design choices across different models. Standard FFN models like GPT-2 and BERT use a consistent 4x ratio. SwiGLU models show more variation: LLaMA-7B uses 2.69x (close to parameter parity with 4x standard), while Mistral-7B and PaLM-8B use higher ratios, trading additional parameters for greater expressiveness.
Notice the pattern: models using SwiGLU tend to use ratios around 2.7x to 4x, depending on whether they prioritize matching the parameter count of standard FFNs or pushing expressiveness further. Mistral-7B, for example, uses a higher ratio (3.5x), accepting more parameters in exchange for potentially better quality.
