LLaMA Components › Key Parameters

Michael BrenndoerferUpdated August 2, 20252 min read
Language AI HandbookAll chapters
Part 29 of 67Modern Decoder Models
Chapter 208LLaMA Components › Key Parameters
Reading section 10 of 12 · Key Parameters · 2 minSection 10 of 12 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 208 of 513

Section 10 of 12 · 2 min read

LLaMA Components › Key ParametersOne section · Sections in this chapter10 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When implementing or configuring LLaMA-style models, several parameters significantly impact model behavior and performance:

Model Dimension (d_model)

  • Controls the width of the model and the size of token representations
  • Typical values: 4096 (7B), 5120 (13B), 8192 (70B)
  • Larger values increase capacity but also memory and compute requirements
  • Must be divisible by the number of attention heads

Number of Attention Heads (n_heads)

  • Determines how many parallel attention patterns the model can learn
  • Typical values: 32 (7B), 40 (13B), 64 (70B)
  • More heads allow diverse attention patterns but increase complexity
  • Head dimension (d_model / n_heads) should typically be 64-128 for efficiency

FFN Hidden Dimension (d_ff)

  • Sets the expansion factor in the feed-forward network
  • For SwiGLU, typically ≈2.67×d_model\approx 2.67 \times d\_model (compared to 4×4 \times for standard FFN)
  • Often rounded to be divisible by 256 or 1024 for hardware efficiency
  • Example: LLaMA 7B uses 11008 (vs. naive 10923 from 4096×2.674096 \times 2.67)

RoPE Base (rope_base)

  • Controls the frequency range of position encodings
  • Default: 10000 (inherited from original sinusoidal encodings)
  • Higher values extend effective context length but may reduce precision for nearby positions
  • Some models use 500000 or higher for very long contexts

RMSNorm Epsilon (eps)

  • Small constant for numerical stability in normalization
  • Typical value: 10−610^{-6} or 10−510^{-5}
  • Rarely needs tuning unless encountering numerical issues

Number of Layers (n_layers)

  • Determines model depth
  • Typical values: 32 (7B), 40 (13B), 80 (70B)
  • More layers increase capacity and sequential dependency but slow inference
  • Deeper models require more careful initialization and learning rate tuning