Architecture Hyperparameters › Architecture Design Principles
Language AI HandbookAll chapters Architecture Hyperparameters › Architecture Design PrinciplesOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We've now covered the individual hyperparameters: depth, width, heads, FFN dimensions, and their variants. But knowing what each parameter does is different from knowing how to choose values. Architecture design is both an art and a science, combining theoretical understanding with empirical intuition and practical constraints.
The field has converged on patterns that work. Successful open models like LLaMA and Mistral, along with Qwen, share similar proportions, and these patterns provide reliable starting points for new designs. The following principles distill this collective wisdom into actionable guidance.
Principle 1: Start with Proven Ratios
Rather than searching the vast space of possible architectures, begin with ratios that have proven successful across many models and tasks:
- for standard FFN, or for SwiGLU
- , which determines
- Depth/width ratio: to (very rough guideline)
def suggest_architecture(target_params_billions, ffn_type="swiglu", V=32000):
"""
Suggest architecture hyperparameters for a target parameter count.
This uses heuristics based on successful model designs.
"""
target_params = target_params_billions * 1e9
# Rough relationship: params ≈ 12 * L * d_model^2 (ignoring embeddings)
# Also, L ≈ d_model / 100 for balanced designs
# So: params ≈ 12 * (d_model/100) * d_model^2 = 0.12 * d_model^3
# Therefore: d_model ≈ (params / 0.12)^(1/3)
d_model_estimate = (target_params / 0.12) ** (1 / 3)
# Round to nice multiple of 128
d_model = int(round(d_model_estimate / 128) * 128)
d_model = max(512, min(d_model, 16384)) # Reasonable bounds
# Set depth based on width
L = max(6, d_model // 80) # Slightly deeper than d_model/100
# Head dimension of 128, compute number of heads
d_head = 128
n_heads = d_model // d_head
# FFN dimension
if ffn_type == "swiglu":
d_ff_raw = int(8 / 3 * d_model)
d_ff = int(round(d_ff_raw / 256) * 256) # Multiple of 256
else:
d_ff = 4 * d_model
# Compute actual parameter count
actual = full_parameter_count(
V=V, L=L, d_model=d_model, n_heads=n_heads, d_ff=d_ff, ffn_type=ffn_type
)
return {
"target_B": target_params_billions,
"actual_B": actual["total"] / 1e9,
"L": L,
"d_model": d_model,
"n_heads": n_heads,
"d_head": d_head,
"d_ff": d_ff,
"V": V,
}
# Generate suggestions for various scales
target_sizes = [0.125, 0.35, 1.0, 3.0, 7.0, 13.0, 30.0, 70.0]
suggestions = [suggest_architecture(size) for size in target_sizes]Suggested architectures for target parameter counts: Target Actual Layers d_model Heads d_head d_ff ---------------------------------------------------------------------- 0.12B 0.19B 12 1024 8 128 2816 0.35B 0.46B 17 1408 11 128 3840 1.00B 1.31B 25 2048 16 128 5376 3.00B 3.87B 36 2944 23 128 7936 7.00B 8.62B 48 3840 30 128 10240 13.00B 15.96B 59 4736 37 128 12544 30.00B 36.90B 78 6272 49 128 16640 70.00B 86.88B 104 8320 65 128 22272
These suggestions provide reasonable starting points, but optimal architectures depend on hardware constraints, training budget, and intended use case. The suggestions follow the pattern of successful open models, scaling both depth and width together while maintaining consistent per-head dimensions.
Principle 2: Hardware Awareness
The best architecture on paper can be the worst in practice if it doesn't align with hardware constraints. Modern GPUs have specific requirements for efficient matrix operations, and dimensions that seem arbitrary (like 11008 instead of 11000) often reflect careful optimization for these constraints.
Efficient computation requires dimensions that match hardware characteristics. Main constraints include:
- Tensor core alignment: NVIDIA tensor cores work most efficiently with dimensions that are multiples of 8 (for FP16) or 16 (for INT8). Multiples of 64 or 128 are even better.
- Memory alignment: Dimensions that are powers of 2 or multiples of 256 often achieve better memory bandwidth utilization.
- Parallelism: The number of heads should divide evenly across GPUs in tensor-parallel setups. For 8-GPU training, 32 or 64 heads work well.
def check_hardware_alignment(d_model, n_heads, d_ff):
"""Check if dimensions are hardware-friendly."""
checks = {}
# Tensor core alignment (multiples of 8 for FP16)
checks["d_model_tensor_aligned"] = d_model % 8 == 0
checks["d_ff_tensor_aligned"] = d_ff % 8 == 0
# Better alignment (multiples of 64)
checks["d_model_well_aligned"] = d_model % 64 == 0
checks["d_ff_well_aligned"] = d_ff % 64 == 0
# Best alignment (multiples of 128)
checks["d_model_best_aligned"] = d_model % 128 == 0
checks["d_ff_best_aligned"] = d_ff % 128 == 0
# Multi-GPU friendliness (heads divisible by 8)
checks["heads_multi_gpu"] = n_heads % 8 == 0
# Even head division
d_head = d_model // n_heads
checks["even_head_division"] = d_model % n_heads == 0
checks["d_head"] = d_head
return checks
# Check LLaMA-7B configuration
llama_check = check_hardware_alignment(d_model=4096, n_heads=32, d_ff=11008)Hardware alignment check for LLaMA-7B (d_model=4096, n_heads=32, d_ff=11008): d_model_tensor_aligned: ✓ d_ff_tensor_aligned: ✓ d_model_well_aligned: ✓ d_ff_well_aligned: ✓ d_model_best_aligned: ✓ d_ff_best_aligned: ✓ heads_multi_gpu: ✓ even_head_division: ✓ d_head: = 128
LLaMA's configuration is carefully chosen for hardware efficiency. The model dimension and head count align perfectly, and the FFN dimension is chosen to balance parameter equivalence with computational efficiency (11008 is close to the theoretical 10923 but rounds to a more efficient value).
Principle 3: Consider the Full Pipeline
Architecture design doesn't happen in isolation. The optimal model shape depends on how you'll train it, how long your sequences are, and where you'll deploy it. A model optimized for training efficiency might be suboptimal for inference, and vice versa. Consider the full pipeline from training to deployment.
Key interactions to consider:
- Sequence length: Longer sequences increase memory for attention (quadratic in sequence length) and KV cache. GQA becomes more valuable for long-context models.
- Batch size: Larger batches favor wider models that can better utilize parallel computation.
- Training tokens: Scaling laws suggest parameter count and training data should scale together. A 7B model typically needs 1-2 trillion tokens for optimal performance.
- Inference hardware: If deploying to specific hardware, optimize dimensions for that target rather than for training efficiency.