Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLaMA Architecture › Training Efficiency

Michael BrenndoerferUpdated August 8, 20257 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 207LLaMA Architecture › Training Efficiency

Reading section 6 of 11 · Training Efficiency · 7 minSection 6 of 11 · 7 min

Section 6 of 11 · 7 min read

LLaMA Architecture › Training EfficiencyOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

LLaMA's training required substantial compute, but the team optimized for efficiency at every level. These optimizations affect training cost as well as the energy and environmental impact of large model development.

Hardware and ParallelismLink Copied

LLaMA models were trained on NVIDIA A100 GPUs using a combination of parallelism strategies:

Data parallelism: Distributing batches across GPUs
Tensor parallelism: Splitting individual operations across GPUs
Pipeline parallelism: Distributing layers across GPUs

In[17]:

Code

# Training compute for different LLaMA sizes
training_stats = {
    "7B": {
        "gpu_hours": 82432,
        "gpus": 2048,  # A100s
        "wall_clock_days": 1.7,
        "tokens_per_second": 4.4e6,
    },
    "13B": {
        "gpu_hours": 135168,
        "gpus": 2048,
        "wall_clock_days": 2.8,
        "tokens_per_second": 2.7e6,
    },
    "33B": {
        "gpu_hours": 530432,
        "gpus": 2048,
        "wall_clock_days": 10.8,
        "tokens_per_second": 1.1e6,
    },
    "65B": {
        "gpu_hours": 1022362,
        "gpus": 2048,
        "wall_clock_days": 21.0,
        "tokens_per_second": 0.57e6,
    },
}

Out[18]:

Console

LLaMA Training Compute Requirements
===========================================================================
Model         GPU-Hours   Wall Clock      Tokens/sec   Efficiency
---------------------------------------------------------------------------
7B               82,432        1.7 d         4.40 M/s       4718/GPU-s
13B             135,168        2.8 d         2.70 M/s       2877/GPU-s
33B             530,432       10.8 d         1.10 M/s        733/GPU-s
65B           1,022,362       21.0 d         0.57 M/s        380/GPU-s
---------------------------------------------------------------------------
Note: All models trained on 2,048 A100 80GB GPUs

The efficiency numbers reveal an important pattern: larger models process fewer tokens per second but achieve more "learning" per token. The 65B model, being 9x larger than the 7B, requires about 12x the compute. This slightly superlinear scaling comes from memory bandwidth constraints that affect larger models more severely, though fixed overhead costs help moderate the scaling somewhat.

The parallelism strategy deserves some elaboration. Data parallelism, tensor parallelism, and pipeline parallelism address fundamentally different bottlenecks. Data parallelism is the simplest: each GPU receives a different batch and computes gradients independently, which are then averaged. This scales efficiently but requires each GPU to hold the full model in memory. For models exceeding GPU memory capacity, tensor parallelism splits individual matrix multiplications across GPUs, so no single GPU holds a complete model. Pipeline parallelism assigns different layers to different GPUs, passing activations between them like a factory assembly line. LLaMA's training used all three strategies simultaneously, a technically demanding setup that required careful load balancing to avoid idle GPUs waiting on others.

The infrastructure required for this kind of distributed training was itself a significant contribution. The LLaMA paper's training code and configuration details provided a practical reference for others attempting large-scale pretraining, rather than merely an existence proof that it could be done.

Out[19]:

Visualization

Log-log scatter plot of GPU-hours versus parameter count for LLaMA models, showing a linear trend with fitted line showing slightly superlinear scaling. — Training compute scales slightly superlinearly with model size. On a log-log plot, the relationship is nearly linear with a slope of approximately 1.1, meaning compute requirements grow somewhat faster than parameter count.

The slope of approximately 1.1 on the log-log plot means that doubling parameters increases compute by about $2^{1.1} \approx 2.15\times$ , not $2\times$ . This slight superlinearity comes from memory bandwidth constraints that affect larger models more severely.

Memory OptimizationsLink Copied

Training models with billions of parameters requires careful memory management. LLaMA employed several standard techniques:

Gradient checkpointing: Instead of storing all intermediate activations for backpropagation, recompute them during the backward pass. This trades compute for memory, reducing memory requirements by a factor proportional to the square root of the number of layers.
Mixed precision training: Using FP16 or BF16 for most operations while maintaining FP32 master weights. This halves memory requirements for activations and gradients while maintaining numerical stability where it matters.
Efficient attention: Using FlashAttention or similar optimizations to reduce memory from $O(n^2)$ to $O(n)$ in sequence length, necessary for the 2048-token context length.

Each of these optimizations involves a tradeoff, and understanding the tradeoffs explains why they were all used together. Gradient checkpointing reduces peak memory by a factor of roughly $\sqrt{L}$ (where $L$ is the number of layers) at the cost of approximately 33% additional compute, since activations must be recomputed during the backward pass. For a 65-layer model, this reduces peak activation memory by roughly 8x, which is substantial. Mixed precision training is nearly free in terms of speed on modern hardware, since A100 GPUs execute BF16 operations faster than FP32 while master weights in FP32 maintain the numerical precision needed for optimizer updates. FlashAttention addresses the quadratic memory cost of storing the full attention matrix for sequences of length 2048, an otherwise prohibitive requirement at high batch sizes. Together, the three techniques made training feasible on the available hardware at reasonable cost.

Out[20]:

Visualization

Stacked bar chart showing memory usage reduction: baseline at 100%, mixed precision halves it, gradient checkpointing further reduces, and FlashAttention provides final reduction. — Memory savings from optimization techniques used in LLaMA training. Each technique contributes to fitting large models on available GPU memory.

Carbon FootprintLink Copied

The LLaMA paper included carbon footprint estimates, a practice that has become standard for responsible AI development:

Out[21]:

Console

Estimated Carbon Emissions for LLaMA Training
=======================================================
Model       GPU-Hours   Power (kW)   CO2 (tons)
-------------------------------------------------------
7B             82,432          0.4         14.1
13B           135,168          0.4         23.2
33B           530,432          0.4         91.0
65B         1,022,362          0.4        175.4
-------------------------------------------------------
Total       1,770,394            -        303.8

Note: Estimates assume US average grid carbon intensity
Actual emissions depend on datacenter location and energy sources

These emissions, while substantial, are lower than many larger models. The LLaMA 65B produces roughly equivalent capability to GPT-3 175B while requiring less training compute, translating to lower environmental impact.