Large-Scale Training and Sparse Models

Covers GPU memory hierarchy, CUDA cores, tensor cores, GPU specifications.

Covers memory breakdown (activations, parameters, gradients, optimizer states), memory estimation, OOM debugging.

Covers DDP algorithm, gradient synchronization, all-reduce operations, DDP scaling.

Covers column parallelism, row parallelism, communication patterns, Megatron-style parallelism.

Covers pipeline stages, micro-batching, pipeline bubbles, pipeline schedules (GPipe, 1F1B).

Covers ZeRO stage 1 (optimizer state partitioning), ZeRO stage 2 (gradient partitioning), ZeRO stage 3 (parameter partitioning), ZeRO memory savings.

Covers FSDP concepts, FSDP vs ZeRO, FSDP sharding strategies, FSDP usage.

Covers checkpointing concept, checkpoint selection, checkpointing overhead, selective checkpointing.

Covers floating point formats, loss scaling, BF16 advantages, mixed precision implementation.

Covers gradient compression, communication overlap, topology-aware communication, NCCL optimization.

Covers checkpoint contents, checkpoint frequency, async checkpointing, fault recovery.

Covers warmup motivation, linear warmup, warmup duration, warmup for large batches.

Covers step decay, exponential decay, inverse square root decay, decay scheduling.

Covers cosine decay formula, cosine with restarts, cosine schedule parameters, cosine vs linear.

Covers batch size effects, learning rate scaling, batch size limits, LAMB optimizer.

Covers weight decay formula, decoupled weight decay, weight decay selection, weight decay interaction with Adam.

Covers accumulation procedure, accumulation steps, accumulation for memory, accumulation correctness.

Covers loss spikes, gradient norm monitoring, stability techniques, training stability debugging.

Covers hyperparameter search, hyperparameter transfer, critical vs robust hyperparameters, default recipes.

Covers dense vs sparse trade-offs, conditional computation motivation, sparse model efficiency, sparse model challenges.

Covers expert architecture, expert as FFN, expert capacity, expert count selection, expert placement in transformer.

Covers router architecture, routing score computation, router training, router learned behavior.

Covers top-1 routing, top-2 routing, k selection trade-offs, routing implementation, combining expert outputs.

Covers expert utilization imbalance, collapse failure mode, load metrics, balanced routing importance.

Covers load balancing loss formulation, loss coefficient tuning, balancing vs task loss, auxiliary loss implementation.

Covers router instability, z-loss formulation, z-loss benefits, z-loss coefficient, combined auxiliary losses.

Covers expert placement strategies, all-to-all communication, communication overhead, expert parallelism implementation.

Covers Switch Transformer design, top-1 routing choice, capacity factor, Switch scaling results.

Covers Mixtral architecture, Mixtral expert design, Mixtral performance, Mixtral efficiency, Mixtral vs dense models.