Build a GPT from Scratch

Start with a small next-token probability table. A rule-based generator exposes every choice and makes the difference between training and inference concrete.

Follow one sentence through tokenization, batching, model output, sampling, and decoding. Each stage gets an interface that later chapters will replace with working code.

Set up the Python environment, tests, configuration, run directories, and random seeds. The same project works on CPU and moves to a GPU when one is available.

Assemble the mock pipeline and account for every value and shape. Save the trace so later implementations can be checked against it.

Attach mathematical notation to executable scalar functions. Work through a small neuron and inspect the floating-point behavior of each operation.

Calculate vector operations by hand, then reproduce them in Python. Dot products, norms, and projections lead directly to token representations and attention scores.

Work from row-by-column multiplication to batched tensors. Track the shapes used by GPT and learn where reshaping or broadcasting can silently change their meaning.

Build conditional probabilities from a tiny corpus and calculate the probability of a complete sentence. Sampling experiments connect the equations to generated text.

Turn a short vector of logits into probabilities and a loss. The NumPy implementation is checked against hand calculations and PyTorch.

Trace a loss backward through a small computation graph. Manual derivatives, finite differences, and autograd must agree before gradient descent begins.

Write a scalar reverse-mode autodiff engine. Tests cover branching graphs, reused values, gradient accumulation, and numerical agreement.

Build an MLP from the custom autodiff values and train it on a plotted two-dimensional dataset. Broken learning rates and missing gradient resets become debugging exercises.

Package autodiff, layers, optimization, and tests into a small library. Train a nonlinear classifier and submit the gradients, loss curve, and decision surface as evidence.

Create a public-domain corpus whose documents have recorded sources, licenses, and checksums. Cleaning, deduplication, rejected records, and train-validation splits remain inspectable.

Inspect English, accented text, Chinese characters, emoji, and code as Unicode and UTF-8 bytes. Then write reversible character and byte tokenizers.

Count byte pairs, apply merges, and train a BPE tokenizer on the book corpus. Each learned token can be traced through its merge history.

Turn encoded documents into shifted input-target windows and deterministic batches. Decode sampled rows to catch boundary, mask, and split errors early.

Run the data pipeline from source documents to encoded shards. Deliver the tokenizer, manifests, leakage checks, and decoded sample batches needed for training.

Count token transitions, smooth the table, score held-out text, and generate samples. Because the table fits on the page, every probability can be inspected.

Replace the count table with learned logits and optimize it with gradients. Direct comparison shows what changed and what remained a bigram model.

Look up dense token vectors and feed several previous tokens into an MLP. Experiments vary the context length and test whether order matters.

Move training out of ad hoc notebook cells. The shared loop handles updates, validation, metrics, checkpoints, and the one-batch overfit test.

Train the count bigram, neural bigram, and fixed-context model on the same data. Compare loss, runtime, parameter count, context, and fixed-seed samples.

Let each token gather information from earlier positions through causal averaging. Loop, cumulative-sum, and matrix versions must return the same values.

Calculate queries, keys, values, attention scores, and weighted sums for three small vectors. The same numbers are then produced by one attention head.

Add the causal mask, scaled dot products, and stable softmax. Intervention tests prove that changing a future token cannot alter an earlier output.

Split channels across heads and put them back together without losing their meaning. A readable reference version checks the vectorized implementation.

Show what attention does with permuted input when it has no positions. Learned position embeddings make order available and introduce a fixed context limit.

Build the per-token MLP and reconcile its dimensions and parameter count. Activation plots and short runs compare ReLU, GELU, and expansion size.

Calculate LayerNorm for one token vector and check the implementation against PyTorch. Deep-stack experiments measure what residual paths and normalization do to gradients.

Combine attention and the MLP in a pre-norm residual block. A shape trace follows one batch through every projection, normalization, and addition.

Stack the decoder blocks between token embeddings and vocabulary logits. Verify the forward contract, initialization, parameter count, and causal behavior.

Write the decoding loop and compare greedy, temperature, top-k, and top-p sampling on fixed logits and prompts. Seeded runs make the comparison repeatable.

Train the complete model on the book corpus. Submit causal tests, shape and parameter ledgers, loss curves, checkpoints, runtime measurements, and fixed-seed samples.

Calculate model parameters, memory use, token count, training time, and cost before choosing a run. Measured throughput checks the estimate on available hardware.

Check expected initial loss, activation variance, gradient flow, and one-batch overfitting before a long run. Bad initializations provide useful counterexamples.

Work through the first AdamW updates by hand, including moment estimates and bias correction. Then add parameter groups, warmup, decay, clipping, and accumulation.

Investigate deliberately broken runs with high and low learning rates, bad masks, shifted targets, corrupt data, and non-finite gradients. Each confirmed bug becomes a regression test.

Measure token loss on fixed held-out data, inspect samples, and test memorization. Small ablations and uncertainty estimates keep the conclusions within the evidence.

Save the full state needed to resume training, including the random streams and data position. A split run must match an uninterrupted run step for step.

Establish a timing baseline, find the bottleneck, and change one thing at a time. Mixed precision, fused attention, compilation, and data transfer are kept only when measurements support them.

Plan and run pretraining within a stated budget. The report records the configuration, artifact hashes, performance, loss curves, checkpoint choice, samples, and limitations.

Store tokens in shards and read datasets larger than memory. Tests cover deterministic shuffling, worker partitioning, prefetching, and resume position.

Calculate gradient averaging for two replicas before running a small multiprocess job. The optional GPU track measures communication cost and scaling efficiency.

Run a small grid of model and token budgets, fit learning curves, and test the fit on a held-out configuration. Noisy pilots set bounds on what can be extrapolated.

Use measured loader speed, training throughput, memory, and pilot loss to choose the largest sensible run. A supported decision to stay small also satisfies the project.

Create conversation records with source and license fields, render the chat template, and inspect the token-level loss mask. Packing and truncation receive explicit tests.

Fine-tune the base checkpoint on assistant responses and compare it with the untouched model. Measure instruction behavior alongside forgetting on base text.

Calculate sequence log probabilities for chosen and rejected responses, then implement a small DPO loss. The chapter also locates reward models and policy optimization within RLHF.

Test the actual small model for memorization, privacy leakage, harmful output, bias, and prompt injection. Use the results to write the model card and release decision.

Instruction-tune the base model and compare both checkpoints on a frozen test set. Deliver the lineage, response table, forgetting results, safety checks, and model card.

Time checkpoint loading, tokenization, prefill, and token-by-token decoding separately. Memory and latency are measured as prompts and outputs grow.

Reuse keys and values from earlier positions instead of recomputing them. Cached and uncached logits must agree before speed and memory are compared.

Quantize a short weight vector by hand, then apply the same ideas to the model. Report storage, runtime memory, speed, and evaluation changes on the target hardware.

Package the tokenizer, weights, and configuration behind a small streaming API. Validate requests and test limits, cancellation, concurrency, and startup failures.

Build a versioned test set with exact checks, pairwise review, prompt variations, and controlled factual questions. Quality results sit beside latency and failure rates.

Serve the selected checkpoint with a verified KV cache and optional quantization. Compare the final service with the original baseline on the same prompts and hardware.

Replace LayerNorm, learned positions, GELU, and standard multi-head attention one component at a time. RMSNorm, RoPE, SwiGLU, and GQA each get a numerical example and unit tests.

Compare architecture changes under a stated matching rule, such as equal parameters, tokens, compute, or time. Repeated runs show how much of the result is noise.

Take one paper equation, map its symbols to tensor shapes, and implement the smallest useful test. A reduced experiment records every assumption the paper leaves open.

Connect the model already built to continued pretraining, retrieval, tools, sparse models, long context, multimodality, and distillation. Each path starts from a specific limitation.

Train a small Llama-style variant beside the original GPT under a matched budget. Compare correctness, loss, training cost, cache memory, decoding speed, and samples.

Start with raw documents and produce the tokenizer, base model, assistant checkpoint, evaluation, inference service, and documentation. Every result must trace back to its data and configuration.