Language AI Handbook

Michael Brenndoerfer

BY MICHAEL BRENNDOERFER

Language AI Handbook

Start with classical NLP, then work through transformers, LLM training, evaluation, and production

The book spans 20 volumes. Read in order, or browse the contents to find the topic you need.

12,550 pages across 418 chapters

Read online Buy the PDF

Browse contents

5.0 / 5 · 12 reader reviews

“…my concepts have never been clearer. You really are doing an amazing job…”

— Muhammad H., AI/ML Engineer

Read Language AI Handbook free online

The Language AI Handbook connects the pieces of modern language AI. We begin with classical NLP, build up to transformers and LLM training, then cover retrieval, evaluation, safety, and production deployment.

Explore the book’s 20 volumes

Language AI Handbook is organised into 20 volumes, plus an appendix. Each volume covers a subject, divided into parts and chapters. Read from the beginning, or choose the volume you need.

418 published chapters · Free to read online

Start at chapter one

Volume 1Language Data and Classical NLP3 parts · 18 chaptersBuy this volumeBuy PDF · $24

Part I

Text as Data

5 chapters

01Character EncodingCovers ASCII origins and 7-bit limitations, Unicode code points and planes, UTF-8 variable-width encoding scheme, byte order marks and endianness, encoding detection heuristics, common encoding errors and mojibake, practical encoding/decoding in Python.52 min 02Text NormalizationCovers Unicode normalization forms (NFC, NFD, NFKC, NFKD), case folding vs lowercasing, accent and diacritic handling, whitespace normalization, ligature expansion, full-width to half-width conversion, implementing a normalization pipeline.52 min 03Regular ExpressionsCovers regex syntax and metacharacters, character classes and quantifiers, grouping and backreferences, lookahead and lookbehind assertions, greedy vs lazy matching, common NLP patterns (emails, URLs, dates), regex performance considerations.56 min 04Sentence SegmentationCovers period disambiguation challenges, abbreviation handling, rule-based boundary detection, Punkt sentence tokenizer algorithm, evaluation metrics for segmentation, handling edge cases (quotes, parentheses, lists), multilingual segmentation issues.62 min 05Word TokenizationCovers whitespace tokenization limitations, punctuation handling rules, contractions and clitics, language-specific challenges (Chinese, Japanese, German compounds), Penn Treebank tokenization standard, building a rule-based tokenizer, tokenization evaluation.64 min

Part II

Classical Text Representations

9 chapters

06Bag of WordsCovers document-term matrix construction, vocabulary building from corpus, word counting and frequency vectors, sparse matrix representation (CSR/CSC formats), vocabulary pruning (min_df, max_df), binary vs count representations, limitations of word order loss.51 min 07N-gramsCovers bigram and trigram extraction, n-gram vocabulary explosion, n-gram frequency distributions, Zipf's law in n-grams, character n-grams for robustness, skip-grams and flexible windows, n-gram indexing for search.47 min 08N-gram Language ModelsCovers Markov assumption and chain rule, maximum likelihood estimation, probability calculation for sequences, handling unseen n-grams, start and end tokens, generating text from n-gram models, model storage and lookup efficiency.58 min 09Smoothing TechniquesCovers add-one (Laplace) smoothing, add-k smoothing and tuning, Good-Turing smoothing derivation, Kneser-Ney smoothing intuition and formula, interpolation vs backoff, modified Kneser-Ney, comparing smoothing methods empirically.56 min 10PerplexityCovers cross-entropy definition and derivation, perplexity as branching factor, relationship to bits-per-character, held-out evaluation methodology, perplexity vs downstream performance, comparing models with perplexity, perplexity limitations and caveats.63 min 11Term FrequencyCovers raw term frequency, log-scaled term frequency, boolean term frequency, augmented term frequency, L2-normalized frequency vectors, term frequency sparsity patterns, efficient term frequency computation.46 min 12Inverse Document FrequencyCovers document frequency calculation, IDF formula derivation, IDF intuition (rare words matter more), smoothed IDF variants, IDF across corpus splits, relationship to information theory, implementing IDF efficiently.59 min 13TF-IDFCovers TF-IDF formula and variants, TF-IDF vector computation, TF-IDF normalization options, BM25 as TF-IDF extension, document similarity with TF-IDF, TF-IDF for feature extraction, sklearn TfidfVectorizer deep dive.51 min 14BM25Covers BM25 derivation from probabilistic IR, saturation parameter k1, length normalization parameter b, BM25+ and BM25L variants, field-weighted BM25, implementing BM25 scoring, BM25 vs TF-IDF empirically.52 min

Part III

Distributional Semantics

4 chapters

15The Distributional HypothesisCovers Firth's "you shall know a word by the company it keeps," distributional similarity intuition, context window definitions, paradigmatic vs syntagmatic relations, word similarity from distributions, limitations of distributional semantics.46 min 16Co-occurrence MatricesCovers word-word co-occurrence matrices, word-document matrices, context window size effects, weighting by distance, symmetric vs directional contexts, matrix sparsity patterns, efficient construction algorithms.50 min 17Pointwise Mutual InformationCovers PMI formula derivation, PMI interpretation as association, positive PMI (PPMI), shifted PPMI variants, PMI matrix properties, PMI vs raw counts comparison, PMI for collocation extraction.48 min 18Singular Value DecompositionCovers SVD mathematical formulation, truncated SVD for dimensionality reduction, LSA (Latent Semantic Analysis), choosing embedding dimensions, SVD computational complexity, randomized SVD for scale, interpreting SVD dimensions.52 min

Volume 2Embeddings, Tokenization, and Sequence Labeling3 parts · 25 chaptersBuy this volumeBuy PDF · $24

Part IV

Word Embeddings

9 chapters

19Skip-gram ModelCovers skip-gram architecture diagram, input/output representations, softmax over vocabulary, skip-gram objective function, training data generation, window size hyperparameter, skip-gram vs CBOW intuition.41 min 20CBOW ModelCovers CBOW architecture, context word averaging, CBOW objective function, CBOW vs skip-gram training speed, CBOW for frequent words, implementing CBOW forward pass, CBOW gradient derivation.52 min 21Negative SamplingCovers softmax computational bottleneck, negative sampling objective derivation, sampling distribution (unigram^0.75), number of negatives hyperparameter, negative sampling gradient computation, NCE vs negative sampling, implementing efficient sampling.47 min 22Hierarchical SoftmaxCovers binary tree construction (Huffman coding), path probability computation, hierarchical softmax objective, gradient computation along paths, tree structure impact on learning, hierarchical softmax vs negative sampling, when to use each approach.53 min 23Word2Vec TrainingCovers data preprocessing pipeline, subsampling frequent words, learning rate scheduling, minibatch vs online training, convergence monitoring, gensim Word2Vec usage, training from scratch in PyTorch.50 min 24Word AnalogyCovers vector arithmetic for analogies, parallelogram model, analogy evaluation datasets, 3CosAdd vs 3CosMul methods, analogy accuracy metrics, limitations of analogy evaluation, what analogies reveal about embeddings.49 min 25GloVeCovers GloVe objective function derivation, weighted least squares formulation, relationship to matrix factorization, weighting function design, bias terms in GloVe, GloVe vs Word2Vec comparison, training GloVe efficiently.47 min 26FastTextCovers character n-gram representation, word vector as n-gram sum, FastText architecture, handling OOV words, morphological awareness, FastText for morphologically rich languages, training FastText models.51 min 27Embedding EvaluationCovers intrinsic vs extrinsic evaluation, word similarity datasets (SimLex, WordSim), analogy accuracy, embedding visualization (t-SNE, UMAP), downstream task evaluation, embedding bias detection, evaluation pitfalls.54 min

Part V

Subword Tokenization

8 chapters

28The Vocabulary ProblemCovers OOV word problem, vocabulary size explosion, rare word representation, morphological productivity, compound words, code and technical text, the case for subword units.48 min 29Byte Pair EncodingCovers BPE algorithm step-by-step, merge rules learning, vocabulary size control, BPE encoding procedure, BPE decoding procedure, BPE implementation from scratch, BPE hyperparameters.46 min 30WordPieceCovers WordPiece vs BPE differences, likelihood objective for merges, greedy tokenization algorithm, ## prefix notation, WordPiece in BERT, training WordPiece tokenizers, handling unknown characters.48 min 31Unigram Language Model TokenizationCovers unigram LM formulation, EM algorithm for training, Viterbi decoding for tokenization, sampling multiple segmentations, subword regularization, unigram vs BPE comparison, SentencePiece unigram mode.55 min 32SentencePieceCovers treating text as raw bytes, whitespace handling (▁ prefix), BPE and unigram modes, training from raw text, pretokenization elimination, SentencePiece in production, multilingual tokenization.44 min 33Tokenizer TrainingCovers corpus preparation, vocabulary size selection, special tokens configuration, training with HuggingFace tokenizers, saving and loading tokenizers, tokenizer versioning, domain-specific tokenizers.53 min 34Special TokensCovers [CLS], [SEP], [PAD], [MASK], [UNK] tokens, beginning/end of sequence tokens, custom special tokens, special token embeddings, token type IDs, handling special tokens in generation.57 min 35Tokenization ChallengesCovers number tokenization issues, code tokenization, multilingual text mixing, emoji and Unicode edge cases, tokenization artifacts, adversarial tokenization, measuring tokenization quality.61 min

Part VI

Sequence Labeling

8 chapters

36Part-of-Speech TaggingCovers POS tag sets (Penn Treebank, Universal), POS tagging as classification, contextual disambiguation, POS tagging accuracy metrics, POS tagging for downstream tasks, rule-based vs statistical taggers.64 min 37Named Entity RecognitionCovers entity types (PER, ORG, LOC, etc.), NER as sequence labeling, nested entity challenges, entity boundary detection, NER evaluation (exact vs partial match), NER datasets and benchmarks.60 min 38BIO TaggingCovers BIO scheme explanation, BIOES/BILOU variants, converting spans to BIO tags, BIO decoding to spans, handling tagging inconsistencies, BIO for multi-label scenarios, implementing BIO utilities.55 min 39ChunkingCovers noun phrase chunking, chunk types (NP, VP, PP), IOB tagging for chunks, chunking vs full parsing, chunking evaluation, chunking as preprocessing, regex chunking with NLTK.41 min 40Hidden Markov ModelsCovers HMM components (states, observations, transitions), emission and transition probabilities, HMM assumptions (Markov, independence), HMM for POS tagging, HMM parameter estimation, HMM limitations for NLP.49 min 41Viterbi AlgorithmCovers optimal path problem formulation, Viterbi recursion derivation, backpointer tracking, Viterbi complexity analysis, log-space computation, implementing Viterbi efficiently, Viterbi for beam search foundation.47 min 42Conditional Random FieldsCovers CRF vs HMM comparison, CRF feature functions, log-linear formulation, partition function computation, CRF for NER, CRF inference complexity, neural CRF layers.58 min 43CRF TrainingCovers CRF log-likelihood objective, forward-backward algorithm, gradient computation, L-BFGS optimization, feature template design, CRF regularization, CRF training convergence.56 min

Volume 3Linguistic Structure and Meaning2 parts · 11 chapters

Part VII

Linguistic Form: Morphology and Syntax

5 chapters

44Morphology and Word FormationMorphemes, inflection, derivation, compounding, and why word structure matters for language models.37 min 45Morphological Analysis and GenerationFinite-state and neural approaches to analyzing and generating morphologically rich language.46 min 46Grammars and Constituency StructureContext-free grammars, phrase structure, constituency trees, and the limits of symbolic grammars.46 min 47Dependency Syntax and Universal DependenciesHead-dependent relations, dependency trees, projectivity, and cross-lingual annotation with Universal Dependencies.52 min 48Constituency and Dependency ParsingTransition-based, graph-based, chart, and neural parsing methods with their decoding trade-offs.48 min

Part VIII

Semantics and Information Extraction

6 chapters

50Lexical Semantics and Word SensesPolysemy, synonymy, lexical resources, contextual word senses, and word-sense disambiguation.95 min 51Sentence Meaning and Natural Language InferenceCompositional meaning, entailment, contradiction, presupposition, and robust NLI evaluation.66 min 52Semantic Roles and Argument StructurePredicates, arguments, thematic roles, frame semantics, and semantic role labeling.78 min 53Relation, Event, and Temporal ExtractionExtracting relations, events, arguments, and timelines from documents rather than isolated sentences.105 min 54Entity Linking and Knowledge Graph ConstructionLinking mentions to entities, resolving ambiguity, and building provenance-aware knowledge graphs.70 min 55Semantic Parsing and Meaning RepresentationsLogical forms, AMR, text-to-SQL, executable meaning representations, and constrained generation.61 min

Volume 4Neural NLP and Sequence Models3 parts · 29 chaptersBuy this volumeBuy PDF · $24

Part X

Neural Network Foundations

13 chapters

61Linear ClassifiersCovers linear decision boundaries, weight vectors and bias, dot product interpretation, multiclass classification (softmax), linear classifier limitations, training with gradient descent.53 min 62Activation FunctionsCovers sigmoid function and saturation, tanh properties, ReLU and dying ReLU, Leaky ReLU and PReLU, ELU and SELU, GELU derivation and properties, Swish and Mish, choosing activation functions.50 min 63Multilayer PerceptronsCovers hidden layers and depth, weight matrices between layers, forward pass computation, representational capacity, MLP for classification, MLP for regression, MLP architecture design.40 min 64Loss FunctionsCovers cross-entropy loss derivation, MSE for regression, binary vs multiclass cross-entropy, label smoothing, focal loss for imbalance, loss function numerical stability, custom loss functions.56 min 65BackpropagationCovers computational graphs, chain rule review, forward and backward pass, gradient accumulation, backprop complexity analysis, automatic differentiation, implementing backprop from scratch.58 min 66Stochastic Gradient DescentCovers batch vs stochastic gradient descent, minibatch gradient descent, learning rate selection, SGD convergence properties, SGD noise as regularization, learning rate schedules basics, SGD implementation.40 min 67MomentumCovers momentum intuition (ball rolling), momentum update equations, momentum coefficient selection, dampening oscillations, momentum vs vanilla SGD, Nesterov momentum derivation, implementing momentum.50 min 68Adam OptimizerCovers exponential moving averages, first moment (mean) estimation, second moment (variance) estimation, bias correction derivation, Adam update rule, Adam hyperparameters, Adam convergence properties.51 min 69AdamWCovers L2 regularization vs weight decay, why they differ with Adam, AdamW formulation, weight decay coefficient selection, AdamW as default optimizer, AdamW vs Adam empirically.45 min 70Weight InitializationCovers random initialization importance, Xavier/Glorot initialization derivation, He initialization for ReLU, initialization for different activations, layer-wise initialization, initialization debugging, modern initialization practices.53 min 71Batch NormalizationCovers internal covariate shift, batch statistics computation, learnable scale and shift, training vs inference mode, batch norm gradient flow, batch norm placement debates, batch norm limitations.54 min 72DropoutCovers dropout as ensemble, dropout mask sampling, inverted dropout scaling, dropout rate selection, dropout at inference, spatial dropout for sequences, dropout in modern architectures.55 min 73Gradient ClippingCovers gradient explosion detection, clip by value, clip by global norm, gradient clipping implementation, when to use gradient clipping, clipping threshold selection, monitoring gradient norms.50 min

Part XI

Recurrent Neural Networks

9 chapters

74RNN ArchitectureCovers recurrent connection intuition, hidden state as memory, unrolled computation graph, parameter sharing across time, RNN for sequence classification, RNN for sequence generation, RNN equations and dimensions.55 min 75Backpropagation Through TimeCovers BPTT derivation, gradient flow through time, truncated BPTT, BPTT memory requirements, BPTT implementation, gradient accumulation across timesteps.52 min 76Vanishing GradientsCovers gradient product across timesteps, vanishing gradient analysis, long-range dependency failure, gradient visualization, vanishing vs exploding trade-off, architectural solutions overview.45 min 77LSTM ArchitectureCovers cell state as information highway, gate mechanism intuition, LSTM diagram walkthrough, information flow in LSTMs, LSTM for long sequences, LSTM memory capacity.55 min 78LSTM Gate EquationsCovers forget gate equations, input gate equations, cell state update, output gate equations, hidden state computation, LSTM parameter count, implementing LSTM from scratch.40 min 79LSTM Gradient FlowCovers constant error carousel, forget gate gradient highway, gradient flow analysis, LSTM vs vanilla RNN gradients, peephole connections, LSTM gradient clipping needs.52 min 80GRU ArchitectureCovers GRU vs LSTM comparison, reset gate function, update gate function, candidate hidden state, GRU equations, GRU parameter efficiency, when to choose GRU vs LSTM.48 min 81Bidirectional RNNsCovers forward and backward passes, hidden state concatenation, bidirectional architectures, bidirectionality for classification, limitations for generation, implementing bidirectional RNNs.50 min 82Stacked RNNsCovers multiple RNN layers, residual connections for depth, layer normalization in RNNs, depth vs width trade-offs, gradient flow in deep RNNs, practical depth limits.41 min

Part XII

Sequence-to-Sequence

7 chapters

83Encoder-Decoder FrameworkCovers encoder role and design, decoder role and design, context vector as bottleneck, seq2seq for machine translation, seq2seq for summarization, seq2seq training setup.56 min 84Teacher ForcingCovers teacher forcing procedure, exposure bias problem, teacher forcing efficiency, scheduled sampling, curriculum learning, teacher forcing vs autoregressive training.45 min 85Beam SearchCovers greedy decoding limitations, beam search algorithm, beam width selection, length normalization, diverse beam search, beam search implementation, beam search vs sampling.48 min 86Attention IntuitionCovers attention as soft lookup, attention weight interpretation, attention for variable-length inputs, attention visualization, attention vs pooling, attention computation overview.56 min 87Bahdanau AttentionCovers alignment model formulation, score function (additive), attention weight computation, context vector as weighted sum, attention in decoder, Bahdanau attention implementation.51 min 88Luong AttentionCovers dot product attention, general (bilinear) attention, concat attention variant, global vs local attention, Luong vs Bahdanau comparison, attention placement (input vs output).43 min 89Copy MechanismCovers pointer network motivation, copy probability computation, mixing generation and copying, pointer-generator networks, copy mechanism for summarization, OOV handling with copy.60 min

Volume 5Transformers from First Principles4 parts · 27 chaptersBuy this volumeBuy PDF · $24

Part XIII

Self-Attention

6 chapters

90Self-Attention ConceptCovers cross-attention vs self-attention, self-attention motivation, all-pairs interaction, self-attention for representation learning, self-attention computational pattern.50 min 91Query, Key, ValueCovers QKV intuition (database lookup), projection matrices Wq, Wk, Wv, query-key matching, value retrieval, QKV dimensions and shapes, QKV as learned transformations.40 min 92Scaled Dot-Product AttentionCovers dot product for similarity, softmax for normalization, scaling factor derivation (1/√dk), attention output computation, attention in matrix form, attention implementation.48 min 93Attention MaskingCovers padding masks, causal (look-ahead) masks, combining multiple masks, mask shapes and broadcasting, efficient masking implementation, custom attention patterns.52 min 94Multi-Head AttentionCovers multiple attention heads motivation, head dimension splitting, parallel attention computation, output concatenation and projection, head specialization, multi-head vs single head.44 min 95Attention ComplexityCovers O(n²) attention complexity, memory requirements, attention bottleneck in long sequences, FLOPs computation, attention vs RNN complexity, practical scaling limits.62 min

Part XIV

Positional Encoding

7 chapters

96Position ProblemCovers transformer position blindness, why position matters for language, position information requirements, position encoding vs position embedding, absolute vs relative position.47 min 97Sinusoidal Position EncodingCovers sinusoidal formula derivation, wavelength intuition, position encoding visualization, extrapolation properties, sinusoidal encoding implementation, learned vs sinusoidal trade-offs.52 min 98Learned Position EmbeddingsCovers position embedding table, position embedding training, maximum sequence length, learned embedding extrapolation, position embedding analysis, GPT-style position embeddings.48 min 99Relative Position EncodingCovers relative position motivation, relative attention formulation, clipping relative positions, relative position in self-attention, Shaw et al. relative positions, relative bias implementation.49 min 100Rotary Position Embedding (RoPE)Covers RoPE intuition, rotation matrix formulation, RoPE in complex numbers, relative position through rotation, RoPE implementation, RoPE frequency patterns.54 min 101ALiBiCovers ALiBi motivation, linear bias by distance, head-specific slopes, ALiBi extrapolation properties, ALiBi simplicity advantages, ALiBi vs RoPE comparison.52 min 102Position Encoding ComparisonCovers extrapolation benchmarks, training efficiency comparison, implementation complexity, position encoding for long context, hybrid approaches, current best practices.38 min

Part XV

Transformer Blocks

8 chapters

103Residual ConnectionsCovers residual connection formulation, gradient highway interpretation, residual scaling, residual connections in transformers, pre-norm vs post-norm residuals.46 min 104Layer NormalizationCovers layer norm vs batch norm, layer norm formula, learnable affine parameters, layer norm placement, layer norm gradient flow, layer norm implementation.52 min 105RMSNormCovers RMSNorm derivation, removing mean centering, RMSNorm efficiency, RMSNorm vs LayerNorm performance, RMSNorm in modern architectures.58 min 106Pre-Norm vs Post-NormCovers original transformer (post-norm), pre-norm formulation, training stability comparison, gradient flow differences, when to use each, modern consensus.46 min 107Feed-Forward NetworksCovers FFN architecture, hidden dimension expansion, FFN as two linear layers, position independence, FFN parameter count, FFN computational cost.56 min 108FFN Activation FunctionsCovers ReLU in original transformer, GELU adoption, GELU approximations, SiLU/Swish in modern models, activation function comparison.52 min 109Gated Linear UnitsCovers GLU formulation, gating mechanism, SwiGLU derivation, GeGLU variant, GLU parameter efficiency, GLU in modern architectures.44 min 110Transformer Block AssemblyCovers standard block structure, component ordering, block implementation, block initialization, forward pass walkthrough, block hyperparameters.44 min

Part XVI

Transformer Architectures

6 chapters

111Encoder ArchitectureCovers encoder-only design, bidirectional self-attention, encoder for understanding tasks, encoder output usage, BERT-style encoder, encoder layer stacking.45 min 112Decoder ArchitectureCovers decoder-only design, causal masking requirement, autoregressive generation, decoder for generation tasks, GPT-style decoder, decoder layer stacking.38 min 113Encoder-Decoder ArchitectureCovers encoder-decoder interaction, cross-attention mechanism, encoder-decoder for seq2seq, T5-style architecture, information flow, when to use encoder-decoder.42 min 114Cross-AttentionCovers cross-attention formulation, KV from encoder, Q from decoder, cross-attention masking, cross-attention placement, cross-attention implementation.59 min 115Weight TyingCovers input-output embedding tying, encoder-decoder tying, parameter reduction, weight tying effects on training, when to tie weights.53 min 116Architecture HyperparametersCovers depth vs width trade-offs, number of heads selection, hidden dimension ratios, FFN expansion ratio, total parameter calculation, architecture search.39 min

Volume 6Efficient, Long-Context, and Alternative Architectures2 parts · 16 chaptersBuy this volumeBuy PDF · $24

Part XVII

Efficient Attention

9 chapters

117Quadratic Attention BottleneckCovers O(n²) memory analysis, O(n²) compute analysis, attention matrix size, practical sequence limits, bottleneck visualization, motivation for efficiency.48 min 118Sparse Attention PatternsCovers local attention windows, strided attention patterns, block-sparse attention, combining sparse patterns, sparse attention implementation.61 min 119Sliding Window AttentionCovers sliding window formulation, window size selection, dilated sliding windows, sliding window for long sequences, Mistral-style windowed attention.43 min 120Global TokensCovers CLS token global attention, learned global tokens, global-local attention mixing, global token count, implementation strategies.51 min 121LongformerCovers Longformer attention pattern, global attention configuration, Longformer complexity, Longformer for documents, Longformer implementation.55 min 122BigBirdCovers BigBird attention pattern, random attention benefits, BigBird theoretical guarantees, BigBird vs Longformer, BigBird applications.40 min 123Linear AttentionCovers softmax attention reformulation, kernel feature maps, linear complexity attention, linear attention limitations, Performer and variants.42 min 124FlashAttention AlgorithmCovers GPU memory hierarchy, tiling for SRAM, online softmax computation, recomputation strategy, FlashAttention complexity, FlashAttention benefits.45 min 125FlashAttention ImplementationCovers CUDA kernel basics, memory access patterns, FlashAttention-2 improvements, using FlashAttention in PyTorch, FlashAttention limitations.51 min

Part XVIII

Long Context

7 chapters

126Context Length ChallengesCovers training sequence length limits, attention memory scaling, position encoding extrapolation, long-range dependency learning, evaluation challenges.39 min 127Position InterpolationCovers linear position scaling, interpolation vs extrapolation, position interpolation implementation, fine-tuning for longer context, interpolation limitations.49 min 128NTK-aware ScalingCovers RoPE frequency analysis, high-frequency preservation, NTK-aware formula, dynamic NTK scaling, NTK vs linear interpolation.57 min 129YaRNCovers YaRN motivation, attention scaling factor, YaRN formula, YaRN training requirements, YaRN vs alternatives.58 min 130Attention SinksCovers attention sink phenomenon, StreamingLLM approach, sink token design, streaming inference, infinite context generation.57 min 131Memory AugmentationCovers memory network concepts, memory retrieval mechanisms, memory writing and updating, memory-augmented transformers, Memorizing Transformers.51 min 132Recurrent MemoryCovers Transformer-XL approach, segment-level processing, recurrent state passing, relative position in recurrence, recurrent memory limitations.68 min

Volume 7Pre-training Data and Scaling3 parts · 24 chaptersBuy this volumeBuy PDF · $24

Part XX

Data Curation

10 chapters

139Web CrawlingCovers Common Crawl, crawling strategies, robots.txt respect, crawl freshness.52 min 140Document ExtractionCovers HTML parsing, boilerplate removal, content extraction, trafilatura and similar tools.55 min 141Language IdentificationCovers language ID models, multilingual document handling, code-switching, language filtering.52 min 142DeduplicationCovers exact deduplication, near-duplicate detection, document vs substring dedup, dedup at scale.50 min 143MinHashCovers MinHash algorithm, Jaccard similarity estimation, MinHash LSH, MinHash implementation.47 min 144Quality FilteringCovers heuristic filters, perplexity filtering, classifier-based filtering, filter thresholds.50 min 145Toxicity FilteringCovers toxicity classifiers, toxicity thresholds, over-filtering risks, toxicity filter evaluation.49 min 146PII RemovalCovers PII detection methods, PII removal strategies, PII removal evaluation, privacy preservation.53 min 147Data MixingCovers domain proportions, quality weighting, data mixing experiments, optimal mixing.59 min 148Synthetic DataCovers synthetic data generation, quality verification, synthetic data diversity, distillation.65 min

Part XXII

Pre-training Objectives

7 chapters

155Causal Language ModelingCovers CLM objective formulation, autoregressive factorization, CLM loss computation, CLM for generation, CLM training data, CLM scaling properties.42 min 156Masked Language ModelingCovers MLM objective formulation, masking strategies (15% rule), [MASK] token usage, MLM for understanding, MLM training dynamics.49 min 157Whole Word MaskingCovers subword masking problems, whole word masking procedure, WWM implementation, WWM vs random masking, WWM for different tokenizers.54 min 158Span CorruptionCovers span selection strategies, span length distribution, sentinel tokens, T5-style corruption, span corruption benefits.51 min 159Prefix Language ModelingCovers prefix LM formulation, prefix LM attention pattern, prefix LM for generation, prefix LM training, UniLM-style objectives.40 min 160Replaced Token DetectionCovers generator-discriminator setup, replaced vs original detection, RTD efficiency advantages, ELECTRA training procedure, RTD vs MLM comparison.58 min 161Denoising ObjectivesCovers token deletion, token shuffling, sentence permutation, document rotation, BART-style denoising, combining denoising tasks.47 min

Part XXIII

Scaling Laws

7 chapters

162Power Laws in Deep LearningCovers power law definition, log-log linear relationships, power law fitting, power law universality, power law intuition.69 min 163Kaplan Scaling LawsCovers loss vs parameters, loss vs data, loss vs compute, Kaplan optimal allocation, Kaplan predictions.40 min 164Chinchilla Scaling LawsCovers Chinchilla experiments, revised scaling coefficients, optimal tokens per parameter, Chinchilla vs Kaplan, Chinchilla implications.56 min 165Compute-Optimal TrainingCovers compute budget allocation, tokens vs parameters ratio, training efficiency, compute-optimal recipes, practical guidelines.40 min 166Data-Constrained ScalingCovers data repetition effects, optimal repetition strategies, data augmentation scaling, synthetic data scaling.39 min 167Inference ScalingCovers training vs inference compute, inference-optimal models, over-training for efficiency, deployment cost modeling.42 min 168Predicting Model PerformanceCovers loss extrapolation, capability prediction, scaling law uncertainty, prediction reliability, practical forecasting.58 min

Volume 8Encoder, Multilingual, and Translation Models2 parts · 14 chaptersBuy this volumeBuy PDF · $24

Part XXIV

BERT and Variants

8 chapters

169BERT ArchitectureCovers BERT model sizes, BERT layer configuration, BERT embedding layers, BERT attention patterns, BERT output representations.52 min 170BERT Pre-trainingCovers pre-training data preparation, MLM implementation, NSP task design, pre-training hyperparameters, pre-training duration.42 min 171BERT Fine-tuningCovers classification fine-tuning, sequence labeling fine-tuning, question answering fine-tuning, fine-tuning hyperparameters, catastrophic forgetting.44 min 172BERT RepresentationsCovers [CLS] token usage, layer selection strategies, pooling strategies, BERT as feature extractor, frozen vs fine-tuned representations.53 min 173RoBERTaCovers dynamic masking, NSP removal, larger batches, more data, RoBERTa training recipe, RoBERTa vs BERT performance.55 min 174ALBERTCovers factorized embeddings, cross-layer parameter sharing, sentence order prediction, ALBERT efficiency, ALBERT performance trade-offs.42 min 175ELECTRACovers generator training, discriminator training, RTD objective, ELECTRA sample efficiency, ELECTRA scaling, ELECTRA fine-tuning.42 min 176DeBERTaCovers disentangled attention formulation, enhanced mask decoder, DeBERTa position encoding, DeBERTa improvements, DeBERTa-v3 advances.61 min

Part XXV

Encoder-Decoder Models

6 chapters

177T5 ArchitectureCovers T5 encoder-decoder design, T5 attention patterns, T5 model sizes, T5 relative positions, T5 implementation.51 min 178T5 Pre-trainingCovers span corruption procedure, sentinel tokens, corruption rate, T5 pre-training data, T5 training scale.38 min 179T5 Task FormattingCovers task prefixes, classification as generation, NER as generation, QA as generation, task formatting examples.57 min 180BART ArchitectureCovers BART encoder-decoder, BART attention configuration, BART vs T5 comparison, BART model sizes.57 min 181BART Pre-trainingCovers token masking, token deletion, text infilling, sentence permutation, document rotation, objective combinations.40 min 182mT5Covers mT5 training data, language sampling, cross-lingual transfer, mT5 vs T5 performance, multilingual tokenization.46 min

Volume 9Decoder Language Models and Capabilities3 parts · 23 chaptersBuy this volumeBuy PDF · $24

Part XXVIII

GPT Architecture

10 chapters

197GPT-1Covers GPT-1 architecture, GPT-1 pre-training, GPT-1 fine-tuning approach, GPT-1 transfer learning, GPT-1 historical significance.68 min 198GPT-2Covers GPT-2 model sizes, GPT-2 architectural changes, zero-shot task performance, GPT-2 training data (WebText), GPT-2 generation quality.52 min 199GPT-3Covers GPT-3 scale (175B), few-shot prompting discovery, in-context learning analysis, GPT-3 capabilities, GPT-3 limitations.93 min 200In-Context LearningCovers ICL phenomenon, ICL vs fine-tuning, example selection strategies, ICL scaling behavior, ICL theoretical understanding.51 min 201Autoregressive GenerationCovers generation procedure, KV caching for efficiency, generation stopping criteria, generation speed optimization, generation code implementation.77 min 202Decoding TemperatureCovers temperature scaling, temperature effects on distribution, temperature selection guidelines, temperature vs quality trade-off.55 min 203Top-k SamplingCovers top-k truncation, k selection strategies, top-k limitations, top-k implementation, combining with temperature.52 min 204Nucleus SamplingCovers top-p formulation, cumulative probability threshold, nucleus sampling benefits, p selection guidelines, nucleus vs top-k.42 min 205Repetition PenaltiesCovers repetition in generation, repetition penalty formulation, frequency penalty, presence penalty, n-gram blocking.56 min 206Constrained DecodingCovers grammar-guided generation, JSON schema constraints, regex constraints, constrained beam search, constrained sampling.41 min

Part XXIX

Modern Decoder Models

7 chapters

207LLaMA ArchitectureCovers LLaMA design philosophy, LLaMA architectural choices, LLaMA training data, LLaMA efficiency, LLaMA significance.50 min 208LLaMA ComponentsCovers pre-norm with RMSNorm, SwiGLU FFN, RoPE implementation, component interactions, implementation details.42 min 209Grouped Query AttentionCovers GQA motivation, GQA formulation, KV head grouping, GQA memory savings, GQA vs MHA performance, GQA implementation.39 min 210Multi-Query AttentionCovers MQA extreme sharing, MQA memory benefits, MQA quality trade-offs, MQA for inference, MQA vs GQA.39 min 211Mistral ArchitectureCovers Mistral design choices, sliding window attention, Mistral efficiency, Mistral performance, Mistral vs LLaMA.48 min 212Qwen ArchitectureCovers Qwen architectural choices, Qwen training approach, Qwen multilingual capabilities, Qwen variants.49 min 213Phi ModelsCovers Phi design philosophy, textbook-quality data, Phi training approach, Phi efficiency, small model capabilities.42 min

Part XXX

Emergent Capabilities

6 chapters

214Emergence in Neural NetworksCovers emergence definition, phase transitions, emergence examples, emergence mechanisms, emergence debate.37 min 215In-Context Learning EmergenceCovers ICL emergence curves, ICL vs fine-tuning scaling, ICL mechanism hypotheses, ICL as meta-learning.53 min 216Chain-of-Thought EmergenceCovers CoT emergence observations, CoT elicitation, CoT scaling behavior, CoT mechanism theories.43 min 217Emergence vs MetricsCovers discontinuous metrics, accuracy threshold effects, smooth underlying capabilities, re-examining emergence claims.44 min 218Inverse ScalingCovers inverse scaling phenomena, distractor tasks, sycophancy scaling, inverse scaling prize findings.84 min 219GrokkingCovers grokking phenomenon, grokking in arithmetic, grokking mechanism theories, grokking phase transitions, practical implications.78 min

Volume 10Large-Scale Training and Sparse Models3 parts · 29 chaptersBuy this volumeBuy PDF · $24

Part XXXI

Training Infrastructure

11 chapters

220GPU ArchitectureCovers GPU memory hierarchy, CUDA cores, tensor cores, GPU specifications.53 min 221Memory ManagementCovers memory breakdown (activations, parameters, gradients, optimizer states), memory estimation, OOM debugging.47 min 222Data ParallelismCovers DDP algorithm, gradient synchronization, all-reduce operations, DDP scaling.61 min 223Tensor ParallelismCovers column parallelism, row parallelism, communication patterns, Megatron-style parallelism.50 min 224Pipeline ParallelismCovers pipeline stages, micro-batching, pipeline bubbles, pipeline schedules (GPipe, 1F1B).54 min 225ZeRO OptimizationCovers ZeRO stage 1 (optimizer state partitioning), ZeRO stage 2 (gradient partitioning), ZeRO stage 3 (parameter partitioning), ZeRO memory savings.46 min 226FSDPCovers FSDP concepts, FSDP vs ZeRO, FSDP sharding strategies, FSDP usage.47 min 227Activation CheckpointingCovers checkpointing concept, checkpoint selection, checkpointing overhead, selective checkpointing.56 min 228Mixed Precision TrainingCovers floating point formats, loss scaling, BF16 advantages, mixed precision implementation.58 min 229Communication OptimizationCovers gradient compression, communication overlap, topology-aware communication, NCCL optimization.55 min 230Checkpointing and RecoveryCovers checkpoint contents, checkpoint frequency, async checkpointing, fault recovery.53 min

Part XXXII

Training Optimization

8 chapters

231Learning Rate WarmupCovers warmup motivation, linear warmup, warmup duration, warmup for large batches.58 min 232Learning Rate DecayCovers step decay, exponential decay, inverse square root decay, decay scheduling.51 min 233Cosine Learning Rate ScheduleCovers cosine decay formula, cosine with restarts, cosine schedule parameters, cosine vs linear.65 min 234Large Batch TrainingCovers batch size effects, learning rate scaling, batch size limits, LAMB optimizer.49 min 235Weight DecayCovers weight decay formula, decoupled weight decay, weight decay selection, weight decay interaction with Adam.56 min 236Gradient AccumulationCovers accumulation procedure, accumulation steps, accumulation for memory, accumulation correctness.49 min 237Training StabilityCovers loss spikes, gradient norm monitoring, stability techniques, training stability debugging.60 min 238Hyperparameter SelectionCovers hyperparameter search, hyperparameter transfer, critical vs robust hyperparameters, default recipes.58 min

Part XXXIII

Mixture of Experts

10 chapters

239Sparse ModelsCovers dense vs sparse trade-offs, conditional computation motivation, sparse model efficiency, sparse model challenges.43 min 240Expert NetworksCovers expert architecture, expert as FFN, expert capacity, expert count selection, expert placement in transformer.57 min 241Gating NetworksCovers router architecture, routing score computation, router training, router learned behavior.38 min 242Top-K RoutingCovers top-1 routing, top-2 routing, k selection trade-offs, routing implementation, combining expert outputs.54 min 243Load BalancingCovers expert utilization imbalance, collapse failure mode, load metrics, balanced routing importance.57 min 244Auxiliary Balancing LossCovers load balancing loss formulation, loss coefficient tuning, balancing vs task loss, auxiliary loss implementation.49 min 245Router Z-LossCovers router instability, z-loss formulation, z-loss benefits, z-loss coefficient, combined auxiliary losses.61 min 246Expert ParallelismCovers expert placement strategies, all-to-all communication, communication overhead, expert parallelism implementation.58 min 247Switch TransformerCovers Switch Transformer design, top-1 routing choice, capacity factor, Switch scaling results.64 min 248MixtralCovers Mixtral architecture, Mixtral expert design, Mixtral performance, Mixtral efficiency, Mixtral vs dense models.77 min

Volume 11Fine-tuning and Efficient Adaptation3 parts · 23 chaptersBuy this volumeBuy PDF · $24

Part XXXIV

Fine-tuning Fundamentals

5 chapters

249Transfer LearningCovers transfer learning paradigm, pre-training/fine-tuning split, what transfers, transfer learning efficiency.60 min 250Full Fine-tuningCovers full fine-tuning procedure, fine-tuning hyperparameters, learning rate selection, batch size effects.66 min 251Catastrophic ForgettingCovers forgetting phenomenon, forgetting measurement, forgetting mitigation, pre-trained capability preservation.66 min 252Fine-tuning Learning RatesCovers discriminative fine-tuning, layer-wise learning rates, warmup for fine-tuning, learning rate decay.57 min 253Fine-tuning Data EfficiencyCovers few-shot fine-tuning, data augmentation, sample efficiency patterns, small data strategies.46 min

Part XXXV

Parameter-Efficient Fine-tuning

12 chapters

254PEFT MotivationCovers parameter storage costs, multi-task deployment, PEFT efficiency, PEFT quality trade-offs.53 min 255LoRA ConceptCovers weight update decomposition, low-rank assumption, LoRA efficiency gains, LoRA flexibility.56 min 256LoRA MathematicsCovers LoRA formulation W + BA, rank selection, initialization scheme, LoRA gradient computation.67 min 257LoRA ImplementationCovers LoRA module design, merging weights, LoRA training loop, LoRA in PyTorch, HuggingFace PEFT usage.55 min 258LoRA HyperparametersCovers rank selection guidelines, alpha/rank ratio, which layers to adapt, LoRA dropout.57 min 259QLoRACovers 4-bit quantization for base model, NF4 data type, double quantization, QLoRA memory savings.54 min 260AdaLoRACovers importance-based pruning, SVD-based adaptation, dynamic rank, AdaLoRA training procedure.61 min 261IA3Covers IA3 formulation, learned rescaling vectors, IA3 parameter efficiency, IA3 vs LoRA.60 min 262Prefix TuningCovers prefix tuning formulation, prefix length selection, prefix tuning for generation, prefix vs LoRA.64 min 263Prompt TuningCovers prompt tuning formulation, prompt initialization, prompt tuning scaling, prompt length effects.61 min 264Adapter LayersCovers adapter architecture, adapter placement, adapter dimensionality, adapter fusion.69 min 265PEFT ComparisonCovers performance comparison, parameter efficiency comparison, task suitability, practical recommendations.66 min

Part XXXVI

Instruction Tuning

6 chapters

266Instruction FollowingCovers instruction tuning motivation, instruction format design, instruction diversity, instruction quality.54 min 267Instruction Data CreationCovers human annotation, template-based generation, seed task expansion, quality filtering.64 min 268Self-InstructCovers self-instruct procedure, instruction generation, response generation, filtering strategies.71 min 269Instruction FormatCovers prompt templates, system messages, multi-turn format, chat templates, role definitions.56 min 270Instruction Tuning TrainingCovers instruction tuning data mixing, training hyperparameters, loss masking, multi-task learning.44 min 271Instruction Following EvaluationCovers instruction following benchmarks, human evaluation, automatic evaluation, instruction difficulty.59 min

Volume 12Alignment, Preferences, and Oversight1 part · 16 chaptersBuy this volumeBuy PDF · $24

Part XXXVII

Alignment and RLHF

16 chapters

272Alignment ProblemCovers alignment definition, helpfulness vs harmlessness, alignment challenges, alignment approaches overview.70 min 273Human Preference DataCovers preference collection UI, comparison design, annotator guidelines, preference data quality.67 min 274Bradley-Terry ModelCovers pairwise comparison model, preference probability, Bradley-Terry likelihood, preference strength.63 min 275Reward ModelingCovers reward model architecture, preference loss function, reward model training, reward model evaluation.68 min 276Reward HackingCovers reward hacking examples, distribution shift, over-optimization, reward hacking mitigation.68 min 277Policy Gradient MethodsCovers policy definition, REINFORCE algorithm, policy gradient derivation, variance reduction.59 min 278PPO AlgorithmCovers clipped objective, PPO derivation, trust region intuition, PPO implementation.71 min 279PPO for Language ModelsCovers LLM as policy, action space (tokens), reward assignment, KL penalty importance.64 min 280RLHF PipelineCovers SFT stage, reward model training, PPO fine-tuning, RLHF hyperparameters, RLHF debugging.61 min 281KL Divergence PenaltyCovers KL penalty motivation, KL coefficient selection, adaptive KL, KL effects on training.69 min 282DPO ConceptCovers DPO motivation, removing reward model, DPO intuition, DPO benefits.53 min 283DPO DerivationCovers DPO from RLHF objective, optimal policy derivation, DPO loss function, DPO as classification.45 min 284DPO ImplementationCovers DPO data format, DPO loss computation, DPO training procedure, DPO hyperparameters.56 min 285DPO VariantsCovers IPO formulation, KTO for unpaired feedback, ORPO, cDPO, comparing alignment methods.65 min 286RLAIFCovers AI as annotator, constitutional AI principles, AI preference generation, RLAIF scalability.51 min 287Iterative AlignmentCovers iterative DPO, online preference learning, self-improvement loops, alignment stability.48 min

Volume 13Reasoning and Verifiable Learning1 part · 7 chaptersBuy this volumeBuy PDF · $24

Part XXXIX

Reasoning

7 chapters

293Reasoning FoundationsCovers reasoning types, reasoning in LLMs, reasoning failure modes, reasoning evaluation.58 min 294Chain-of-ThoughtCovers CoT prompting, zero-shot CoT, CoT fine-tuning, CoT limitations.52 min 295Reasoning StrategiesCovers self-consistency, tree of thought, least-to-most prompting, decomposition strategies.49 min 296Reasoning VerificationCovers step verification, process reward models, verification-guided search, self-correction.57 min 297Mathematical ReasoningCovers math problem solving, symbolic integration, math benchmarks, math reasoning training.54 min 298Reasoning LimitationsCovers systematic failures, spurious correlations, reasoning shortcuts, robustness challenges.56 min 299Reasoning FrontiersCovers o1-style reasoning, test-time compute scaling, reasoning-capable models, open research questions.57 min

Volume 14Efficient Inference and On-Device Models2 parts · 20 chaptersBuy this volumeBuy PDF · $24

Part XLI

Model Compression

6 chapters

307Knowledge DistillationCovers distillation objective, temperature in distillation, teacher selection, distillation for LLMs.45 min 308Distillation VariantsCovers feature distillation, attention transfer, progressive distillation, on-policy distillation.48 min 309Pruning BasicsCovers weight pruning, structured vs unstructured, pruning criteria, pruning schedule.51 min 310Structured PruningCovers head pruning, layer pruning, width pruning, structured pruning implementation.48 min 311Model MergingCovers weight averaging, task arithmetic, TIES merging, DARE merging.44 min 312Model Merging ApplicationsCovers multi-task merging, style merging, capability composition, merging evaluation.50 min

Part XLII

Inference Optimization

14 chapters

313KV CacheCovers KV cache motivation, cache structure, cache memory requirements, cache management.56 min 314KV Cache MemoryCovers cache size calculation, batch size effects, sequence length effects, memory bottleneck.52 min 315Paged AttentionCovers memory fragmentation problem, page-based allocation, vLLM approach, paged attention benefits.68 min 316KV Cache CompressionCovers cache eviction strategies, attention sink preservation, H2O algorithm, cache quantization.56 min 317Weight Quantization BasicsCovers quantization fundamentals, per-tensor vs per-channel, symmetric vs asymmetric, calibration.59 min 318INT8 QuantizationCovers INT8 range mapping, absmax quantization, smooth quantization, INT8 accuracy.59 min 319INT4 QuantizationCovers 4-bit challenges, group-wise quantization, 4-bit accuracy trade-offs, 4-bit formats.59 min 320GPTQCovers GPTQ algorithm, layer-wise quantization, Hessian approximation, GPTQ implementation.49 min 321AWQCovers salient weight preservation, AWQ algorithm, AWQ vs GPTQ, AWQ benefits.44 min 322GGUF FormatCovers GGML/GGUF history, quantization types, GGUF file format, llama.cpp integration.55 min 323Speculative Decoding: Fast LLM Inference Without Quality LossCovers speculative decoding concept, draft model selection, verification procedure, acceptance rate.54 min 324Speculative Decoding Math: Algorithms & Speedup LimitsCovers acceptance criterion, expected speedup, draft quality effects, optimal draft length.52 min 325Continuous Batching: Optimizing LLM Inference ThroughputCovers static vs continuous batching, iteration-level scheduling, request completion handling, throughput gains.58 min 326LLM Inference Serving: Architecture, Routing & Auto-ScalingMaster LLM inference serving architecture, token-aware load balancing, and auto-scaling. Optimize time-to-first-token and throughput for production systems.73 min

Volume 15Retrieval, Knowledge, and Memory2 parts · 19 chaptersBuy this volumeBuy PDF · $24

Part XLIV

Retrieval-Augmented Generation

14 chapters

332RAG Motivation: Solving Hallucinations & Knowledge GapsDiscover why LLMs need Retrieval-Augmented Generation. Learn how RAG bridges knowledge gaps, reduces hallucinations, and enables non-parametric memory.59 min 333RAG Architecture: Components, Timing & Design PatternsMaster RAG system design by exploring retriever-generator interactions, timing strategies like iterative retrieval, and architectural variations like RETRO.60 min 334Dense Retrieval: Semantic Search & Bi-Encoder ImplementationMaster dense retrieval for semantic search. Explore bi-encoder architectures, embedding metrics, and contrastive learning to overcome keyword limitations.62 min 335Contrastive Learning for Retrieval: InfoNCE & DPR GuideMaster contrastive learning for dense retrieval. Learn to train models using InfoNCE loss, in-batch negatives, and hard negative mining strategies effectively.57 min 336Document Chunking: Optimizing RAG Retrieval PipelinesMaster document chunking for RAG systems. Explore fixed-size, recursive, and semantic strategies to balance retrieval precision with context window limits.63 min 337Embedding Models: Architecture, Pooling & SelectionLearn how embedding models convert text to vectors for RAG. Covers bi-encoder architecture, pooling strategies, dimensionality trade-offs, and model selection.65 min 338Vector Similarity Search: Metrics & Approximate MethodsExplore vector similarity search for RAG systems. Compare cosine, dot product, and Euclidean metrics, and implement exact vs. approximate search with FAISS.65 min 339HNSW Index: Architecture for Fast Vector SearchMaster Hierarchical Navigable Small World (HNSW) graphs for vector search. Learn graph architecture, construction, and tuning for high-speed retrieval.67 min 340IVF Index: Clustering-Based Vector Search & PartitioningMaster IVF indexes for scalable vector search. Learn clustering-based partitioning, nprobe tuning, and IVF-PQ compression for billion-scale retrieval.66 min 341Product Quantization: Vector Compression for ANN SearchLearn how Product Quantization compresses embeddings up to 100x using learned codebooks and asymmetric distance computation for scalable vector search.62 min 342Hybrid Search: BM25 and Dense Retrieval CombinedLearn how hybrid search fuses BM25 keyword retrieval with dense vector retrieval using reciprocal rank fusion and weighted score combination to improve recall.65 min 343Reranking: Cross-Encoders for Precise Information RetrievalLearn how reranking with cross-encoders solves bi-encoder limitations. Master two-stage retrieval, training strategies, and latency optimization for production search systems.61 min 344RAG Prompt Engineering: Context Placement & Citation StrategiesMaster RAG prompt engineering with strategic context placement, citation formats, and truncation strategies to improve LLM accuracy and reduce hallucinations.53 min 345RAG Evaluation: Metrics for Retrieval and Generation QualityMaster RAG evaluation with metrics for retrieval quality (Precision@K, NDCG, MRR) and generation faithfulness using the RAGAS framework for AI systems.65 min

Part XLVI

Continual Learning

5 chapters

352Continual Learning ProblemCovers continual learning definition, catastrophic forgetting, continual learning scenarios.57 min 353Regularization MethodsCovers elastic weight consolidation, synaptic intelligence, parameter importance, regularization trade-offs.48 min 354Replay MethodsCovers replay buffer design, pseudo-rehearsal, generative replay, replay selection.52 min 355Architecture MethodsCovers progressive networks, expert expansion, architecture search, modular approaches.46 min 356Continual Learning EvaluationCovers forward transfer, backward transfer, evaluation protocols, continual benchmarks.56 min

Volume 16Agents and AI Coding Systems2 parts · 16 chaptersBuy this volumeBuy PDF · $24

Part XLVII

Code Generation

6 chapters

357Code LLM TrainingCovers code training data, code tokenization, fill-in-the-middle training, code pre-training objectives.56 min 358Code UnderstandingCovers code explanation, bug detection, code review, code search.48 min 359Code CompletionCovers completion context, completion ranking, completion latency, completion UX.55 min 360Code GenerationCovers docstring-to-code, test-to-code, code generation strategies, generation quality.52 min 361Code ExecutionCovers sandboxed execution, execution feedback, iterative refinement, execution safety.53 min 362Code EvaluationCovers functional correctness, pass@k metric, code benchmarks, beyond correctness.53 min

Part XLVIII

Tool Use and Agents

10 chapters

363Tool Use Motivation: Why LLMs Need External Tools for AccuracyDiscover why LLMs require external tools to overcome knowledge cutoffs, computational limits, and hallucinations. Learn about tool-augmented AI systems.57 min 364Function Calling: Structured Tool Use for Large Language ModelsLearn how function calling enables LLMs to invoke external tools and APIs through structured JSON schemas, bridging natural language and executable code.56 min 365ReAct Pattern: Interleaving Reasoning and Action for LLM AgentsLearn how the ReAct pattern enables LLM agents to interleave reasoning with tool execution. Master thought-action-observation loops for autonomous AI systems.50 min 366Tool Selection for LLM Agents: Routing Strategies and ImplementationMaster LLM tool selection through embedding-based routing, hybrid strategies, and semantic interfaces. Learn to build scalable multi-tool agent systems.58 min 367Agent Architectures: Control Loops, State & PlanningMaster LLM agent architectures including control loops, state management strategies, planning mechanisms, and termination conditions for autonomous AI systems.53 min 368Agent Memory Systems: From Context to Persistent StorageLearn how AI agents manage short-term context and long-term memory. Explore vector databases, retrieval algorithms, and memory hierarchies that enable persistent, learning systems.52 min 369Agent Evaluation: Metrics, Benchmarks and Safety StandardsLearn to evaluate AI agents with task completion metrics, trajectory analysis, and safety testing. Covers WebArena, SWE-bench, GAIA, and OSWorld benchmarks.52 min 370PlanningCovers task decomposition, goal-directed planning, plan execution, plan revision and recovery.61 min 371Multi-Agent SystemsCovers agent coordination, communication protocols, role assignment, multi-agent benchmarks.55 min 372Agent SafetyCovers unsafe action prevention, agent alignment, sandboxing, monitoring and intervention.55 min

Volume 17Speech, Multimodal, and Embodied Language2 parts · 17 chaptersBuy this volumeBuy PDF · $24

Part L

Multimodal Models

12 chapters

381Vision TransformerCovers image patching, patch embeddings, ViT architecture, ViT pre-training.53 min 382CLIPCovers CLIP architecture, CLIP training objective, CLIP zero-shot classification, CLIP embeddings.49 min 383Vision Encoders for VLMsCovers ViT variants for VLMs, SigLIP improvements, image resolution handling, encoder selection.52 min 384Vision-Language ProjectionCovers linear projection, MLP projection, Q-Former approach, projection training.52 min 385LLaVA ArchitectureCovers LLaVA design, two-stage training, visual conversation, LLaVA variants.58 min 386Flamingo ArchitectureCovers cross-attention to images, gated cross-attention, few-shot visual learning, Flamingo training.57 min 387Multimodal Training DataCovers image-text pairs, interleaved documents, visual instruction data, data quality.63 min 388Multimodal EvaluationCovers VQA benchmarks, multimodal understanding benchmarks, multimodal generation evaluation.54 min 389Multimodal FoundationsCovers multimodal learning principles, cross-modal alignment, joint embedding spaces, multimodal challenges.50 min 390Image UnderstandingCovers visual question answering, image captioning, visual grounding, scene understanding.53 min 391Image GenerationCovers diffusion models, text-to-image generation, image editing, generation evaluation.55 min 392Multimodal ApplicationsCovers document understanding, medical imaging, video understanding, multimodal reasoning tasks.61 min

Part LI

Speech and Audio

5 chapters

393Speech RepresentationsCovers mel spectrograms, mel filterbanks, feature normalization, audio preprocessing.60 min 394Whisper ArchitectureCovers Whisper encoder-decoder, multitask training, language tokens, timestamp prediction.59 min 395Whisper TrainingCovers Whisper training data, weak supervision, multilingual training, Whisper capabilities.59 min 396Speech-Language IntegrationCovers speech encoder + LLM, audio tokens, speech-to-text-to-LLM vs end-to-end, speech LLM architectures.59 min 397Text-to-SpeechCovers TTS architecture overview, vocoder role, TTS quality metrics, neural TTS approaches.58 min

Volume 18Evaluation, Factuality, and Reliability4 parts · 30 chaptersBuy this volumeBuy PDF · $24

Part LIII

Evaluation Fundamentals

10 chapters

405Perplexity EvaluationCovers perplexity calculation, perplexity interpretation, perplexity limitations, comparing perplexities.54 min 406Cross-Entropy LossCovers cross-entropy definition, bits-per-character, cross-entropy vs perplexity, loss curves.55 min 407BLEU ScoreCovers n-gram precision, brevity penalty, BLEU formula, BLEU limitations, corpus vs sentence BLEU.45 min 408ROUGE ScoresCovers ROUGE-N, ROUGE-L, ROUGE-W, ROUGE interpretation, ROUGE limitations.56 min 409BERTScoreCovers BERTScore computation, token alignment, BERTScore variants, BERTScore vs BLEU.52 min 410Exact Match and F1Covers exact match scoring, token-level F1, normalization for matching, metric selection.52 min 411CalibrationCovers calibration definition, expected calibration error, calibration plots, calibration methods.55 min 412Evaluation FundamentalsCovers evaluation design principles, metric selection, evaluation pitfalls, evaluation frameworks.59 min 413Benchmark DesignCovers benchmark construction, dataset collection, annotation guidelines, benchmark validity.56 min 414Evaluation ChallengesCovers benchmark contamination, evaluation brittleness, gaming metrics, evaluation best practices.49 min

Part LIV

Benchmark Evaluation

8 chapters

415MMLUCovers MMLU structure, subject coverage, MMLU evaluation protocol, MMLU limitations.54 min 416HellaSwagCovers HellaSwag task design, adversarial filtering, HellaSwag evaluation, HellaSwag saturation.54 min 417GSM8KCovers GSM8K problem types, chain-of-thought evaluation, GSM8K accuracy metrics, math reasoning assessment.49 min 418HumanEvalCovers HumanEval structure, functional correctness, pass@k metric, HumanEval limitations.40 min 419MBPPCovers MBPP dataset, MBPP vs HumanEval, code evaluation challenges.53 min 420TruthfulQACovers TruthfulQA design, truthfulness vs informativeness, TruthfulQA evaluation methods.52 min 421Benchmark ContaminationCovers contamination problem, contamination detection methods, n-gram overlap analysis, contamination mitigation.46 min 422Benchmark SaturationCovers ceiling effects, benchmark retirement, dynamic benchmarks, benchmark evolution.55 min

Part LV

Human and Model Evaluation

6 chapters

423Human Evaluation DesignCovers evaluation interface design, task instructions, annotator selection, evaluation cost.56 min 424Inter-Annotator AgreementCovers Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, handling disagreement.47 min 425Preference EvaluationCovers A/B comparison design, Elo rating systems, preference aggregation, statistical significance.59 min 426LLM-as-Judge: Scalable AI Evaluation with Language ModelsLearn how to build LLM-as-Judge evaluation pipelines: prompt design, judge model selection, calibration against human annotations, and bias mitigation.55 min 427Position Bias in LLM JudgesCovers position bias measurement, bias mitigation (swapping), verbosity bias, sycophancy.46 min 428Evaluation Prompt Engineering: Designing Reliable LLM JudgesLearn how to design reliable LLM judge prompts using explicit criteria, few-shot examples, and chain-of-thought formatting to maximize evaluation accuracy.59 min

Part LVI

Hallucination and Factuality

6 chapters

429Hallucination Types in Language Models: A Complete GuideLearn how language models hallucinate: intrinsic and extrinsic hallucination, factual errors, fabrication, and inconsistency with NLI-based detection.53 min 430Hallucination Detection: NLI, Self-Consistency & Learned ModelsLearn four methods for detecting LLM hallucinations: entailment-based scoring, knowledge base verification, self-consistency checks, and learned detection models.51 min 431Hallucination CausesCovers training data issues, exposure bias, knowledge gaps, generation pressure.54 min 432Hallucination Mitigation: RAG, Decoding, and TrainingLearn how to reduce LLM hallucination using retrieval augmentation, self-consistency decoding, DPO training, and calibrated uncertainty expression.60 min 433Attribution and Citation: Sourcing LLM OutputsLearn how language models link generated claims to source documents, evaluate citation accuracy with NLI, and measure attribution precision and recall.50 min 434Uncertainty QuantificationCovers confidence calibration, verbalized uncertainty, sampling-based uncertainty, uncertainty communication.57 min

Volume 19Responsible, Secure, and Interpretable Language AI3 parts · 24 chaptersBuy this volumeBuy PDF · $24

Part LVIII

Bias and Fairness

5 chapters

441Bias in Language ModelsCovers bias sources, bias types (demographic, cultural), bias in training data, bias amplification.53 min 442Bias MeasurementCovers embedding association tests, generation bias metrics, classification bias metrics, bias benchmarks.59 min 443Bias Mitigation: Debiasing, CDA, and Fair Fine-tuningPractical techniques for reducing demographic bias in language models: data balancing, embedding debiasing, adversarial training, and prompt-based interventions.56 min 444Fairness Metrics: Demographic Parity, Equalized Odds, and Trade-offsLearn the key mathematical definitions of algorithmic fairness, from demographic parity to equalized odds, and why satisfying all metrics simultaneously is impossible.61 min 445Representation HarmsCovers stereotyping, erasure, demeaning associations, measuring representation harms.54 min

Part LIX

Interpretability

11 chapters

446Interpretability GoalsCovers debugging, trust, safety, scientific understanding, interpretability approaches overview.46 min 447Attention VisualizationCovers attention weight extraction, attention head visualization, attention interpretation caveats, attention tools.51 min 448Attention Analysis LimitationsCovers attention vs importance, attention manipulation studies, gradient-based alternatives.47 min 449Probing ClassifiersCovers linear probing methodology, probing task design, probing interpretation, control tasks.52 min 450Probing LayersCovers layer selection, representation evolution, task localization, layer probing patterns.55 min 451Activation PatchingCovers patching methodology, locating information, patching experiments, causal tracing.49 min 452Logit LensCovers logit lens concept, intermediate vocabulary projection, tuned lens, lens interpretation.45 min 453Sparse AutoencodersCovers SAE architecture, sparsity constraints, dictionary learning, SAE for LLMs.59 min 454Feature InterpretationCovers feature activation patterns, feature naming, automated interpretation, feature circuits.57 min 455Mechanistic InterpretabilityReverse-engineer transformer networks into human-understandable algorithms by identifying circuits, induction heads, and mechanistic discoveries.52 min 456Activation Steering: Steering Vectors and Representation EngineeringLearn how steering vectors and activation addition let you modify language model behavior at inference time by injecting directional nudges into the residual stream.47 min

Part LX

Safety and Security

8 chapters

457Safety RisksCovers harmful content generation, misuse scenarios, unintended harms, safety threat models.52 min 458Red TeamingLearn how red teams systematically probe language models for safety failures, covering attack taxonomies, ASR metrics, RL-based automated attack generation, and real-world findings.61 min 459JailbreakingCovers jailbreak techniques, prompt injection, adversarial suffixes, jailbreak defenses.61 min 460Prompt InjectionCovers direct prompt injection, indirect prompt injection, injection in RAG, injection defenses.50 min 461Content FilteringCovers classification-based filtering, rule-based filtering, filter placement, filter evaluation.60 min 462GuardrailsCovers input guardrails, output guardrails, guardrail frameworks, guardrail design.61 min 463Memorization and Privacy in Language ModelsHow language models memorize training data, methods for measuring extractable memorization, PII risks in web-scale corpora, and practical privacy mitigations.56 min 464Differential PrivacyLearn how differential privacy protects training data in language models, from the mathematical guarantee to DP-SGD, privacy budgets, and practical LLM fine-tuning.50 min

Volume 20Applications, Production, and Compound AI Systems2 parts · 16 chaptersBuy this volumeBuy PDF · $24

Part LXIII

LLM Applications

7 chapters

478Text Generation ApplicationsCovers creative writing, content generation, text transformation, generation quality control.Read 479SummarizationCovers extractive vs abstractive, length control, faithfulness, multi-document summarization.Read 480Question AnsweringCovers open-domain QA, reading comprehension, knowledge-intensive QA, QA evaluation.Read 481Information ExtractionCovers named entity recognition, relation extraction, event extraction, structured output generation.Read 482Classification ApplicationsCovers sentiment analysis, intent detection, topic classification, zero-shot classification.Read 483Conversational AICovers dialogue management, context tracking, persona consistency, conversation evaluation.Read 484Creative ApplicationsCovers story generation, poetry, code creativity, creative constraints and control.Read

Part LXV

Production Systems

9 chapters

491Model ServingCovers serving frameworks, model loading, request handling, serving configuration.57 min 492Latency OptimizationCovers latency breakdown, batching latency, streaming responses, latency monitoring.52 min 493Throughput OptimizationCovers batch size tuning, GPU utilization, concurrent requests, throughput measurement.58 min 494Auto-scalingCovers scaling metrics, horizontal scaling, scale-up vs scale-out, scaling policies.54 min 495Model RoutingCovers model selection, A/B testing, model cascades, routing strategies.53 min 496CachingCovers prompt caching, semantic caching, cache invalidation, cache hit rates.52 min 497MonitoringCovers metrics collection, alerting, logging, dashboards.46 min 498Quality MonitoringCovers output quality metrics, drift detection, regression detection, quality alerts.55 min 499Cost ManagementCovers cost modeling, cost optimization, cost allocation, cost monitoring.40 min

AppendixState of Language AI 20252 parts · 14 chapters

Part LXVI

Frontier Methods of 2025

8 chapters

500Constitutional AICovers constitutional principles, critique and revision, CAI training, CAI effectiveness.55 min 501Process Reward ModelsCovers outcome vs process reward, PRM training, PRM for math, PRM limitations.52 min 502Test-Time ComputeCovers multiple sampling, iterative refinement, compute-optimal inference, scaling test-time compute.53 min 503Retrieval-Augmented TrainingCovers RETRO architecture, retrieval during training, retrieved context integration.40 min 504Long-Form GenerationCovers outline-based generation, hierarchical generation, coherence maintenance, long-form evaluation.47 min 505WatermarkingCovers watermarking schemes, statistical detection, watermark robustness, watermark evaluation.65 min 506Model CardsCovers model card contents, intended use documentation, limitation documentation, model card best practices.54 min 507Responsible DeploymentCovers release decisions, staged release, access control, deployment monitoring.57 min

Part LXVII

Frontier Outlook from 2025

6 chapters

508Scaling FrontiersCovers scaling limits, beyond power laws, data wall, architectural innovations for scale.40 min 509Efficiency FrontiersCovers efficient architectures, hardware co-design, inference efficiency, training efficiency advances.44 min 510Capability FrontiersCovers emerging capabilities, world models, planning and agency, multimodal reasoning advances.54 min 511Alignment ChallengesCovers scalable oversight, alignment tax, goal mis-specification, long-term alignment research.54 min 512Societal ImplicationsCovers labor market effects, access and equity, regulation landscape, governance frameworks.55 min 513Research DirectionsCovers open problems, promising research areas, benchmark gaps, community priorities.54 min

More about the book

Begin with the fundamentals that never go out of style: tokenization, embeddings, and the statistical foundations that inform modern approaches. Then dive deep into the transformer architecture. Learn not just how to use it, but how it actually works. Understand self-attention mathematically, grasp why positional encodings matter, and see how architectural choices like layer normalization affect training dynamics.

The landscape of pretrained models can be overwhelming. BERT, GPT, T5, LLaMA, Claude: what are the real differences? When should you use an encoder versus a decoder? How do you actually fine-tune a model for your use case? This handbook cuts through the confusion with clear explanations of each architecture family, their strengths and weaknesses, and practical guidance on selection and adaptation.

From retrieval-augmented generation to RLHF alignment, from efficient inference to safety considerations, every topic a modern NLP practitioner needs is covered with the depth that lets you actually apply it. Whether you're building production systems, conducting research, or simply want to truly understand the technology reshaping our world, this is your comprehensive guide to language AI.

Own the library

The whole growing library, yours forever.

Reading online is free, and always will be. The paid library unlocks every current volume as its own carefully typeset PDF, plus each future volume when it is published. There is intentionally no combined PDF.

All 393 written chapters across 19 available volumes
19 written volume PDFs plus the Volume 3 placeholder. Files are delivered separately.
All future expansions, updates, and errata, free, forever
Every future volume included, including 95 planned chapters
Downloadable editions for durable offline reading

Best value, save $257 (56%)

$199

one-time

instead of $456 for the 19 available volume PDFs bought separately

5.0 from 12 readers

Get all-volume access · $199 Start with a single volume · $24

Secure checkout via Stripe · Individual PDF links delivered to your inbox

Reader verdict

5.0

out of 5

12 five-star reader reviews

Genuine feedback, with names abbreviated for privacy.

Meet the community

“Really appreciate the way you have explained the concepts. Simple to understand, yet builds up knowledge exponentially, all while reinforcing it with examples repeatedly. I've read the last two chapters `74.Self-attention & 75.Q,K,V` and I've to admit, my concepts have never been clearer. You really are doing an amazing job breaking down maths and complex concepts like a story!”

Muhammad H.AI/ML Engineer

“I would like to thank you for your work. Specifically, the chapter: 'LIME Explainability: Complete Guide to Local Interpretable Model-Agnostic Explanations' was extremely helpful for me in understanding LIME and communicating to colleagues. Your work is inspiring and valuable. Thank you for doing it.”

Will G.Data Scientist

“I am enjoying reading your writings. I appreciate the effort you put into it. I want to drop a note of thanks. Thank you!”

Krishnan S.Researcher

“Your writing feels like a treasure trove of great material. So, thank you for compiling everything on your website!”

Griffin P.Quantitative Researcher

“Really appreciate the way you have explained the concepts. Simple to understand, yet builds up knowledge exponentially, all while reinforcing it with examples repeatedly. I've read the last two chapters `74.Self-attention & 75.Q,K,V` and I've to admit, my concepts have never been clearer. You really are doing an amazing job breaking down maths and complex concepts like a story!”

Muhammad H.AI/ML Engineer

“I would like to thank you for your work. Specifically, the chapter: 'LIME Explainability: Complete Guide to Local Interpretable Model-Agnostic Explanations' was extremely helpful for me in understanding LIME and communicating to colleagues. Your work is inspiring and valuable. Thank you for doing it.”

Will G.Data Scientist

“I am enjoying reading your writings. I appreciate the effort you put into it. I want to drop a note of thanks. Thank you!”

Krishnan S.Researcher

“Your writing feels like a treasure trove of great material. So, thank you for compiling everything on your website!”

Griffin P.Quantitative Researcher

“I found your book Language AI Handbook's explanations of Transformer architectures and production deployment strategies to be exceptionally clear and insightful”

San Z.Researcher

“Thanks so much for establishing such an amazing community. I appreciate the opportunity to be a learner and a contributor to the community.”

Cyrille L.AI/ML Engineer

“Thank you Michael. It's top!”

David X.Data Scientist

“I came across your collection of online handbooks covering quantitative finance, AI, and machine learning. I wanted to reach out and thank you for making such high-quality material openly available. They are incredibly helpful for my CFA Level II review and for brushing up ahead of ML interviews.”

Karim Z.Reader

“I found your book Language AI Handbook's explanations of Transformer architectures and production deployment strategies to be exceptionally clear and insightful”

San Z.Researcher

“Thanks so much for establishing such an amazing community. I appreciate the opportunity to be a learner and a contributor to the community.”

Cyrille L.AI/ML Engineer

“Thank you Michael. It's top!”

David X.Data Scientist

“I came across your collection of online handbooks covering quantitative finance, AI, and machine learning. I wanted to reach out and thank you for making such high-quality material openly available. They are incredibly helpful for my CFA Level II review and for brushing up ahead of ML interviews.”

Karim Z.Reader

“I came across one of your books, Language AI Handbook, and man, I'm loving it. I've read many books but have never found such a comprehensive book on large models that covers everything from start to finish with such depth.”

Uon L.Reader

“I just came across your website where your books are available for free to read. Thank you so much for making this available to the public. This will help a lot of people.”

Vikas C.AI/ML Engineer

“Thanks for putting cool stuff out into the world.”

Charlie B.Applied AI

“I read a few of your blogs on NLP and oh WOW, they really are something, now I exactly know how all of the terms Entropy, Cross-entropy and how it is connected to Perplexity, will definitely read more of your blogs.”

Kanishk K.Reader

“I came across one of your books, Language AI Handbook, and man, I'm loving it. I've read many books but have never found such a comprehensive book on large models that covers everything from start to finish with such depth.”

Uon L.Reader

“I just came across your website where your books are available for free to read. Thank you so much for making this available to the public. This will help a lot of people.”

Vikas C.AI/ML Engineer

“Thanks for putting cool stuff out into the world.”

Charlie B.Applied AI

“I read a few of your blogs on NLP and oh WOW, they really are something, now I exactly know how all of the terms Entropy, Cross-entropy and how it is connected to Perplexity, will definitely read more of your blogs.”

Kanishk K.Reader

Read all 12 reader reviews

“I would like to thank you for your work. Specifically, the chapter: 'LIME Explainability: Complete Guide to Local Interpretable Model-Agnostic Explanations' was extremely helpful for me in understanding LIME and communicating to colleagues. Your work is inspiring and valuable. Thank you for doing it.”

Will G.Data Scientist

“Really appreciate the way you have explained the concepts. Simple to understand, yet builds up knowledge exponentially, all while reinforcing it with examples repeatedly. I've read the last two chapters `74.Self-attention & 75.Q,K,V` and I've to admit, my concepts have never been clearer. You really are doing an amazing job breaking down maths and complex concepts like a story!”

Muhammad H.AI/ML Engineer

“Thanks so much for establishing such an amazing community. I appreciate the opportunity to be a learner and a contributor to the community.”

Cyrille L.AI/ML Engineer

“I just came across your website where your books are available for free to read. Thank you so much for making this available to the public. This will help a lot of people.”

Vikas C.AI/ML Engineer

“I am enjoying reading your writings. I appreciate the effort you put into it. I want to drop a note of thanks. Thank you!”

Krishnan S.Researcher

“Thank you Michael. It's top!”

David X.Data Scientist

“Thanks for putting cool stuff out into the world.”

Charlie B.Applied AI

“I found your book Language AI Handbook's explanations of Transformer architectures and production deployment strategies to be exceptionally clear and insightful”

San Z.Researcher

“Your writing feels like a treasure trove of great material. So, thank you for compiling everything on your website!”

Griffin P.Quantitative Researcher

“I came across your collection of online handbooks covering quantitative finance, AI, and machine learning. I wanted to reach out and thank you for making such high-quality material openly available. They are incredibly helpful for my CFA Level II review and for brushing up ahead of ML interviews.”

Karim Z.Reader

“I read a few of your blogs on NLP and oh WOW, they really are something, now I exactly know how all of the terms Entropy, Cross-entropy and how it is connected to Perplexity, will definitely read more of your blogs.”

Kanishk K.Reader

“I came across one of your books, Language AI Handbook, and man, I'm loving it. I've read many books but have never found such a comprehensive book on large models that covers everything from start to finish with such depth.”

Uon L.Reader

Author and edition details

About the author and this edition

Michael Brenndoerfer

Michael has spent more than a decade working across software engineering, data, AI, and business. He writes to understand difficult ideas more deeply and to share what he learns in a clear, practical way.

Report an error or suggest a correction

Edition: First edition
Published: 2025
Last reviewed: July 31, 2026

Version history

Kept current, not frozen in time

Library access includes every future volume and update. When a volume changes, its updated PDF appears in My books at no extra cost.

Current release

Aug 4, 2026

Edition 2026.08.0

Initial volume-library release with 19 written PDFs, a Volume 3 placeholder, and no combined edition.

Cite this book

Reference

Citation details

Cite or share this article.

BIBTEXAcademic

@book{brenndoerfer2025languageaihandbook, author = {Michael Brenndoerfer}, title = {Language AI Handbook}, year = {2025}, url = {https://mbrenndoerfer.com/books/language-ai-handbook}, publisher = {mbrenndoerfer.com}, note = {Accessed: 2026-11-13} }

APAAcademic

Michael Brenndoerfer (2025). Language AI Handbook. Retrieved from https://mbrenndoerfer.com/books/language-ai-handbook

MLAAcademic

Michael Brenndoerfer. "Language AI Handbook." 2026. Web. November 13, 2026. <https://mbrenndoerfer.com/books/language-ai-handbook>.

CHICAGOAcademic

Michael Brenndoerfer. "Language AI Handbook." Accessed November 13, 2026. https://mbrenndoerfer.com/books/language-ai-handbook.

HARVARDAcademic

Michael Brenndoerfer (2025) 'Language AI Handbook'. Available at: https://mbrenndoerfer.com/books/language-ai-handbook (Accessed: November 13, 2026).

SimpleBasic

Michael Brenndoerfer (2025). Language AI Handbook. https://mbrenndoerfer.com/books/language-ai-handbook

DIRECT LINKURL

https://mbrenndoerfer.com/books/language-ai-handbook

Language AI Handbook

Read Language AI Handbook free online

More about the book

The whole growing library, yours forever.

12 five-star reader reviews

About the author and this edition

Michael Brenndoerfer

Kept current, not frozen in time

Reference

Stay up to date