Language AI Handbook

Covers ASCII origins and 7-bit limitations, Unicode code points and planes, UTF-8 variable-width encoding scheme, byte order marks and endianness, encoding detection heuristics, common encoding errors and mojibake, practical encoding/decoding in Python.

Covers Unicode normalization forms (NFC, NFD, NFKC, NFKD), case folding vs lowercasing, accent and diacritic handling, whitespace normalization, ligature expansion, full-width to half-width conversion, implementing a normalization pipeline.

Covers regex syntax and metacharacters, character classes and quantifiers, grouping and backreferences, lookahead and lookbehind assertions, greedy vs lazy matching, common NLP patterns (emails, URLs, dates), regex performance considerations.

Covers period disambiguation challenges, abbreviation handling, rule-based boundary detection, Punkt sentence tokenizer algorithm, evaluation metrics for segmentation, handling edge cases (quotes, parentheses, lists), multilingual segmentation issues.

Covers whitespace tokenization limitations, punctuation handling rules, contractions and clitics, language-specific challenges (Chinese, Japanese, German compounds), Penn Treebank tokenization standard, building a rule-based tokenizer, tokenization evaluation.

Covers document-term matrix construction, vocabulary building from corpus, word counting and frequency vectors, sparse matrix representation (CSR/CSC formats), vocabulary pruning (min_df, max_df), binary vs count representations, limitations of word order loss.

Covers bigram and trigram extraction, n-gram vocabulary explosion, n-gram frequency distributions, Zipf's law in n-grams, character n-grams for robustness, skip-grams and flexible windows, n-gram indexing for search.

Covers Markov assumption and chain rule, maximum likelihood estimation, probability calculation for sequences, handling unseen n-grams, start and end tokens, generating text from n-gram models, model storage and lookup efficiency.

Covers add-one (Laplace) smoothing, add-k smoothing and tuning, Good-Turing smoothing derivation, Kneser-Ney smoothing intuition and formula, interpolation vs backoff, modified Kneser-Ney, comparing smoothing methods empirically.

Covers cross-entropy definition and derivation, perplexity as branching factor, relationship to bits-per-character, held-out evaluation methodology, perplexity vs downstream performance, comparing models with perplexity, perplexity limitations and caveats.

Covers raw term frequency, log-scaled term frequency, boolean term frequency, augmented term frequency, L2-normalized frequency vectors, term frequency sparsity patterns, efficient term frequency computation.

Covers document frequency calculation, IDF formula derivation, IDF intuition (rare words matter more), smoothed IDF variants, IDF across corpus splits, relationship to information theory, implementing IDF efficiently.

Covers TF-IDF formula and variants, TF-IDF vector computation, TF-IDF normalization options, BM25 as TF-IDF extension, document similarity with TF-IDF, TF-IDF for feature extraction, sklearn TfidfVectorizer deep dive.

Covers BM25 derivation from probabilistic IR, saturation parameter k1, length normalization parameter b, BM25+ and BM25L variants, field-weighted BM25, implementing BM25 scoring, BM25 vs TF-IDF empirically.

Covers Firth's "you shall know a word by the company it keeps," distributional similarity intuition, context window definitions, paradigmatic vs syntagmatic relations, word similarity from distributions, limitations of distributional semantics.

Covers word-word co-occurrence matrices, word-document matrices, context window size effects, weighting by distance, symmetric vs directional contexts, matrix sparsity patterns, efficient construction algorithms.

Covers PMI formula derivation, PMI interpretation as association, positive PMI (PPMI), shifted PPMI variants, PMI matrix properties, PMI vs raw counts comparison, PMI for collocation extraction.

Covers SVD mathematical formulation, truncated SVD for dimensionality reduction, LSA (Latent Semantic Analysis), choosing embedding dimensions, SVD computational complexity, randomized SVD for scale, interpreting SVD dimensions.

Covers skip-gram architecture diagram, input/output representations, softmax over vocabulary, skip-gram objective function, training data generation, window size hyperparameter, skip-gram vs CBOW intuition.

Covers CBOW architecture, context word averaging, CBOW objective function, CBOW vs skip-gram training speed, CBOW for frequent words, implementing CBOW forward pass, CBOW gradient derivation.

Covers softmax computational bottleneck, negative sampling objective derivation, sampling distribution (unigram^0.75), number of negatives hyperparameter, negative sampling gradient computation, NCE vs negative sampling, implementing efficient sampling.

Covers binary tree construction (Huffman coding), path probability computation, hierarchical softmax objective, gradient computation along paths, tree structure impact on learning, hierarchical softmax vs negative sampling, when to use each approach.

Covers data preprocessing pipeline, subsampling frequent words, learning rate scheduling, minibatch vs online training, convergence monitoring, gensim Word2Vec usage, training from scratch in PyTorch.

Covers vector arithmetic for analogies, parallelogram model, analogy evaluation datasets, 3CosAdd vs 3CosMul methods, analogy accuracy metrics, limitations of analogy evaluation, what analogies reveal about embeddings.

Covers GloVe objective function derivation, weighted least squares formulation, relationship to matrix factorization, weighting function design, bias terms in GloVe, GloVe vs Word2Vec comparison, training GloVe efficiently.

Covers character n-gram representation, word vector as n-gram sum, FastText architecture, handling OOV words, morphological awareness, FastText for morphologically rich languages, training FastText models.

Covers intrinsic vs extrinsic evaluation, word similarity datasets (SimLex, WordSim), analogy accuracy, embedding visualization (t-SNE, UMAP), downstream task evaluation, embedding bias detection, evaluation pitfalls.

Covers OOV word problem, vocabulary size explosion, rare word representation, morphological productivity, compound words, code and technical text, the case for subword units.

Covers BPE algorithm step-by-step, merge rules learning, vocabulary size control, BPE encoding procedure, BPE decoding procedure, BPE implementation from scratch, BPE hyperparameters.

Covers WordPiece vs BPE differences, likelihood objective for merges, greedy tokenization algorithm, ## prefix notation, WordPiece in BERT, training WordPiece tokenizers, handling unknown characters.

Covers unigram LM formulation, EM algorithm for training, Viterbi decoding for tokenization, sampling multiple segmentations, subword regularization, unigram vs BPE comparison, SentencePiece unigram mode.

Covers treating text as raw bytes, whitespace handling (▁ prefix), BPE and unigram modes, training from raw text, pretokenization elimination, SentencePiece in production, multilingual tokenization.

Covers corpus preparation, vocabulary size selection, special tokens configuration, training with HuggingFace tokenizers, saving and loading tokenizers, tokenizer versioning, domain-specific tokenizers.

Covers [CLS], [SEP], [PAD], [MASK], [UNK] tokens, beginning/end of sequence tokens, custom special tokens, special token embeddings, token type IDs, handling special tokens in generation.

Covers number tokenization issues, code tokenization, multilingual text mixing, emoji and Unicode edge cases, tokenization artifacts, adversarial tokenization, measuring tokenization quality.

Covers POS tag sets (Penn Treebank, Universal), POS tagging as classification, contextual disambiguation, POS tagging accuracy metrics, POS tagging for downstream tasks, rule-based vs statistical taggers.

Covers entity types (PER, ORG, LOC, etc.), NER as sequence labeling, nested entity challenges, entity boundary detection, NER evaluation (exact vs partial match), NER datasets and benchmarks.

Covers BIO scheme explanation, BIOES/BILOU variants, converting spans to BIO tags, BIO decoding to spans, handling tagging inconsistencies, BIO for multi-label scenarios, implementing BIO utilities.

Covers noun phrase chunking, chunk types (NP, VP, PP), IOB tagging for chunks, chunking vs full parsing, chunking evaluation, chunking as preprocessing, regex chunking with NLTK.

Covers HMM components (states, observations, transitions), emission and transition probabilities, HMM assumptions (Markov, independence), HMM for POS tagging, HMM parameter estimation, HMM limitations for NLP.

Covers optimal path problem formulation, Viterbi recursion derivation, backpointer tracking, Viterbi complexity analysis, log-space computation, implementing Viterbi efficiently, Viterbi for beam search foundation.

Covers CRF vs HMM comparison, CRF feature functions, log-linear formulation, partition function computation, CRF for NER, CRF inference complexity, neural CRF layers.

Covers CRF log-likelihood objective, forward-backward algorithm, gradient computation, L-BFGS optimization, feature template design, CRF regularization, CRF training convergence.

Morphemes, inflection, derivation, compounding, and why word structure matters for language models.

Finite-state and neural approaches to analyzing and generating morphologically rich language.

Covers linear decision boundaries, weight vectors and bias, dot product interpretation, multiclass classification (softmax), linear classifier limitations, training with gradient descent.

Covers sigmoid function and saturation, tanh properties, ReLU and dying ReLU, Leaky ReLU and PReLU, ELU and SELU, GELU derivation and properties, Swish and Mish, choosing activation functions.

Covers hidden layers and depth, weight matrices between layers, forward pass computation, representational capacity, MLP for classification, MLP for regression, MLP architecture design.

Covers cross-entropy loss derivation, MSE for regression, binary vs multiclass cross-entropy, label smoothing, focal loss for imbalance, loss function numerical stability, custom loss functions.

Covers computational graphs, chain rule review, forward and backward pass, gradient accumulation, backprop complexity analysis, automatic differentiation, implementing backprop from scratch.

Covers batch vs stochastic gradient descent, minibatch gradient descent, learning rate selection, SGD convergence properties, SGD noise as regularization, learning rate schedules basics, SGD implementation.

Covers momentum intuition (ball rolling), momentum update equations, momentum coefficient selection, dampening oscillations, momentum vs vanilla SGD, Nesterov momentum derivation, implementing momentum.

Covers exponential moving averages, first moment (mean) estimation, second moment (variance) estimation, bias correction derivation, Adam update rule, Adam hyperparameters, Adam convergence properties.

Covers L2 regularization vs weight decay, why they differ with Adam, AdamW formulation, weight decay coefficient selection, AdamW as default optimizer, AdamW vs Adam empirically.

Covers random initialization importance, Xavier/Glorot initialization derivation, He initialization for ReLU, initialization for different activations, layer-wise initialization, initialization debugging, modern initialization practices.

Covers internal covariate shift, batch statistics computation, learnable scale and shift, training vs inference mode, batch norm gradient flow, batch norm placement debates, batch norm limitations.

Covers dropout as ensemble, dropout mask sampling, inverted dropout scaling, dropout rate selection, dropout at inference, spatial dropout for sequences, dropout in modern architectures.

Covers gradient explosion detection, clip by value, clip by global norm, gradient clipping implementation, when to use gradient clipping, clipping threshold selection, monitoring gradient norms.

Covers recurrent connection intuition, hidden state as memory, unrolled computation graph, parameter sharing across time, RNN for sequence classification, RNN for sequence generation, RNN equations and dimensions.

Covers BPTT derivation, gradient flow through time, truncated BPTT, BPTT memory requirements, BPTT implementation, gradient accumulation across timesteps.

Covers gradient product across timesteps, vanishing gradient analysis, long-range dependency failure, gradient visualization, vanishing vs exploding trade-off, architectural solutions overview.

Covers cell state as information highway, gate mechanism intuition, LSTM diagram walkthrough, information flow in LSTMs, LSTM for long sequences, LSTM memory capacity.

Covers forget gate equations, input gate equations, cell state update, output gate equations, hidden state computation, LSTM parameter count, implementing LSTM from scratch.

Covers constant error carousel, forget gate gradient highway, gradient flow analysis, LSTM vs vanilla RNN gradients, peephole connections, LSTM gradient clipping needs.

Covers GRU vs LSTM comparison, reset gate function, update gate function, candidate hidden state, GRU equations, GRU parameter efficiency, when to choose GRU vs LSTM.

Covers forward and backward passes, hidden state concatenation, bidirectional architectures, bidirectionality for classification, limitations for generation, implementing bidirectional RNNs.

Covers multiple RNN layers, residual connections for depth, layer normalization in RNNs, depth vs width trade-offs, gradient flow in deep RNNs, practical depth limits.

Covers encoder role and design, decoder role and design, context vector as bottleneck, seq2seq for machine translation, seq2seq for summarization, seq2seq training setup.

Covers teacher forcing procedure, exposure bias problem, teacher forcing efficiency, scheduled sampling, curriculum learning, teacher forcing vs autoregressive training.

Covers greedy decoding limitations, beam search algorithm, beam width selection, length normalization, diverse beam search, beam search implementation, beam search vs sampling.

Covers attention as soft lookup, attention weight interpretation, attention for variable-length inputs, attention visualization, attention vs pooling, attention computation overview.

Covers alignment model formulation, score function (additive), attention weight computation, context vector as weighted sum, attention in decoder, Bahdanau attention implementation.

Covers dot product attention, general (bilinear) attention, concat attention variant, global vs local attention, Luong vs Bahdanau comparison, attention placement (input vs output).

Covers pointer network motivation, copy probability computation, mixing generation and copying, pointer-generator networks, copy mechanism for summarization, OOV handling with copy.

Covers cross-attention vs self-attention, self-attention motivation, all-pairs interaction, self-attention for representation learning, self-attention computational pattern.

Covers QKV intuition (database lookup), projection matrices Wq, Wk, Wv, query-key matching, value retrieval, QKV dimensions and shapes, QKV as learned transformations.

Covers dot product for similarity, softmax for normalization, scaling factor derivation (1/√dk), attention output computation, attention in matrix form, attention implementation.

Covers padding masks, causal (look-ahead) masks, combining multiple masks, mask shapes and broadcasting, efficient masking implementation, custom attention patterns.

Covers multiple attention heads motivation, head dimension splitting, parallel attention computation, output concatenation and projection, head specialization, multi-head vs single head.

Covers O(n²) attention complexity, memory requirements, attention bottleneck in long sequences, FLOPs computation, attention vs RNN complexity, practical scaling limits.

Covers transformer position blindness, why position matters for language, position information requirements, position encoding vs position embedding, absolute vs relative position.

Covers sinusoidal formula derivation, wavelength intuition, position encoding visualization, extrapolation properties, sinusoidal encoding implementation, learned vs sinusoidal trade-offs.

Covers position embedding table, position embedding training, maximum sequence length, learned embedding extrapolation, position embedding analysis, GPT-style position embeddings.

Covers relative position motivation, relative attention formulation, clipping relative positions, relative position in self-attention, Shaw et al. relative positions, relative bias implementation.

Covers RoPE intuition, rotation matrix formulation, RoPE in complex numbers, relative position through rotation, RoPE implementation, RoPE frequency patterns.

Covers ALiBi motivation, linear bias by distance, head-specific slopes, ALiBi extrapolation properties, ALiBi simplicity advantages, ALiBi vs RoPE comparison.

Covers extrapolation benchmarks, training efficiency comparison, implementation complexity, position encoding for long context, hybrid approaches, current best practices.

Covers residual connection formulation, gradient highway interpretation, residual scaling, residual connections in transformers, pre-norm vs post-norm residuals.

Covers layer norm vs batch norm, layer norm formula, learnable affine parameters, layer norm placement, layer norm gradient flow, layer norm implementation.

Covers RMSNorm derivation, removing mean centering, RMSNorm efficiency, RMSNorm vs LayerNorm performance, RMSNorm in modern architectures.

Covers original transformer (post-norm), pre-norm formulation, training stability comparison, gradient flow differences, when to use each, modern consensus.

Covers FFN architecture, hidden dimension expansion, FFN as two linear layers, position independence, FFN parameter count, FFN computational cost.

Covers ReLU in original transformer, GELU adoption, GELU approximations, SiLU/Swish in modern models, activation function comparison.

Covers GLU formulation, gating mechanism, SwiGLU derivation, GeGLU variant, GLU parameter efficiency, GLU in modern architectures.

Covers standard block structure, component ordering, block implementation, block initialization, forward pass walkthrough, block hyperparameters.

Covers encoder-only design, bidirectional self-attention, encoder for understanding tasks, encoder output usage, BERT-style encoder, encoder layer stacking.

Covers decoder-only design, causal masking requirement, autoregressive generation, decoder for generation tasks, GPT-style decoder, decoder layer stacking.

Covers encoder-decoder interaction, cross-attention mechanism, encoder-decoder for seq2seq, T5-style architecture, information flow, when to use encoder-decoder.

Covers cross-attention formulation, KV from encoder, Q from decoder, cross-attention masking, cross-attention placement, cross-attention implementation.

Covers input-output embedding tying, encoder-decoder tying, parameter reduction, weight tying effects on training, when to tie weights.

Covers depth vs width trade-offs, number of heads selection, hidden dimension ratios, FFN expansion ratio, total parameter calculation, architecture search.

Covers O(n²) memory analysis, O(n²) compute analysis, attention matrix size, practical sequence limits, bottleneck visualization, motivation for efficiency.

Covers local attention windows, strided attention patterns, block-sparse attention, combining sparse patterns, sparse attention implementation.

Covers sliding window formulation, window size selection, dilated sliding windows, sliding window for long sequences, Mistral-style windowed attention.

Covers CLS token global attention, learned global tokens, global-local attention mixing, global token count, implementation strategies.

Covers Longformer attention pattern, global attention configuration, Longformer complexity, Longformer for documents, Longformer implementation.

Covers BigBird attention pattern, random attention benefits, BigBird theoretical guarantees, BigBird vs Longformer, BigBird applications.

Covers softmax attention reformulation, kernel feature maps, linear complexity attention, linear attention limitations, Performer and variants.

Covers GPU memory hierarchy, tiling for SRAM, online softmax computation, recomputation strategy, FlashAttention complexity, FlashAttention benefits.

Covers CUDA kernel basics, memory access patterns, FlashAttention-2 improvements, using FlashAttention in PyTorch, FlashAttention limitations.

Covers training sequence length limits, attention memory scaling, position encoding extrapolation, long-range dependency learning, evaluation challenges.

Covers linear position scaling, interpolation vs extrapolation, position interpolation implementation, fine-tuning for longer context, interpolation limitations.

Covers RoPE frequency analysis, high-frequency preservation, NTK-aware formula, dynamic NTK scaling, NTK vs linear interpolation.

Covers YaRN motivation, attention scaling factor, YaRN formula, YaRN training requirements, YaRN vs alternatives.

Covers attention sink phenomenon, StreamingLLM approach, sink token design, streaming inference, infinite context generation.

Covers memory network concepts, memory retrieval mechanisms, memory writing and updating, memory-augmented transformers, Memorizing Transformers.

Covers Transformer-XL approach, segment-level processing, recurrent state passing, relative position in recurrence, recurrent memory limitations.

Covers Common Crawl, crawling strategies, robots.txt respect, crawl freshness.

Covers HTML parsing, boilerplate removal, content extraction, trafilatura and similar tools.

Covers language ID models, multilingual document handling, code-switching, language filtering.

Covers exact deduplication, near-duplicate detection, document vs substring dedup, dedup at scale.

Covers MinHash algorithm, Jaccard similarity estimation, MinHash LSH, MinHash implementation.

Covers heuristic filters, perplexity filtering, classifier-based filtering, filter thresholds.

Covers toxicity classifiers, toxicity thresholds, over-filtering risks, toxicity filter evaluation.

Covers PII detection methods, PII removal strategies, PII removal evaluation, privacy preservation.

Covers domain proportions, quality weighting, data mixing experiments, optimal mixing.

Covers synthetic data generation, quality verification, synthetic data diversity, distillation.

Covers CLM objective formulation, autoregressive factorization, CLM loss computation, CLM for generation, CLM training data, CLM scaling properties.

Covers MLM objective formulation, masking strategies (15% rule), [MASK] token usage, MLM for understanding, MLM training dynamics.

Covers subword masking problems, whole word masking procedure, WWM implementation, WWM vs random masking, WWM for different tokenizers.

Covers span selection strategies, span length distribution, sentinel tokens, T5-style corruption, span corruption benefits.

Covers prefix LM formulation, prefix LM attention pattern, prefix LM for generation, prefix LM training, UniLM-style objectives.

Covers generator-discriminator setup, replaced vs original detection, RTD efficiency advantages, ELECTRA training procedure, RTD vs MLM comparison.

Covers token deletion, token shuffling, sentence permutation, document rotation, BART-style denoising, combining denoising tasks.

Covers power law definition, log-log linear relationships, power law fitting, power law universality, power law intuition.

Covers loss vs parameters, loss vs data, loss vs compute, Kaplan optimal allocation, Kaplan predictions.

Covers Chinchilla experiments, revised scaling coefficients, optimal tokens per parameter, Chinchilla vs Kaplan, Chinchilla implications.

Covers compute budget allocation, tokens vs parameters ratio, training efficiency, compute-optimal recipes, practical guidelines.

Covers data repetition effects, optimal repetition strategies, data augmentation scaling, synthetic data scaling.

Covers training vs inference compute, inference-optimal models, over-training for efficiency, deployment cost modeling.

Covers loss extrapolation, capability prediction, scaling law uncertainty, prediction reliability, practical forecasting.

Covers BERT model sizes, BERT layer configuration, BERT embedding layers, BERT attention patterns, BERT output representations.

Covers pre-training data preparation, MLM implementation, NSP task design, pre-training hyperparameters, pre-training duration.

Covers classification fine-tuning, sequence labeling fine-tuning, question answering fine-tuning, fine-tuning hyperparameters, catastrophic forgetting.

Covers [CLS] token usage, layer selection strategies, pooling strategies, BERT as feature extractor, frozen vs fine-tuned representations.

Covers dynamic masking, NSP removal, larger batches, more data, RoBERTa training recipe, RoBERTa vs BERT performance.

Covers factorized embeddings, cross-layer parameter sharing, sentence order prediction, ALBERT efficiency, ALBERT performance trade-offs.

Covers generator training, discriminator training, RTD objective, ELECTRA sample efficiency, ELECTRA scaling, ELECTRA fine-tuning.

Covers disentangled attention formulation, enhanced mask decoder, DeBERTa position encoding, DeBERTa improvements, DeBERTa-v3 advances.

Covers T5 encoder-decoder design, T5 attention patterns, T5 model sizes, T5 relative positions, T5 implementation.

Covers span corruption procedure, sentinel tokens, corruption rate, T5 pre-training data, T5 training scale.

Covers task prefixes, classification as generation, NER as generation, QA as generation, task formatting examples.

Covers BART encoder-decoder, BART attention configuration, BART vs T5 comparison, BART model sizes.

Covers token masking, token deletion, text infilling, sentence permutation, document rotation, objective combinations.

Covers mT5 training data, language sampling, cross-lingual transfer, mT5 vs T5 performance, multilingual tokenization.

Covers GPT-1 architecture, GPT-1 pre-training, GPT-1 fine-tuning approach, GPT-1 transfer learning, GPT-1 historical significance.

Covers GPT-2 model sizes, GPT-2 architectural changes, zero-shot task performance, GPT-2 training data (WebText), GPT-2 generation quality.

Covers GPT-3 scale (175B), few-shot prompting discovery, in-context learning analysis, GPT-3 capabilities, GPT-3 limitations.

Covers ICL phenomenon, ICL vs fine-tuning, example selection strategies, ICL scaling behavior, ICL theoretical understanding.

Covers generation procedure, KV caching for efficiency, generation stopping criteria, generation speed optimization, generation code implementation.

Covers temperature scaling, temperature effects on distribution, temperature selection guidelines, temperature vs quality trade-off.

Covers top-k truncation, k selection strategies, top-k limitations, top-k implementation, combining with temperature.

Covers top-p formulation, cumulative probability threshold, nucleus sampling benefits, p selection guidelines, nucleus vs top-k.

Covers repetition in generation, repetition penalty formulation, frequency penalty, presence penalty, n-gram blocking.

Covers grammar-guided generation, JSON schema constraints, regex constraints, constrained beam search, constrained sampling.

Covers LLaMA design philosophy, LLaMA architectural choices, LLaMA training data, LLaMA efficiency, LLaMA significance.

Covers pre-norm with RMSNorm, SwiGLU FFN, RoPE implementation, component interactions, implementation details.

Covers GQA motivation, GQA formulation, KV head grouping, GQA memory savings, GQA vs MHA performance, GQA implementation.

Covers MQA extreme sharing, MQA memory benefits, MQA quality trade-offs, MQA for inference, MQA vs GQA.

Covers Mistral design choices, sliding window attention, Mistral efficiency, Mistral performance, Mistral vs LLaMA.

Covers Qwen architectural choices, Qwen training approach, Qwen multilingual capabilities, Qwen variants.

Covers Phi design philosophy, textbook-quality data, Phi training approach, Phi efficiency, small model capabilities.

Covers emergence definition, phase transitions, emergence examples, emergence mechanisms, emergence debate.

Covers ICL emergence curves, ICL vs fine-tuning scaling, ICL mechanism hypotheses, ICL as meta-learning.

Covers CoT emergence observations, CoT elicitation, CoT scaling behavior, CoT mechanism theories.

Covers discontinuous metrics, accuracy threshold effects, smooth underlying capabilities, re-examining emergence claims.

Covers inverse scaling phenomena, distractor tasks, sycophancy scaling, inverse scaling prize findings.

Covers grokking phenomenon, grokking in arithmetic, grokking mechanism theories, grokking phase transitions, practical implications.

Covers GPU memory hierarchy, CUDA cores, tensor cores, GPU specifications.

Covers memory breakdown (activations, parameters, gradients, optimizer states), memory estimation, OOM debugging.

Covers DDP algorithm, gradient synchronization, all-reduce operations, DDP scaling.

Covers column parallelism, row parallelism, communication patterns, Megatron-style parallelism.

Covers pipeline stages, micro-batching, pipeline bubbles, pipeline schedules (GPipe, 1F1B).

Covers ZeRO stage 1 (optimizer state partitioning), ZeRO stage 2 (gradient partitioning), ZeRO stage 3 (parameter partitioning), ZeRO memory savings.

Covers FSDP concepts, FSDP vs ZeRO, FSDP sharding strategies, FSDP usage.

Covers checkpointing concept, checkpoint selection, checkpointing overhead, selective checkpointing.

Covers floating point formats, loss scaling, BF16 advantages, mixed precision implementation.

Covers gradient compression, communication overlap, topology-aware communication, NCCL optimization.

Covers checkpoint contents, checkpoint frequency, async checkpointing, fault recovery.

Covers warmup motivation, linear warmup, warmup duration, warmup for large batches.

Covers step decay, exponential decay, inverse square root decay, decay scheduling.

Covers cosine decay formula, cosine with restarts, cosine schedule parameters, cosine vs linear.

Covers batch size effects, learning rate scaling, batch size limits, LAMB optimizer.

Covers weight decay formula, decoupled weight decay, weight decay selection, weight decay interaction with Adam.

Covers accumulation procedure, accumulation steps, accumulation for memory, accumulation correctness.

Covers loss spikes, gradient norm monitoring, stability techniques, training stability debugging.

Covers hyperparameter search, hyperparameter transfer, critical vs robust hyperparameters, default recipes.

Covers dense vs sparse trade-offs, conditional computation motivation, sparse model efficiency, sparse model challenges.

Covers expert architecture, expert as FFN, expert capacity, expert count selection, expert placement in transformer.

Covers router architecture, routing score computation, router training, router learned behavior.

Covers top-1 routing, top-2 routing, k selection trade-offs, routing implementation, combining expert outputs.

Covers expert utilization imbalance, collapse failure mode, load metrics, balanced routing importance.

Covers load balancing loss formulation, loss coefficient tuning, balancing vs task loss, auxiliary loss implementation.

Covers router instability, z-loss formulation, z-loss benefits, z-loss coefficient, combined auxiliary losses.

Covers expert placement strategies, all-to-all communication, communication overhead, expert parallelism implementation.

Covers Switch Transformer design, top-1 routing choice, capacity factor, Switch scaling results.

Covers Mixtral architecture, Mixtral expert design, Mixtral performance, Mixtral efficiency, Mixtral vs dense models.

Covers transfer learning paradigm, pre-training/fine-tuning split, what transfers, transfer learning efficiency.

Covers full fine-tuning procedure, fine-tuning hyperparameters, learning rate selection, batch size effects.

Covers forgetting phenomenon, forgetting measurement, forgetting mitigation, pre-trained capability preservation.

Covers discriminative fine-tuning, layer-wise learning rates, warmup for fine-tuning, learning rate decay.

Covers few-shot fine-tuning, data augmentation, sample efficiency patterns, small data strategies.

Covers parameter storage costs, multi-task deployment, PEFT efficiency, PEFT quality trade-offs.

Covers weight update decomposition, low-rank assumption, LoRA efficiency gains, LoRA flexibility.

Covers LoRA formulation W + BA, rank selection, initialization scheme, LoRA gradient computation.

Covers LoRA module design, merging weights, LoRA training loop, LoRA in PyTorch, HuggingFace PEFT usage.

Covers rank selection guidelines, alpha/rank ratio, which layers to adapt, LoRA dropout.

Covers 4-bit quantization for base model, NF4 data type, double quantization, QLoRA memory savings.

Covers importance-based pruning, SVD-based adaptation, dynamic rank, AdaLoRA training procedure.

Covers IA3 formulation, learned rescaling vectors, IA3 parameter efficiency, IA3 vs LoRA.

Covers prefix tuning formulation, prefix length selection, prefix tuning for generation, prefix vs LoRA.

Covers prompt tuning formulation, prompt initialization, prompt tuning scaling, prompt length effects.

Covers adapter architecture, adapter placement, adapter dimensionality, adapter fusion.

Covers performance comparison, parameter efficiency comparison, task suitability, practical recommendations.

Covers instruction tuning motivation, instruction format design, instruction diversity, instruction quality.

Covers human annotation, template-based generation, seed task expansion, quality filtering.

Covers self-instruct procedure, instruction generation, response generation, filtering strategies.

Covers prompt templates, system messages, multi-turn format, chat templates, role definitions.

Covers instruction tuning data mixing, training hyperparameters, loss masking, multi-task learning.

Covers instruction following benchmarks, human evaluation, automatic evaluation, instruction difficulty.

Covers alignment definition, helpfulness vs harmlessness, alignment challenges, alignment approaches overview.

Covers preference collection UI, comparison design, annotator guidelines, preference data quality.

Covers pairwise comparison model, preference probability, Bradley-Terry likelihood, preference strength.

Covers reward model architecture, preference loss function, reward model training, reward model evaluation.

Covers reward hacking examples, distribution shift, over-optimization, reward hacking mitigation.

Covers policy definition, REINFORCE algorithm, policy gradient derivation, variance reduction.

Covers clipped objective, PPO derivation, trust region intuition, PPO implementation.

Covers LLM as policy, action space (tokens), reward assignment, KL penalty importance.

Covers SFT stage, reward model training, PPO fine-tuning, RLHF hyperparameters, RLHF debugging.

Covers KL penalty motivation, KL coefficient selection, adaptive KL, KL effects on training.

Covers DPO motivation, removing reward model, DPO intuition, DPO benefits.

Covers DPO from RLHF objective, optimal policy derivation, DPO loss function, DPO as classification.

Covers DPO data format, DPO loss computation, DPO training procedure, DPO hyperparameters.

Covers IPO formulation, KTO for unpaired feedback, ORPO, cDPO, comparing alignment methods.

Covers AI as annotator, constitutional AI principles, AI preference generation, RLAIF scalability.

Covers iterative DPO, online preference learning, self-improvement loops, alignment stability.

Covers reasoning types, reasoning in LLMs, reasoning failure modes, reasoning evaluation.

Covers CoT prompting, zero-shot CoT, CoT fine-tuning, CoT limitations.

Covers self-consistency, tree of thought, least-to-most prompting, decomposition strategies.

Covers step verification, process reward models, verification-guided search, self-correction.

Covers math problem solving, symbolic integration, math benchmarks, math reasoning training.

Covers systematic failures, spurious correlations, reasoning shortcuts, robustness challenges.

Covers o1-style reasoning, test-time compute scaling, reasoning-capable models, open research questions.

Covers distillation objective, temperature in distillation, teacher selection, distillation for LLMs.

Covers feature distillation, attention transfer, progressive distillation, on-policy distillation.

Covers weight pruning, structured vs unstructured, pruning criteria, pruning schedule.

Covers head pruning, layer pruning, width pruning, structured pruning implementation.

Covers weight averaging, task arithmetic, TIES merging, DARE merging.

Covers multi-task merging, style merging, capability composition, merging evaluation.

Covers KV cache motivation, cache structure, cache memory requirements, cache management.

Covers cache size calculation, batch size effects, sequence length effects, memory bottleneck.

Covers memory fragmentation problem, page-based allocation, vLLM approach, paged attention benefits.

Covers cache eviction strategies, attention sink preservation, H2O algorithm, cache quantization.

Covers quantization fundamentals, per-tensor vs per-channel, symmetric vs asymmetric, calibration.

Covers INT8 range mapping, absmax quantization, smooth quantization, INT8 accuracy.

Covers 4-bit challenges, group-wise quantization, 4-bit accuracy trade-offs, 4-bit formats.

Covers GPTQ algorithm, layer-wise quantization, Hessian approximation, GPTQ implementation.

Covers salient weight preservation, AWQ algorithm, AWQ vs GPTQ, AWQ benefits.

Covers GGML/GGUF history, quantization types, GGUF file format, llama.cpp integration.

Covers speculative decoding concept, draft model selection, verification procedure, acceptance rate.

Covers acceptance criterion, expected speedup, draft quality effects, optimal draft length.

Covers static vs continuous batching, iteration-level scheduling, request completion handling, throughput gains.

Master LLM inference serving architecture, token-aware load balancing, and auto-scaling. Optimize time-to-first-token and throughput for production systems.

Discover why LLMs need Retrieval-Augmented Generation. Learn how RAG bridges knowledge gaps, reduces hallucinations, and enables non-parametric memory.

Master RAG system design by exploring retriever-generator interactions, timing strategies like iterative retrieval, and architectural variations like RETRO.

Master dense retrieval for semantic search. Explore bi-encoder architectures, embedding metrics, and contrastive learning to overcome keyword limitations.

Master contrastive learning for dense retrieval. Learn to train models using InfoNCE loss, in-batch negatives, and hard negative mining strategies effectively.

Master document chunking for RAG systems. Explore fixed-size, recursive, and semantic strategies to balance retrieval precision with context window limits.

Learn how embedding models convert text to vectors for RAG. Covers bi-encoder architecture, pooling strategies, dimensionality trade-offs, and model selection.

Explore vector similarity search for RAG systems. Compare cosine, dot product, and Euclidean metrics, and implement exact vs. approximate search with FAISS.

Master Hierarchical Navigable Small World (HNSW) graphs for vector search. Learn graph architecture, construction, and tuning for high-speed retrieval.

Master IVF indexes for scalable vector search. Learn clustering-based partitioning, nprobe tuning, and IVF-PQ compression for billion-scale retrieval.

Learn how Product Quantization compresses embeddings up to 100x using learned codebooks and asymmetric distance computation for scalable vector search.

Learn how hybrid search fuses BM25 keyword retrieval with dense vector retrieval using reciprocal rank fusion and weighted score combination to improve recall.

Learn how reranking with cross-encoders solves bi-encoder limitations. Master two-stage retrieval, training strategies, and latency optimization for production search systems.

Master RAG prompt engineering with strategic context placement, citation formats, and truncation strategies to improve LLM accuracy and reduce hallucinations.

Master RAG evaluation with metrics for retrieval quality (Precision@K, NDCG, MRR) and generation faithfulness using the RAGAS framework for AI systems.

Covers continual learning definition, catastrophic forgetting, continual learning scenarios.

Covers elastic weight consolidation, synaptic intelligence, parameter importance, regularization trade-offs.

Covers replay buffer design, pseudo-rehearsal, generative replay, replay selection.

Covers progressive networks, expert expansion, architecture search, modular approaches.

Covers forward transfer, backward transfer, evaluation protocols, continual benchmarks.

Covers code training data, code tokenization, fill-in-the-middle training, code pre-training objectives.

Covers code explanation, bug detection, code review, code search.

Covers completion context, completion ranking, completion latency, completion UX.

Covers docstring-to-code, test-to-code, code generation strategies, generation quality.

Covers sandboxed execution, execution feedback, iterative refinement, execution safety.

Covers functional correctness, pass@k metric, code benchmarks, beyond correctness.

Discover why LLMs require external tools to overcome knowledge cutoffs, computational limits, and hallucinations. Learn about tool-augmented AI systems.

Learn how function calling enables LLMs to invoke external tools and APIs through structured JSON schemas, bridging natural language and executable code.

Learn how the ReAct pattern enables LLM agents to interleave reasoning with tool execution. Master thought-action-observation loops for autonomous AI systems.

Master LLM tool selection through embedding-based routing, hybrid strategies, and semantic interfaces. Learn to build scalable multi-tool agent systems.

Master LLM agent architectures including control loops, state management strategies, planning mechanisms, and termination conditions for autonomous AI systems.

Learn how AI agents manage short-term context and long-term memory. Explore vector databases, retrieval algorithms, and memory hierarchies that enable persistent, learning systems.

Learn to evaluate AI agents with task completion metrics, trajectory analysis, and safety testing. Covers WebArena, SWE-bench, GAIA, and OSWorld benchmarks.

Covers task decomposition, goal-directed planning, plan execution, plan revision and recovery.

Covers agent coordination, communication protocols, role assignment, multi-agent benchmarks.

Covers unsafe action prevention, agent alignment, sandboxing, monitoring and intervention.

Covers image patching, patch embeddings, ViT architecture, ViT pre-training.

Covers CLIP architecture, CLIP training objective, CLIP zero-shot classification, CLIP embeddings.

Covers ViT variants for VLMs, SigLIP improvements, image resolution handling, encoder selection.

Covers linear projection, MLP projection, Q-Former approach, projection training.

Covers LLaVA design, two-stage training, visual conversation, LLaVA variants.

Covers cross-attention to images, gated cross-attention, few-shot visual learning, Flamingo training.

Covers image-text pairs, interleaved documents, visual instruction data, data quality.

Covers VQA benchmarks, multimodal understanding benchmarks, multimodal generation evaluation.

Covers multimodal learning principles, cross-modal alignment, joint embedding spaces, multimodal challenges.

Covers visual question answering, image captioning, visual grounding, scene understanding.

Covers diffusion models, text-to-image generation, image editing, generation evaluation.

Covers document understanding, medical imaging, video understanding, multimodal reasoning tasks.

Covers mel spectrograms, mel filterbanks, feature normalization, audio preprocessing.

Covers Whisper encoder-decoder, multitask training, language tokens, timestamp prediction.

Covers Whisper training data, weak supervision, multilingual training, Whisper capabilities.

Covers speech encoder + LLM, audio tokens, speech-to-text-to-LLM vs end-to-end, speech LLM architectures.

Covers TTS architecture overview, vocoder role, TTS quality metrics, neural TTS approaches.

Covers perplexity calculation, perplexity interpretation, perplexity limitations, comparing perplexities.

Covers cross-entropy definition, bits-per-character, cross-entropy vs perplexity, loss curves.

Covers n-gram precision, brevity penalty, BLEU formula, BLEU limitations, corpus vs sentence BLEU.

Covers ROUGE-N, ROUGE-L, ROUGE-W, ROUGE interpretation, ROUGE limitations.

Covers BERTScore computation, token alignment, BERTScore variants, BERTScore vs BLEU.

Covers exact match scoring, token-level F1, normalization for matching, metric selection.

Covers calibration definition, expected calibration error, calibration plots, calibration methods.

Covers evaluation design principles, metric selection, evaluation pitfalls, evaluation frameworks.

Covers benchmark construction, dataset collection, annotation guidelines, benchmark validity.

Covers benchmark contamination, evaluation brittleness, gaming metrics, evaluation best practices.

Covers MMLU structure, subject coverage, MMLU evaluation protocol, MMLU limitations.

Covers HellaSwag task design, adversarial filtering, HellaSwag evaluation, HellaSwag saturation.

Covers GSM8K problem types, chain-of-thought evaluation, GSM8K accuracy metrics, math reasoning assessment.

Covers HumanEval structure, functional correctness, pass@k metric, HumanEval limitations.

Covers MBPP dataset, MBPP vs HumanEval, code evaluation challenges.

Covers TruthfulQA design, truthfulness vs informativeness, TruthfulQA evaluation methods.

Covers contamination problem, contamination detection methods, n-gram overlap analysis, contamination mitigation.

Covers ceiling effects, benchmark retirement, dynamic benchmarks, benchmark evolution.

Covers evaluation interface design, task instructions, annotator selection, evaluation cost.

Covers Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, handling disagreement.

Covers A/B comparison design, Elo rating systems, preference aggregation, statistical significance.

Learn how to build LLM-as-Judge evaluation pipelines: prompt design, judge model selection, calibration against human annotations, and bias mitigation.

Covers position bias measurement, bias mitigation (swapping), verbosity bias, sycophancy.

Learn how to design reliable LLM judge prompts using explicit criteria, few-shot examples, and chain-of-thought formatting to maximize evaluation accuracy.

Learn how language models hallucinate: intrinsic and extrinsic hallucination, factual errors, fabrication, and inconsistency with NLI-based detection.

Learn four methods for detecting LLM hallucinations: entailment-based scoring, knowledge base verification, self-consistency checks, and learned detection models.

Covers training data issues, exposure bias, knowledge gaps, generation pressure.

Learn how to reduce LLM hallucination using retrieval augmentation, self-consistency decoding, DPO training, and calibrated uncertainty expression.

Learn how language models link generated claims to source documents, evaluate citation accuracy with NLI, and measure attribution precision and recall.

Covers confidence calibration, verbalized uncertainty, sampling-based uncertainty, uncertainty communication.

Covers bias sources, bias types (demographic, cultural), bias in training data, bias amplification.

Covers embedding association tests, generation bias metrics, classification bias metrics, bias benchmarks.

Practical techniques for reducing demographic bias in language models: data balancing, embedding debiasing, adversarial training, and prompt-based interventions.

Learn the key mathematical definitions of algorithmic fairness, from demographic parity to equalized odds, and why satisfying all metrics simultaneously is impossible.

Covers stereotyping, erasure, demeaning associations, measuring representation harms.

Covers debugging, trust, safety, scientific understanding, interpretability approaches overview.

Covers attention weight extraction, attention head visualization, attention interpretation caveats, attention tools.

Covers attention vs importance, attention manipulation studies, gradient-based alternatives.

Covers linear probing methodology, probing task design, probing interpretation, control tasks.

Covers layer selection, representation evolution, task localization, layer probing patterns.

Covers patching methodology, locating information, patching experiments, causal tracing.

Covers logit lens concept, intermediate vocabulary projection, tuned lens, lens interpretation.

Covers SAE architecture, sparsity constraints, dictionary learning, SAE for LLMs.

Covers feature activation patterns, feature naming, automated interpretation, feature circuits.

Reverse-engineer transformer networks into human-understandable algorithms by identifying circuits, induction heads, and mechanistic discoveries.

Learn how steering vectors and activation addition let you modify language model behavior at inference time by injecting directional nudges into the residual stream.

Covers harmful content generation, misuse scenarios, unintended harms, safety threat models.

Learn how red teams systematically probe language models for safety failures, covering attack taxonomies, ASR metrics, RL-based automated attack generation, and real-world findings.

Covers jailbreak techniques, prompt injection, adversarial suffixes, jailbreak defenses.

Covers direct prompt injection, indirect prompt injection, injection in RAG, injection defenses.

Covers classification-based filtering, rule-based filtering, filter placement, filter evaluation.

Covers input guardrails, output guardrails, guardrail frameworks, guardrail design.

How language models memorize training data, methods for measuring extractable memorization, PII risks in web-scale corpora, and practical privacy mitigations.

Learn how differential privacy protects training data in language models, from the mathematical guarantee to DP-SGD, privacy budgets, and practical LLM fine-tuning.

Covers creative writing, content generation, text transformation, generation quality control.

Covers extractive vs abstractive, length control, faithfulness, multi-document summarization.

Covers open-domain QA, reading comprehension, knowledge-intensive QA, QA evaluation.

Covers named entity recognition, relation extraction, event extraction, structured output generation.

Covers sentiment analysis, intent detection, topic classification, zero-shot classification.

Covers dialogue management, context tracking, persona consistency, conversation evaluation.

Covers story generation, poetry, code creativity, creative constraints and control.

Covers serving frameworks, model loading, request handling, serving configuration.

Covers latency breakdown, batching latency, streaming responses, latency monitoring.

Covers batch size tuning, GPU utilization, concurrent requests, throughput measurement.

Covers scaling metrics, horizontal scaling, scale-up vs scale-out, scaling policies.