Language AI Handbook

Covers CBOW architecture, context word averaging, CBOW objective function, CBOW vs skip-gram training speed, CBOW for frequent words, implementing CBOW forward pass, CBOW gradient derivation.

Covers softmax computational bottleneck, negative sampling objective derivation, sampling distribution (unigram^0.75), number of negatives hyperparameter, negative sampling gradient computation, NCE vs negative sampling, implementing efficient sampling.

Covers binary tree construction (Huffman coding), path probability computation, hierarchical softmax objective, gradient computation along paths, tree structure impact on learning, hierarchical softmax vs negative sampling, when to use each approach.

Covers data preprocessing pipeline, subsampling frequent words, learning rate scheduling, minibatch vs online training, convergence monitoring, gensim Word2Vec usage, training from scratch in PyTorch.

Covers vector arithmetic for analogies, parallelogram model, analogy evaluation datasets, 3CosAdd vs 3CosMul methods, analogy accuracy metrics, limitations of analogy evaluation, what analogies reveal about embeddings.

Covers GloVe objective function derivation, weighted least squares formulation, relationship to matrix factorization, weighting function design, bias terms in GloVe, GloVe vs Word2Vec comparison, training GloVe efficiently.

Covers character n-gram representation, word vector as n-gram sum, FastText architecture, handling OOV words, morphological awareness, FastText for morphologically rich languages, training FastText models.

Covers intrinsic vs extrinsic evaluation, word similarity datasets (SimLex, WordSim), analogy accuracy, embedding visualization (t-SNE, UMAP), downstream task evaluation, embedding bias detection, evaluation pitfalls.

Covers OOV word problem, vocabulary size explosion, rare word representation, morphological productivity, compound words, code and technical text, the case for subword units.

Covers BPE algorithm step-by-step, merge rules learning, vocabulary size control, BPE encoding procedure, BPE decoding procedure, BPE implementation from scratch, BPE hyperparameters.

Covers WordPiece vs BPE differences, likelihood objective for merges, greedy tokenization algorithm, ## prefix notation, WordPiece in BERT, training WordPiece tokenizers, handling unknown characters.

Covers unigram LM formulation, EM algorithm for training, Viterbi decoding for tokenization, sampling multiple segmentations, subword regularization, unigram vs BPE comparison, SentencePiece unigram mode.

Covers treating text as raw bytes, whitespace handling (▁ prefix), BPE and unigram modes, training from raw text, pretokenization elimination, SentencePiece in production, multilingual tokenization.

Covers corpus preparation, vocabulary size selection, special tokens configuration, training with HuggingFace tokenizers, saving and loading tokenizers, tokenizer versioning, domain-specific tokenizers.

Covers [CLS], [SEP], [PAD], [MASK], [UNK] tokens, beginning/end of sequence tokens, custom special tokens, special token embeddings, token type IDs, handling special tokens in generation.

Covers number tokenization issues, code tokenization, multilingual text mixing, emoji and Unicode edge cases, tokenization artifacts, adversarial tokenization, measuring tokenization quality.

Covers POS tag sets (Penn Treebank, Universal), POS tagging as classification, contextual disambiguation, POS tagging accuracy metrics, POS tagging for downstream tasks, rule-based vs statistical taggers.

Covers entity types (PER, ORG, LOC, etc.), NER as sequence labeling, nested entity challenges, entity boundary detection, NER evaluation (exact vs partial match), NER datasets and benchmarks.

Covers BIO scheme explanation, BIOES/BILOU variants, converting spans to BIO tags, BIO decoding to spans, handling tagging inconsistencies, BIO for multi-label scenarios, implementing BIO utilities.

Covers noun phrase chunking, chunk types (NP, VP, PP), IOB tagging for chunks, chunking vs full parsing, chunking evaluation, chunking as preprocessing, regex chunking with NLTK.

Covers HMM components (states, observations, transitions), emission and transition probabilities, HMM assumptions (Markov, independence), HMM for POS tagging, HMM parameter estimation, HMM limitations for NLP.

Covers optimal path problem formulation, Viterbi recursion derivation, backpointer tracking, Viterbi complexity analysis, log-space computation, implementing Viterbi efficiently, Viterbi for beam search foundation.

Covers CRF vs HMM comparison, CRF feature functions, log-linear formulation, partition function computation, CRF for NER, CRF inference complexity, neural CRF layers.

Covers CRF log-likelihood objective, forward-backward algorithm, gradient computation, L-BFGS optimization, feature template design, CRF regularization, CRF training convergence.

Covers linear decision boundaries, weight vectors and bias, dot product interpretation, multiclass classification (softmax), linear classifier limitations, training with gradient descent.

Covers sigmoid function and saturation, tanh properties, ReLU and dying ReLU, Leaky ReLU and PReLU, ELU and SELU, GELU derivation and properties, Swish and Mish, choosing activation functions.

Covers hidden layers and depth, weight matrices between layers, forward pass computation, representational capacity, MLP for classification, MLP for regression, MLP architecture design.

Covers cross-entropy loss derivation, MSE for regression, binary vs multiclass cross-entropy, label smoothing, focal loss for imbalance, loss function numerical stability, custom loss functions.

Covers computational graphs, chain rule review, forward and backward pass, gradient accumulation, backprop complexity analysis, automatic differentiation, implementing backprop from scratch.

Covers batch vs stochastic gradient descent, minibatch gradient descent, learning rate selection, SGD convergence properties, SGD noise as regularization, learning rate schedules basics, SGD implementation.

Covers momentum intuition (ball rolling), momentum update equations, momentum coefficient selection, dampening oscillations, momentum vs vanilla SGD, Nesterov momentum derivation, implementing momentum.

Covers exponential moving averages, first moment (mean) estimation, second moment (variance) estimation, bias correction derivation, Adam update rule, Adam hyperparameters, Adam convergence properties.

Covers L2 regularization vs weight decay, why they differ with Adam, AdamW formulation, weight decay coefficient selection, AdamW as default optimizer, AdamW vs Adam empirically.

Covers random initialization importance, Xavier/Glorot initialization derivation, He initialization for ReLU, initialization for different activations, layer-wise initialization, initialization debugging, modern initialization practices.

Covers internal covariate shift, batch statistics computation, learnable scale and shift, training vs inference mode, batch norm gradient flow, batch norm placement debates, batch norm limitations.

Covers dropout as ensemble, dropout mask sampling, inverted dropout scaling, dropout rate selection, dropout at inference, spatial dropout for sequences, dropout in modern architectures.

Covers gradient explosion detection, clip by value, clip by global norm, gradient clipping implementation, when to use gradient clipping, clipping threshold selection, monitoring gradient norms.

Covers recurrent connection intuition, hidden state as memory, unrolled computation graph, parameter sharing across time, RNN for sequence classification, RNN for sequence generation, RNN equations and dimensions.

Covers BPTT derivation, gradient flow through time, truncated BPTT, BPTT memory requirements, BPTT implementation, gradient accumulation across timesteps.

Covers gradient product across timesteps, vanishing gradient analysis, long-range dependency failure, gradient visualization, vanishing vs exploding trade-off, architectural solutions overview.

Covers cell state as information highway, gate mechanism intuition, LSTM diagram walkthrough, information flow in LSTMs, LSTM for long sequences, LSTM memory capacity.

Covers forget gate equations, input gate equations, cell state update, output gate equations, hidden state computation, LSTM parameter count, implementing LSTM from scratch.

Covers constant error carousel, forget gate gradient highway, gradient flow analysis, LSTM vs vanilla RNN gradients, peephole connections, LSTM gradient clipping needs.

Covers GRU vs LSTM comparison, reset gate function, update gate function, candidate hidden state, GRU equations, GRU parameter efficiency, when to choose GRU vs LSTM.

Covers forward and backward passes, hidden state concatenation, bidirectional architectures, bidirectionality for classification, limitations for generation, implementing bidirectional RNNs.

Covers multiple RNN layers, residual connections for depth, layer normalization in RNNs, depth vs width trade-offs, gradient flow in deep RNNs, practical depth limits.

Covers encoder role and design, decoder role and design, context vector as bottleneck, seq2seq for machine translation, seq2seq for summarization, seq2seq training setup.

Covers teacher forcing procedure, exposure bias problem, teacher forcing efficiency, scheduled sampling, curriculum learning, teacher forcing vs autoregressive training.

Covers greedy decoding limitations, beam search algorithm, beam width selection, length normalization, diverse beam search, beam search implementation, beam search vs sampling.

Covers attention as soft lookup, attention weight interpretation, attention for variable-length inputs, attention visualization, attention vs pooling, attention computation overview.

Covers alignment model formulation, score function (additive), attention weight computation, context vector as weighted sum, attention in decoder, Bahdanau attention implementation.

Covers dot product attention, general (bilinear) attention, concat attention variant, global vs local attention, Luong vs Bahdanau comparison, attention placement (input vs output).

Covers pointer network motivation, copy probability computation, mixing generation and copying, pointer-generator networks, copy mechanism for summarization, OOV handling with copy.

Covers cross-attention vs self-attention, self-attention motivation, all-pairs interaction, self-attention for representation learning, self-attention computational pattern.

Covers QKV intuition (database lookup), projection matrices Wq, Wk, Wv, query-key matching, value retrieval, QKV dimensions and shapes, QKV as learned transformations.

Covers dot product for similarity, softmax for normalization, scaling factor derivation (1/√dk), attention output computation, attention in matrix form, attention implementation.

Covers padding masks, causal (look-ahead) masks, combining multiple masks, mask shapes and broadcasting, efficient masking implementation, custom attention patterns.

Covers multiple attention heads motivation, head dimension splitting, parallel attention computation, output concatenation and projection, head specialization, multi-head vs single head.

Covers O(n²) attention complexity, memory requirements, attention bottleneck in long sequences, FLOPs computation, attention vs RNN complexity, practical scaling limits.

Covers transformer position blindness, why position matters for language, position information requirements, position encoding vs position embedding, absolute vs relative position.

Covers sinusoidal formula derivation, wavelength intuition, position encoding visualization, extrapolation properties, sinusoidal encoding implementation, learned vs sinusoidal trade-offs.

Covers position embedding table, position embedding training, maximum sequence length, learned embedding extrapolation, position embedding analysis, GPT-style position embeddings.

Covers relative position motivation, relative attention formulation, clipping relative positions, relative position in self-attention, Shaw et al. relative positions, relative bias implementation.

Covers RoPE intuition, rotation matrix formulation, RoPE in complex numbers, relative position through rotation, RoPE implementation, RoPE frequency patterns.

Covers ALiBi motivation, linear bias by distance, head-specific slopes, ALiBi extrapolation properties, ALiBi simplicity advantages, ALiBi vs RoPE comparison.

Covers extrapolation benchmarks, training efficiency comparison, implementation complexity, position encoding for long context, hybrid approaches, current best practices.

Covers residual connection formulation, gradient highway interpretation, residual scaling, residual connections in transformers, pre-norm vs post-norm residuals.

Covers layer norm vs batch norm, layer norm formula, learnable affine parameters, layer norm placement, layer norm gradient flow, layer norm implementation.

Covers RMSNorm derivation, removing mean centering, RMSNorm efficiency, RMSNorm vs LayerNorm performance, RMSNorm in modern architectures.

Covers original transformer (post-norm), pre-norm formulation, training stability comparison, gradient flow differences, when to use each, modern consensus.

Covers FFN architecture, hidden dimension expansion, FFN as two linear layers, position independence, FFN parameter count, FFN computational cost.

Covers ReLU in original transformer, GELU adoption, GELU approximations, SiLU/Swish in modern models, activation function comparison.

Covers GLU formulation, gating mechanism, SwiGLU derivation, GeGLU variant, GLU parameter efficiency, GLU in modern architectures.

Covers standard block structure, component ordering, block implementation, block initialization, forward pass walkthrough, block hyperparameters.

Covers encoder-only design, bidirectional self-attention, encoder for understanding tasks, encoder output usage, BERT-style encoder, encoder layer stacking.

Covers decoder-only design, causal masking requirement, autoregressive generation, decoder for generation tasks, GPT-style decoder, decoder layer stacking.

Covers encoder-decoder interaction, cross-attention mechanism, encoder-decoder for seq2seq, T5-style architecture, information flow, when to use encoder-decoder.

Covers cross-attention formulation, KV from encoder, Q from decoder, cross-attention masking, cross-attention placement, cross-attention implementation.

Covers input-output embedding tying, encoder-decoder tying, parameter reduction, weight tying effects on training, when to tie weights.

Covers depth vs width trade-offs, number of heads selection, hidden dimension ratios, FFN expansion ratio, total parameter calculation, architecture search.

Covers O(n²) memory analysis, O(n²) compute analysis, attention matrix size, practical sequence limits, bottleneck visualization, motivation for efficiency.

Covers local attention windows, strided attention patterns, block-sparse attention, combining sparse patterns, sparse attention implementation.

Covers sliding window formulation, window size selection, dilated sliding windows, sliding window for long sequences, Mistral-style windowed attention.

Covers CLS token global attention, learned global tokens, global-local attention mixing, global token count, implementation strategies.

Covers Longformer attention pattern, global attention configuration, Longformer complexity, Longformer for documents, Longformer implementation.

Covers BigBird attention pattern, random attention benefits, BigBird theoretical guarantees, BigBird vs Longformer, BigBird applications.

Covers softmax attention reformulation, kernel feature maps, linear complexity attention, linear attention limitations, Performer and variants.

Covers GPU memory hierarchy, tiling for SRAM, online softmax computation, recomputation strategy, FlashAttention complexity, FlashAttention benefits.

Covers CUDA kernel basics, memory access patterns, FlashAttention-2 improvements, using FlashAttention in PyTorch, FlashAttention limitations.

Covers training sequence length limits, attention memory scaling, position encoding extrapolation, long-range dependency learning, evaluation challenges.

Covers linear position scaling, interpolation vs extrapolation, position interpolation implementation, fine-tuning for longer context, interpolation limitations.

Covers RoPE frequency analysis, high-frequency preservation, NTK-aware formula, dynamic NTK scaling, NTK vs linear interpolation.

Covers YaRN motivation, attention scaling factor, YaRN formula, YaRN training requirements, YaRN vs alternatives.

Covers attention sink phenomenon, StreamingLLM approach, sink token design, streaming inference, infinite context generation.

Covers memory network concepts, memory retrieval mechanisms, memory writing and updating, memory-augmented transformers, Memorizing Transformers.

Covers Transformer-XL approach, segment-level processing, recurrent state passing, relative position in recurrence, recurrent memory limitations.

Covers CLM objective formulation, autoregressive factorization, CLM loss computation, CLM for generation, CLM training data, CLM scaling properties.

Covers MLM objective formulation, masking strategies (15% rule), [MASK] token usage, MLM for understanding, MLM training dynamics.

Covers subword masking problems, whole word masking procedure, WWM implementation, WWM vs random masking, WWM for different tokenizers.

Covers span selection strategies, span length distribution, sentinel tokens, T5-style corruption, span corruption benefits.

Covers prefix LM formulation, prefix LM attention pattern, prefix LM for generation, prefix LM training, UniLM-style objectives.

Covers generator-discriminator setup, replaced vs original detection, RTD efficiency advantages, ELECTRA training procedure, RTD vs MLM comparison.

Covers token deletion, token shuffling, sentence permutation, document rotation, BART-style denoising, combining denoising tasks.

Covers BERT model sizes, BERT layer configuration, BERT embedding layers, BERT attention patterns, BERT output representations.

Covers pre-training data preparation, MLM implementation, NSP task design, pre-training hyperparameters, pre-training duration.

Covers classification fine-tuning, sequence labeling fine-tuning, question answering fine-tuning, fine-tuning hyperparameters, catastrophic forgetting.

Covers [CLS] token usage, layer selection strategies, pooling strategies, BERT as feature extractor, frozen vs fine-tuned representations.

Covers dynamic masking, NSP removal, larger batches, more data, RoBERTa training recipe, RoBERTa vs BERT performance.

Covers factorized embeddings, cross-layer parameter sharing, sentence order prediction, ALBERT efficiency, ALBERT performance trade-offs.

Covers generator training, discriminator training, RTD objective, ELECTRA sample efficiency, ELECTRA scaling, ELECTRA fine-tuning.

Covers disentangled attention formulation, enhanced mask decoder, DeBERTa position encoding, DeBERTa improvements, DeBERTa-v3 advances.

Covers GPT-1 architecture, GPT-1 pre-training, GPT-1 fine-tuning approach, GPT-1 transfer learning, GPT-1 historical significance.

Covers GPT-2 model sizes, GPT-2 architectural changes, zero-shot task performance, GPT-2 training data (WebText), GPT-2 generation quality.

Covers GPT-3 scale (175B), few-shot prompting discovery, in-context learning analysis, GPT-3 capabilities, GPT-3 limitations.

Covers ICL phenomenon, ICL vs fine-tuning, example selection strategies, ICL scaling behavior, ICL theoretical understanding.

Covers generation procedure, KV caching for efficiency, generation stopping criteria, generation speed optimization, generation code implementation.

Covers temperature scaling, temperature effects on distribution, temperature selection guidelines, temperature vs quality trade-off.

Covers top-k truncation, k selection strategies, top-k limitations, top-k implementation, combining with temperature.

Covers top-p formulation, cumulative probability threshold, nucleus sampling benefits, p selection guidelines, nucleus vs top-k.

Covers repetition in generation, repetition penalty formulation, frequency penalty, presence penalty, n-gram blocking.

Covers grammar-guided generation, JSON schema constraints, regex constraints, constrained beam search, constrained sampling.

Covers LLaMA design philosophy, LLaMA architectural choices, LLaMA training data, LLaMA efficiency, LLaMA significance.

Covers pre-norm with RMSNorm, SwiGLU FFN, RoPE implementation, component interactions, implementation details.

Covers GQA motivation, GQA formulation, KV head grouping, GQA memory savings, GQA vs MHA performance, GQA implementation.

Covers MQA extreme sharing, MQA memory benefits, MQA quality trade-offs, MQA for inference, MQA vs GQA.

Covers Mistral design choices, sliding window attention, Mistral efficiency, Mistral performance, Mistral vs LLaMA.

Covers Qwen architectural choices, Qwen training approach, Qwen multilingual capabilities, Qwen variants.

Covers Phi design philosophy, textbook-quality data, Phi training approach, Phi efficiency, small model capabilities.

Covers T5 encoder-decoder design, T5 attention patterns, T5 model sizes, T5 relative positions, T5 implementation.

Covers span corruption procedure, sentinel tokens, corruption rate, T5 pre-training data, T5 training scale.

Covers task prefixes, classification as generation, NER as generation, QA as generation, task formatting examples.

Covers BART encoder-decoder, BART attention configuration, BART vs T5 comparison, BART model sizes.

Covers token masking, token deletion, text infilling, sentence permutation, document rotation, objective combinations.

Covers mT5 training data, language sampling, cross-lingual transfer, mT5 vs T5 performance, multilingual tokenization.

Covers power law definition, log-log linear relationships, power law fitting, power law universality, power law intuition.

Covers loss vs parameters, loss vs data, loss vs compute, Kaplan optimal allocation, Kaplan predictions.

Covers Chinchilla experiments, revised scaling coefficients, optimal tokens per parameter, Chinchilla vs Kaplan, Chinchilla implications.

Covers compute budget allocation, tokens vs parameters ratio, training efficiency, compute-optimal recipes, practical guidelines.

Covers data repetition effects, optimal repetition strategies, data augmentation scaling, synthetic data scaling.

Covers training vs inference compute, inference-optimal models, over-training for efficiency, deployment cost modeling.

Covers loss extrapolation, capability prediction, scaling law uncertainty, prediction reliability, practical forecasting.

Covers emergence definition, phase transitions, emergence examples, emergence mechanisms, emergence debate.

Covers ICL emergence curves, ICL vs fine-tuning scaling, ICL mechanism hypotheses, ICL as meta-learning.

Covers CoT emergence observations, CoT elicitation, CoT scaling behavior, CoT mechanism theories.

Covers discontinuous metrics, accuracy threshold effects, smooth underlying capabilities, re-examining emergence claims.

Covers inverse scaling phenomena, distractor tasks, sycophancy scaling, inverse scaling prize findings.

Covers grokking phenomenon, grokking in arithmetic, grokking mechanism theories, grokking phase transitions, practical implications.

Covers dense vs sparse trade-offs, conditional computation motivation, sparse model efficiency, sparse model challenges.

Covers expert architecture, expert as FFN, expert capacity, expert count selection, expert placement in transformer.

Covers router architecture, routing score computation, router training, router learned behavior.

Covers top-1 routing, top-2 routing, k selection trade-offs, routing implementation, combining expert outputs.

Covers expert utilization imbalance, collapse failure mode, load metrics, balanced routing importance.

Covers load balancing loss formulation, loss coefficient tuning, balancing vs task loss, auxiliary loss implementation.

Covers router instability, z-loss formulation, z-loss benefits, z-loss coefficient, combined auxiliary losses.

Covers expert placement strategies, all-to-all communication, communication overhead, expert parallelism implementation.

Covers Switch Transformer design, top-1 routing choice, capacity factor, Switch scaling results.

Covers Mixtral architecture, Mixtral expert design, Mixtral performance, Mixtral efficiency, Mixtral vs dense models.

Covers transfer learning paradigm, pre-training/fine-tuning split, what transfers, transfer learning efficiency.

Covers full fine-tuning procedure, fine-tuning hyperparameters, learning rate selection, batch size effects.

Covers forgetting phenomenon, forgetting measurement, forgetting mitigation, pre-trained capability preservation.

Covers discriminative fine-tuning, layer-wise learning rates, warmup for fine-tuning, learning rate decay.

Covers few-shot fine-tuning, data augmentation, sample efficiency patterns, small data strategies.

Covers parameter storage costs, multi-task deployment, PEFT efficiency, PEFT quality trade-offs.

Covers weight update decomposition, low-rank assumption, LoRA efficiency gains, LoRA flexibility.

Covers LoRA formulation W + BA, rank selection, initialization scheme, LoRA gradient computation.

Covers LoRA module design, merging weights, LoRA training loop, LoRA in PyTorch, HuggingFace PEFT usage.

Covers rank selection guidelines, alpha/rank ratio, which layers to adapt, LoRA dropout.

Covers 4-bit quantization for base model, NF4 data type, double quantization, QLoRA memory savings.

Covers importance-based pruning, SVD-based adaptation, dynamic rank, AdaLoRA training procedure.

Covers IA3 formulation, learned rescaling vectors, IA3 parameter efficiency, IA3 vs LoRA.

Covers prefix tuning formulation, prefix length selection, prefix tuning for generation, prefix vs LoRA.

Covers prompt tuning formulation, prompt initialization, prompt tuning scaling, prompt length effects.

Covers adapter architecture, adapter placement, adapter dimensionality, adapter fusion.

Covers performance comparison, parameter efficiency comparison, task suitability, practical recommendations.

Covers instruction tuning motivation, instruction format design, instruction diversity, instruction quality.

Covers human annotation, template-based generation, seed task expansion, quality filtering.

Covers self-instruct procedure, instruction generation, response generation, filtering strategies.

Covers prompt templates, system messages, multi-turn format, chat templates, role definitions.

Covers instruction tuning data mixing, training hyperparameters, loss masking, multi-task learning.

Covers instruction following benchmarks, human evaluation, automatic evaluation, instruction difficulty.

Covers alignment definition, helpfulness vs harmlessness, alignment challenges, alignment approaches overview.

Covers preference collection UI, comparison design, annotator guidelines, preference data quality.

Covers pairwise comparison model, preference probability, Bradley-Terry likelihood, preference strength.

Covers reward model architecture, preference loss function, reward model training, reward model evaluation.

Covers reward hacking examples, distribution shift, over-optimization, reward hacking mitigation.

Covers policy definition, REINFORCE algorithm, policy gradient derivation, variance reduction.

Covers clipped objective, PPO derivation, trust region intuition, PPO implementation.

Covers LLM as policy, action space (tokens), reward assignment, KL penalty importance.

Covers SFT stage, reward model training, PPO fine-tuning, RLHF hyperparameters, RLHF debugging.

Covers KL penalty motivation, KL coefficient selection, adaptive KL, KL effects on training.

Covers DPO motivation, removing reward model, DPO intuition, DPO benefits.

Covers DPO from RLHF objective, optimal policy derivation, DPO loss function, DPO as classification.

Covers DPO data format, DPO loss computation, DPO training procedure, DPO hyperparameters.

Covers IPO formulation, KTO for unpaired feedback, ORPO, cDPO, comparing alignment methods.

Covers AI as annotator, constitutional AI principles, AI preference generation, RLAIF scalability.

Covers iterative DPO, online preference learning, self-improvement loops, alignment stability.

Covers KV cache motivation, cache structure, cache memory requirements, cache management.

Covers cache size calculation, batch size effects, sequence length effects, memory bottleneck.

Covers memory fragmentation problem, page-based allocation, vLLM approach, paged attention benefits.

Covers cache eviction strategies, attention sink preservation, H2O algorithm, cache quantization.

Covers quantization fundamentals, per-tensor vs per-channel, symmetric vs asymmetric, calibration.

Covers INT8 range mapping, absmax quantization, smooth quantization, INT8 accuracy.

Covers 4-bit challenges, group-wise quantization, 4-bit accuracy trade-offs, 4-bit formats.

Covers GPTQ algorithm, layer-wise quantization, Hessian approximation, GPTQ implementation.

Covers salient weight preservation, AWQ algorithm, AWQ vs GPTQ, AWQ benefits.

Covers GGML/GGUF history, quantization types, GGUF file format, llama.cpp integration.

Covers speculative decoding concept, draft model selection, verification procedure, acceptance rate.

Covers acceptance criterion, expected speedup, draft quality effects, optimal draft length.

Covers static vs continuous batching, iteration-level scheduling, request completion handling, throughput gains.

Covers inference server architecture, request routing, load balancing, auto-scaling, latency optimization.

Covers knowledge limitations, parametric vs non-parametric, RAG benefits, RAG use cases.

Covers retriever component, generator component, retrieval timing, architecture variations.

Covers bi-encoder architecture, embedding similarity, dense vs sparse retrieval, dense retrieval training.

Covers contrastive loss, in-batch negatives, hard negative mining, DPR training procedure.

Covers chunking strategies, chunk size selection, overlap handling, semantic chunking.

Covers embedding model architectures, pooling strategies, embedding dimensions, embedding model selection.

Covers distance metrics, exact vs approximate search, complexity trade-offs, similarity search libraries.

Covers HNSW algorithm, graph construction, search procedure, HNSW parameters.

Covers clustering approach, probe count, IVF-PQ combination, IVF vs HNSW.

Covers PQ algorithm, codebook learning, PQ accuracy trade-offs, PQ for scale.

Covers BM25 + dense fusion, reciprocal rank fusion, weighted combination, hybrid benefits.

Covers cross-encoder architecture, reranking procedure, reranker training, reranker latency.

Covers context placement, citation formats, context truncation, instruction design.

Covers retrieval metrics, generation metrics, end-to-end evaluation, RAGAS framework.

Covers LLM limitations, tool augmentation, tool use examples, tool use benefits.

Covers function schema definition, function call generation, function output handling, function calling fine-tuning.

Covers ReAct formulation, thought-action-observation loop, ReAct prompting, ReAct examples.

Covers tool descriptions, tool routing, multi-tool scenarios, tool selection training.

Covers agent loop design, state management, planning strategies, agent termination.

Covers short-term memory, long-term memory, memory retrieval, memory summarization.

Covers task completion metrics, trajectory evaluation, agent benchmarks, safety evaluation.

Covers image patching, patch embeddings, ViT architecture, ViT pre-training.

Covers CLIP architecture, CLIP training objective, CLIP zero-shot classification, CLIP embeddings.

Covers ViT variants for VLMs, SigLIP improvements, image resolution handling, encoder selection.

Covers linear projection, MLP projection, Q-Former approach, projection training.

Covers LLaVA design, two-stage training, visual conversation, LLaVA variants.

Covers cross-attention to images, gated cross-attention, few-shot visual learning, Flamingo training.

Covers image-text pairs, interleaved documents, visual instruction data, data quality.

Covers VQA benchmarks, multimodal understanding benchmarks, multimodal generation evaluation.

Covers mel spectrograms, mel filterbanks, feature normalization, audio preprocessing.

Covers Whisper encoder-decoder, multitask training, language tokens, timestamp prediction.

Covers Whisper training data, weak supervision, multilingual training, Whisper capabilities.

Covers speech encoder + LLM, audio tokens, speech-to-text-to-LLM vs end-to-end, speech LLM architectures.

Covers TTS architecture overview, vocoder role, TTS quality metrics, neural TTS approaches.

Covers perplexity calculation, perplexity interpretation, perplexity limitations, comparing perplexities.

Covers cross-entropy definition, bits-per-character, cross-entropy vs perplexity, loss curves.

Covers n-gram precision, brevity penalty, BLEU formula, BLEU limitations, corpus vs sentence BLEU.

Covers ROUGE-N, ROUGE-L, ROUGE-W, ROUGE interpretation, ROUGE limitations.

Covers BERTScore computation, token alignment, BERTScore variants, BERTScore vs BLEU.

Covers exact match scoring, token-level F1, normalization for matching, metric selection.

Covers calibration definition, expected calibration error, calibration plots, calibration methods.

Covers MMLU structure, subject coverage, MMLU evaluation protocol, MMLU limitations.

Covers HellaSwag task design, adversarial filtering, HellaSwag evaluation, HellaSwag saturation.

Covers GSM8K problem types, chain-of-thought evaluation, GSM8K accuracy metrics, math reasoning assessment.

Covers HumanEval structure, functional correctness, pass@k metric, HumanEval limitations.

Covers MBPP dataset, MBPP vs HumanEval, code evaluation challenges.

Covers TruthfulQA design, truthfulness vs informativeness, TruthfulQA evaluation methods.

Covers contamination problem, contamination detection methods, n-gram overlap analysis, contamination mitigation.

Covers ceiling effects, benchmark retirement, dynamic benchmarks, benchmark evolution.

Covers evaluation interface design, task instructions, annotator selection, evaluation cost.

Covers Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, handling disagreement.

Covers A/B comparison design, Elo rating systems, preference aggregation, statistical significance.

Covers judge prompt design, judge model selection, judge calibration, judge limitations.

Covers position bias measurement, bias mitigation (swapping), verbosity bias, sycophancy.

Covers prompt sensitivity, evaluation prompt design, few-shot vs zero-shot evaluation, evaluation consistency.

Covers bias sources, bias types (demographic, cultural), bias in training data, bias amplification.

Covers embedding association tests, generation bias metrics, classification bias metrics, bias benchmarks.

Covers data balancing, fine-tuning for fairness, prompt-based mitigation, debiasing embeddings.

Covers demographic parity, equalized odds, fairness trade-offs, choosing fairness metrics.

Covers stereotyping, erasure, demeaning associations, measuring representation harms.

Covers intrinsic vs extrinsic hallucination, factual errors, fabrication, inconsistency.

Covers entailment-based detection, knowledge base verification, self-consistency checks, detection models.

Covers training data issues, exposure bias, knowledge gaps, generation pressure.

Covers retrieval augmentation, decoding strategies, training approaches, uncertainty expression.

Covers inline citation, attribution accuracy, source verification, attribution evaluation.

Covers confidence calibration, verbalized uncertainty, sampling-based uncertainty, uncertainty communication.

Covers harmful content generation, misuse scenarios, unintended harms, safety threat models.

Covers red team methodology, attack taxonomies, red team findings, red team automation.

Covers jailbreak techniques, prompt injection, adversarial suffixes, jailbreak defenses.

Covers direct prompt injection, indirect prompt injection, injection in RAG, injection defenses.

Covers classification-based filtering, rule-based filtering, filter placement, filter evaluation.

Covers input guardrails, output guardrails, guardrail frameworks, guardrail design.

Covers memorization measurement, extractable memorization, PII in training data, privacy risks.

Covers DP-SGD basics, privacy budget, DP accuracy trade-offs, DP for LLMs.

Covers debugging, trust, safety, scientific understanding, interpretability approaches overview.

Covers attention weight extraction, attention head visualization, attention interpretation caveats, attention tools.

Covers attention vs importance, attention manipulation studies, gradient-based alternatives.

Covers linear probing methodology, probing task design, probing interpretation, control tasks.

Covers layer selection, representation evolution, task localization, layer probing patterns.

Covers patching methodology, locating information, patching experiments, causal tracing.

Covers logit lens concept, intermediate vocabulary projection, tuned lens, lens interpretation.

Covers SAE architecture, sparsity constraints, dictionary learning, SAE for LLMs.

Covers feature activation patterns, feature naming, automated interpretation, feature circuits.

Covers circuit analysis, algorithmic tasks, induction heads, mechanistic discoveries.

Covers steering vectors, activation addition, representation engineering, steering applications.

Covers Common Crawl, crawling strategies, robots.txt respect, crawl freshness.

Covers HTML parsing, boilerplate removal, content extraction, trafilatura and similar tools.

Covers language ID models, multilingual document handling, code-switching, language filtering.

Covers exact deduplication, near-duplicate detection, document vs substring dedup, dedup at scale.

Covers MinHash algorithm, Jaccard similarity estimation, MinHash LSH, MinHash implementation.

Covers heuristic filters, perplexity filtering, classifier-based filtering, filter thresholds.

Covers toxicity classifiers, toxicity thresholds, over-filtering risks, toxicity filter evaluation.

Covers PII detection methods, PII removal strategies, PII removal evaluation, privacy preservation.

Covers domain proportions, quality weighting, data mixing experiments, optimal mixing.

Covers synthetic data generation, quality verification, synthetic data diversity, distillation.

Covers GPU memory hierarchy, CUDA cores, tensor cores, GPU specifications.

Covers memory breakdown (activations, parameters, gradients, optimizer states), memory estimation, OOM debugging.

Covers DDP algorithm, gradient synchronization, all-reduce operations, DDP scaling.

Covers column parallelism, row parallelism, communication patterns, Megatron-style parallelism.

Covers pipeline stages, micro-batching, pipeline bubbles, pipeline schedules (GPipe, 1F1B).

Covers ZeRO stage 1 (optimizer state partitioning), ZeRO stage 2 (gradient partitioning), ZeRO stage 3 (parameter partitioning), ZeRO memory savings.

Covers FSDP concepts, FSDP vs ZeRO, FSDP sharding strategies, FSDP usage.

Covers checkpointing concept, checkpoint selection, checkpointing overhead, selective checkpointing.

Covers floating point formats, loss scaling, BF16 advantages, mixed precision implementation.

Covers gradient compression, communication overlap, topology-aware communication, NCCL optimization.

Covers checkpoint contents, checkpoint frequency, async checkpointing, fault recovery.

Covers warmup motivation, linear warmup, warmup duration, warmup for large batches.

Covers step decay, exponential decay, inverse square root decay, decay scheduling.

Covers cosine decay formula, cosine with restarts, cosine schedule parameters, cosine vs linear.

Covers batch size effects, learning rate scaling, batch size limits, LAMB optimizer.

Covers weight decay formula, decoupled weight decay, weight decay selection, weight decay interaction with Adam.

Covers accumulation procedure, accumulation steps, accumulation for memory, accumulation correctness.

Covers loss spikes, gradient norm monitoring, stability techniques, training stability debugging.

Covers hyperparameter search, hyperparameter transfer, critical vs robust hyperparameters, default recipes.

Covers code training data, code tokenization, fill-in-the-middle training, code pre-training objectives.

Covers code explanation, bug detection, code review, code search.

Covers completion context, completion ranking, completion latency, completion UX.

Covers docstring-to-code, test-to-code, code generation strategies, generation quality.

Covers sandboxed execution, execution feedback, iterative refinement, execution safety.

Covers functional correctness, pass@k metric, code benchmarks, beyond correctness.

Covers serving frameworks, model loading, request handling, serving configuration.

Covers latency breakdown, batching latency, streaming responses, latency monitoring.

Covers batch size tuning, GPU utilization, concurrent requests, throughput measurement.

Covers scaling metrics, horizontal scaling, scale-up vs scale-out, scaling policies.

Covers model selection, A/B testing, model cascades, routing strategies.

Covers prompt caching, semantic caching, cache invalidation, cache hit rates.

Covers metrics collection, alerting, logging, dashboards.

Covers output quality metrics, drift detection, regression detection, quality alerts.

Covers cost modeling, cost optimization, cost allocation, cost monitoring.

Covers continual learning definition, catastrophic forgetting, continual learning scenarios.

Covers elastic weight consolidation, synaptic intelligence, parameter importance, regularization trade-offs.

Covers replay buffer design, pseudo-rehearsal, generative replay, replay selection.

Covers progressive networks, expert expansion, architecture search, modular approaches.

Covers forward transfer, backward transfer, evaluation protocols, continual benchmarks.

Covers distillation objective, temperature in distillation, teacher selection, distillation for LLMs.

Covers feature distillation, attention transfer, progressive distillation, on-policy distillation.

Covers weight pruning, structured vs unstructured, pruning criteria, pruning schedule.

Covers head pruning, layer pruning, width pruning, structured pruning implementation.

Covers weight averaging, task arithmetic, TIES merging, DARE merging.

Covers multi-task merging, style merging, capability composition, merging evaluation.

Covers constitutional principles, critique and revision, CAI training, CAI effectiveness.

Covers outcome vs process reward, PRM training, PRM for math, PRM limitations.

Covers multiple sampling, self-consistency, iterative refinement, compute-optimal inference.

Covers CoT prompting, zero-shot CoT, CoT fine-tuning, CoT limitations.

Covers self-consistency procedure, sampling diversity, voting strategies, self-consistency effectiveness.

Covers ToT framework, thought generation, thought evaluation, ToT search.

Covers RETRO architecture, retrieval during training, retrieved context integration.

Covers outline-based generation, hierarchical generation, coherence maintenance, long-form evaluation.

Covers watermarking schemes, statistical detection, watermark robustness, watermark evaluation.

Covers model card contents, intended use documentation, limitation documentation, model card best practices.

Covers release decisions, staged release, access control, deployment monitoring.