409 items · Page 5 of 9
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.
DPO variants include IPO, KTO, ORPO, and cDPO. Compare their objectives, data requirements, computational costs, and suitable alignment tasks.
Implement Direct Preference Optimization in PyTorch. Topics include preference data formatting, loss computation, training loops.
Explains how LLM agents plan, decompose tasks, execute multi-step goals, and recover from failures using architectures like ReWOO, Tree-of-Thought.
Derive the DPO loss function from first principles. Explains how the optimal RLHF policy leads to reward reparameterization and direct preference optimization.
Explains how DPO eliminates reward models from LLM alignment. Topics include the reward-policy duality that enables supervised preference learning.
Explains how KL divergence prevents reward hacking in RLHF by keeping policies close to reference models.
Covers RLHF pipeline with three stages: Supervised Fine-Tuning, Reward Model training, and PPO optimization. Topics include debugging techniques.
Explains how PPO applies to language models. Topics include policy mapping, token action spaces, KL divergence penalties, and advantage estimation for RLHF.
Covers PPO's clipped objective for stable policy updates. Topics include trust regions, GAE advantage estimation.
Covers policy gradient theory for language model alignment. Topics include REINFORCE algorithm, variance reduction with baselines, and foundations for PPO.
Examines reward hacking in RLHF where language models exploit proxy objectives. Topics include distribution shift, over-optimization, and mitigation strategies.
Build neural networks that learn human preferences from pairwise comparisons. Topics include reward model architecture, Bradley-Terry loss.
Explains how the Bradley-Terry model converts pairwise preferences into consistent rankings. Foundation for reward modeling in RLHF and Elo rating systems.
Collect and process human preference data for RLHF. Topics include pairwise comparisons, annotator guidelines, quality metrics, and interface design.
Examines the AI alignment problem and HHH framework. Explains why training language models to be helpful, harmless, and honest presents fundamental challenges.
Evaluate instruction-tuned LLMs using benchmarks like Alpaca Eval and MT-Bench, human evaluation protocols, and LLM-as-Judge automatic methods.
Covers instruction tuning training with data mixing strategies, loss masking, and hyperparameter selection for effective language model fine-tuning.
Explains how chat templates, prompt formats, and role definitions structure conversations for language model instruction tuning and reliable inference.
Explains how Self-Instruct enables language models to generate their own training data through iterative bootstrapping from minimal human-written seed tasks.
Covers practical techniques for creating instruction-tuning datasets. Topics include human annotation, template-based generation, seed expansion.
Explains how instruction tuning transforms base language models into helpful assistants. Examines format design, data diversity, and quality principles.
Compare LoRA, QLoRA, Adapters, IA³, Prefix Tuning, and Prompt Tuning across efficiency, performance, and memory. Practical guide for choosing PEFT methods.
Explains how adapter layers insert trainable bottleneck modules into transformers for parameter-efficient fine-tuning.
Covers prompt tuning for efficient LLM adaptation. Prepend trainable soft prompts to inputs while keeping models frozen. Scales to match full fine-tuning.
Explains how prefix tuning adapts transformers by prepending learnable virtual tokens to attention keys and values. A parameter-efficient fine-tuning method.
Explains how IA3 adapts large language models by rescaling activations with minimal parameters. Compare IA3 vs LoRA for efficient fine-tuning strategies.
Explains how AdaLoRA dynamically allocates rank budgets across weight matrices using SVD parameterization and importance scoring for efficient model adaptation.
Covers QLoRA for fine-tuning large language models on consumer GPUs. Topics include NF4 quantization, double quantization.
Covers LoRA hyperparameter selection for efficient fine-tuning. Topics include rank, alpha, target modules.
Implement LoRA adapters in PyTorch from scratch. Build modules, inject into transformers, merge weights, and use HuggingFace PEFT for production.
Covers LoRA's mathematical foundations including low-rank decomposition, gradient computation, rank selection.
Explains how LoRA reduces fine-tuning parameters by 100-1000x through low-rank matrix decomposition.
Explains why PEFT is essential for LLMs. Analyze storage costs, training memory requirements, and how adapter swapping enables efficient multi-task deployment.
Covers few-shot fine-tuning techniques for language models. Topics include PET, SetFit, and data augmentation to achieve strong results.
Covers learning rate strategies for fine-tuning transformers. Topics include discriminative fine-tuning, layer-wise decay, warmup schedules, and decay methods.
Why neural networks lose earlier capabilities during fine-tuning, with mitigation methods including EWC, L2-SP regularization, replay, and adapters.
Covers full fine-tuning of pre-trained models. Topics include optimal learning rates, batch sizes, warmup schedules, and gradient accumulation techniques.
Explains how transfer learning enables pre-trained models to adapt to new NLP tasks. Topics include pre-training, fine-tuning, layer representations.
Explains how Switch Transformer simplifies MoE with top-1 routing, capacity factors, and training stability for trillion-parameter language models.
Explains how expert parallelism distributes MoE experts across devices using all-to-all communication, enabling efficient training of trillion-parameter models.
Explains how z-loss stabilizes Mixture of Experts training by penalizing large router logits. Topics include formulation, coefficient tuning.
Explains how auxiliary balancing loss prevents expert collapse in MoE models. Topics include loss formulations, coefficient tuning, and PyTorch implementation.
Explains how load balancing prevents expert collapse in Mixture of Experts models. Examines token fractions, load metrics.
Explains how top-K routing selects experts in MoE architectures. Topics include top-1 vs top-2 trade-offs, implementation details.
Examines gating networks in MoE architectures. Topics include router design, softmax gating, Top-K selection, training dynamics.
Explains how expert networks power Mixture of Experts models. Examines FFN-based experts, capacity factors, expert counts, and transformer placement strategies.
Explains how sparse models decouple capacity from compute using conditional computation and mixture of experts to achieve efficient scaling.
Examines grokking: how neural networks suddenly generalize long after memorization. Topics include phase transitions, theories, and training implications.
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.