409 items · Page 6 of 9
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.
Explains why larger language models sometimes perform worse on specific tasks. Topics include distractor tasks, sycophancy, and U-shaped scaling patterns.
Examines whether LLM emergent capabilities are genuine phase transitions or measurement artifacts.
Explains how chain-of-thought reasoning emerges in large language models. Topics include CoT prompting techniques, scaling behavior.
Explains how in-context learning emerges in large language models. Topics include scale thresholds, ICL vs fine-tuning, induction heads, and meta-learning.
Explains how LLMs suddenly acquire capabilities through emergence. Topics include phase transitions, scaling behaviors, and the ongoing metric artifact debate.
Transform scaling laws into predictive tools for AI development. Topics include loss extrapolation, capability forecasting.
Chinchilla-optimal training can be inefficient for deployment. Covers deliberate over-training, inference workloads, and total cost modeling for LLM systems.
Examines data-constrained scaling for LLMs: repetition penalties, modified Chinchilla laws, synthetic data strategies, and optimal compute allocation.
Explains how DeepMind's Chinchilla scaling laws changed LLM training by proving models should use 20 tokens per parameter for compute-optimal performance.
Explains how power laws govern neural network scaling. Topics include log-log analysis, fitting techniques, and how to predict model performance at any scale.
Explains how mT5 extends T5 to 101 languages using temperature-based sampling, the mC4 corpus, and 250K vocabulary for effective cross-lingual transfer.
Covers BART's denoising pre-training approach including text infilling, token masking, sentence permutation, and how corruption schemes enable generation.
Explains how T5 reformulates all NLP tasks as text-to-text problems. Topics include task prefixes, classification, NER.
Covers compute-optimal LLM training using Chinchilla scaling laws. Topics include the 20:1 token ratio, practical allocation formulas.
Explains how T5 uses span corruption for pre-training. Topics include sentinel tokens, geometric span sampling, the C4 corpus.
Covers T5's encoder-decoder architecture, relative position biases, span corruption pretraining, and text-to-text framework for unified NLP tasks.
Covers LLaMA's architectural choices including RMSNorm, SwiGLU, and RoPE, plus training data strategies.
Examines Qwen's architectural innovations including GQA, SwiGLU activation, and multilingual tokenization.
Explains how the unigram LM tokenizer trains via the EM algorithm, uses Viterbi decoding for segmentation, and enables subword regularization.
Examines Mistral 7B's architectural innovations including sliding window attention, grouped query attention, and rolling buffer KV cache.
Covers GQA, the attention mechanism behind LLaMA 2 and Mistral. Topics include KV head sharing, memory savings, implementation, and quality tradeoffs.
Explains how Multi-Query Attention reduces KV cache memory by sharing keys and values across attention heads, enabling efficient long-context inference.
Explains why traditional word-level approaches fail with diverse text, from OOV words to morphological complexity.
Examines Microsoft's Phi model family and how textbook-quality training data enables small models to match larger competitors.
Explains how WordPiece differs from BPE by using a likelihood objective for merges. Topics include the PMI score, greedy tokenization, ## prefix notation.
Examines LLaMA's core architectural components: pre-norm with RMSNorm for stable training, SwiGLU feed-forward networks for expressive computation.
Repetition, frequency, and presence penalties alter token probabilities to reduce loops. Covers n-gram blocking, tuning, and effects on generated text.
Constrained decoding uses token masks and grammars to produce valid JSON, SQL, and regular-expression matches from language models.
Covers mechanics of autoregressive generation in transformers, including the generation loop, KV caching for efficiency, stopping criteria.
Explains how nucleus sampling dynamically selects tokens based on cumulative probability, solving top-k limitations for coherent and creative text generation.
Top-k sampling limits each generation step to the k most probable tokens. Covers probability renormalization, diversity, failure modes, and tuning.
Explains how large language models learn new tasks from prompt demonstrations without weight updates.
Explains how temperature scaling reshapes probability distributions during text generation, with mathematical foundations, implementation details.
Explains how ELECTRA achieves BERT-level performance with 1/4 the compute by detecting replaced tokens instead of predicting masked ones.
Examines GPT-2's architecture, model sizes, WebText training, and zero-shot capabilities that changed language modeling through scale.
Covers BERT fine-tuning for downstream NLP tasks. Topics include task-specific heads, hyperparameter tuning, and strategies to prevent catastrophic forgetting.
GPT-1 paired generative pretraining with task-specific fine-tuning. Covers its architecture, objective, benchmark results, and role in transfer learning.
Examines GPT-3's 175B parameter architecture, the emergence of few-shot learning, in-context learning mechanisms.
Covers DeBERTa's disentangled attention mechanism that separates content and position representations.
Covers BERT pre-training covering masked language modeling, next sentence prediction, data preparation, hyperparameters.
Explains how ALBERT reduces BERT's size by 18x using factorized embeddings and cross-layer parameter sharing while maintaining competitive performance.
Explains how RoBERTa surpassed BERT using the same architecture by removing Next Sentence Prediction, implementing dynamic masking.
Examines the BERT architecture in detail covering model sizes (Base vs Large), three-layer embedding system, bidirectional attention patterns.
Covers BERT representation extraction with [CLS] token usage, layer selection strategies, pooling methods, and the frozen vs fine-tuned trade-off.
Covers prefix LM, the hybrid pretraining objective that enables bidirectional prefix understanding with autoregressive generation.
Explains how BART trains language models using diverse text corruptions including token deletion, shuffling, sentence permutation.
Explains how replaced token detection trains language models 4x more efficiently than masked language modeling.
Explains how span corruption works in T5, including span selection strategies, geometric distributions, sentinel tokens.
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.