Language AI Handbook Articles – Page 7

Michael Brenndoerfer

Language AI Handbook

Content from the Language AI Handbook, covering natural language processing, language models, and AI-powered language applications.

409 items · Page 7 of 9

Community

Learn with the community

Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.

Keep your place across books and articles
Ask questions and take part in discussions
Read with fewer interruptions

Join the community

Free to join · Takes less than a minute

Why

I created this space so readers can learn together, ask questions, and make sense of difficult ideas.

From readers1 / 12

Data, Analytics & AILanguage AI HandbookMachine Learning

Whole Word Masking for BERT Pretraining

Jul 8, 2025·54 min read

Explains how Whole Word Masking improves BERT pre-training by masking complete words instead of subword tokens.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Masked Language Modeling

Jul 7, 2025·49 min read

Explains how masked language modeling enables bidirectional context understanding. Topics include the MLM objective, 15% masking rate, 80-10-10 strategy.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Memory Augmentation for Transformers

Jul 6, 2025·51 min read

Explains how memory-augmented transformers extend context beyond attention limits using external key-value stores, retrieval mechanisms.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Causal Language Modeling: The Foundation of Generative AI

Jul 5, 2025·42 min read

Explains how causal language modeling trains AI to predict the next token. Topics include autoregressive factorization, cross-entropy loss, causal masking.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Recurrent Memory: Extending Transformer Context

Jul 4, 2025·68 min read

Explains how Transformer-XL uses segment-level recurrence to extend effective context length.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Position Interpolation

Jul 3, 2025·49 min read

Explains how Position Interpolation extends transformer context windows by scaling position indices to stay within training distributions.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Context Length Challenges

Jul 2, 2025·39 min read

Examines the fundamental barriers limiting transformer context windows, including quadratic memory scaling, position encoding extrapolation failures.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Attention Sinks and StreamingLLM for Long Generation

Jul 1, 2025·57 min read

Explains why the first tokens in transformer sequences absorb excess attention weight, how this causes streaming inference failures.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

NTK-aware Scaling: Extending Context Length in LLMs

Jun 29, 2025·57 min read

NTK-aware scaling extends transformer context windows by adjusting rotary frequencies. Covers position preservation, longer sequences, and tradeoffs.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

FlashAttention Implementation

Jun 28, 2025·51 min read

Covers FlashAttention's tiled computation and online softmax algorithms. Topics include GPU memory hierarchy, CUDA kernel basics.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

FlashAttention Algorithm

Jun 27, 2025·45 min read

Explains how FlashAttention achieves 2-4x speedups by restructuring attention computation. Topics include GPU memory hierarchy, tiling for SRAM.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

YaRN: Extending Context Length

Jun 26, 2025·58 min read

Explains how YaRN extends LLM context length through wavelength-based frequency interpolation and attention temperature correction.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Linear Attention: Breaking the Quadratic Bottleneck

Jun 25, 2025·42 min read

Explains how linear attention achieves O(nd²) complexity by replacing softmax with kernel functions.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Sliding Window Attention

Jun 24, 2025·43 min read

Explains how sliding window attention reduces transformer complexity from quadratic to linear.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Longformer: Efficient Attention for Long Documents

Jun 23, 2025·55 min read

Explains how Longformer combines sliding window and global attention to process documents of 4,096+ tokens with O(n) complexity instead of O(n²).

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Sparse Attention Patterns: Local, Strided

Jun 22, 2025·61 min read

Implement sparse attention patterns including local windows, strided attention, and block-sparse methods.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

BigBird: Sparse Attention

Jun 21, 2025·40 min read

BigBird combines sliding-window, global, and random attention to reach linear complexity. Covers its guarantees and use with long documents.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Global Tokens for Long-Range Attention

Jun 20, 2025·51 min read

Global tokens create communication hubs in sparse attention, reducing path length and helping efficient transformers exchange information across long documents.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Quadratic Attention Bottleneck

Jun 19, 2025·48 min read

Explains why self-attention has O(n²) complexity, how memory and compute scale quadratically with sequence length.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Encoder-Decoder Architecture: Cross-Attention

Jun 18, 2025·42 min read

Covers encoder-decoder transformer architecture that powers T5 and machine translation. Topics include cross-attention mechanism.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Decoder Architecture: Causal Masking

Jun 17, 2025·38 min read

Covers decoder-only transformers powering GPT, Llama, and modern LLMs. Topics include causal masking, autoregressive generation, KV caching.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Transformer Architecture Hyperparameters: Depth, Width

Jun 16, 2025·39 min read

Design transformer architectures by understanding the key hyperparameters: model depth, width, attention heads, and FFN dimensions.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Cross-Attention: Connecting Encoder and Decoder

Jun 15, 2025·59 min read

Covers cross-attention, the mechanism that bridges encoder and decoder in sequence-to-sequence transformers.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Weight Tying in Transformer Embeddings

Jun 14, 2025·53 min read

Explains how weight tying reduces transformer parameters by sharing the input embedding and output projection matrices.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Encoder Architecture

Jun 13, 2025·45 min read

Explains how encoder-only transformers like BERT use bidirectional self-attention for text understanding.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Gated Linear Units: The FFN Architecture Behind Modern LLMs

Jun 12, 2025·44 min read

Explains how GLUs transform feed-forward networks through multiplicative gating. Topics include SwiGLU, GeGLU, and the parameter trade-offs that power LLaMA.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

FFN Activation Functions: ReLU, GELU

Jun 11, 2025·52 min read

Compare activation functions in transformer feed-forward networks: ReLU's simplicity and dead neuron problem, GELU's smooth probabilistic gating for BERT.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Transformer Blocks: Encoder and Decoder Assembly

Jun 10, 2025·44 min read

Assemble transformer blocks by combining residual connections, normalization, attention, and feed-forward networks.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Layer Normalization: Stabilizing Transformer Training

Jun 9, 2025·52 min read

Explains how layer normalization enables stable transformer training by normalizing.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Feed-Forward Networks in Transformers: Architecture

Jun 8, 2025·56 min read

Explains how feed-forward networks provide nonlinearity in transformers, with 2-layer architecture, 4x dimension expansion, parameter analysis.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Pre-Norm vs Post-Norm

Jun 7, 2025·46 min read

Compare pre-norm and post-norm transformer blocks. Covers layer-normalization placement, gradient flow, training stability, and model depth.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Residual Connections and Gradient Flow in Transformers

Jun 6, 2025·46 min read

Explains how residual connections solve the vanishing gradient problem in deep networks. Topics include the math behind skip connections, gradient highways.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

RMSNorm: Efficient Normalization for Modern LLMs

Jun 5, 2025·58 min read

Covers RMSNorm, the simpler alternative to LayerNorm used in LLaMA, Mistral, and modern LLMs.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbooknlp

Sinusoidal Position Encoding

Jun 4, 2025·52 min read

Covers sinusoidal position encoding, the deterministic method that gives transformers positional awareness.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

The Position Problem

Jun 3, 2025·47 min read

Explains why self-attention is blind to word order and what properties positional encodings need.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Rotary Position Embedding (RoPE)

Jun 2, 2025·54 min read

Explains how RoPE encodes position through vector rotation, making attention scores depend on relative position.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Query, Key, Value: The Foundation of Transformer Attention

Jun 1, 2025·40 min read

Explains how QKV projections enable transformers to learn flexible attention patterns through specialized query, key, and value representations.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Position Encoding Comparison: Sinusoidal, Learned, RoPE

May 31, 2025·38 min read

Compare transformer position encoding methods including sinusoidal, learned embeddings, RoPE, and ALiBi.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Relative Position Encoding

May 30, 2025·49 min read

Explains how relative position encoding improves transformer generalization by encoding token distances rather than absolute positions.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Learned Position Embeddings

May 29, 2025·48 min read

How GPT and BERT encode position through learnable parameters. Topics include embedding tables, position similarity, interpolation techniques.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

ALiBi: Attention with Linear Biases for Position Encoding

May 28, 2025·52 min read

Explains how ALiBi encodes position through linear attention biases instead of embeddings. Topics include head-specific slopes, extrapolation properties.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Multi-Head Attention

May 27, 2025·44 min read

Explains how multi-head attention runs multiple attention operations in parallel, enabling transformers to capture diverse relationships like syntax, semantics.

Open notebook

Data, Analytics & AILanguage AI HandbookMachine Learning

Attention Complexity: Quadratic Scaling, Memory Limits

May 26, 2025·62 min read

Self-attention has O(n²d) compute and quadratic memory growth. Covers when sparse, linear, and other efficient attention variants reduce those costs.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Scaled Dot-Product Attention: The Core Transformer Mechanism

May 25, 2025·48 min read

Covers scaled dot-product attention: how queries and keys compute similarity, why dividing by sqrt(d_k) prevents softmax saturation.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Attention Masking: Controlling Information Flow

May 24, 2025·52 min read

Covers attention masking techniques including padding masks, causal masks, and sparse patterns.

Open notebook

Data, Analytics & AIMachine LearningLanguage AI Handbook

Self-Attention Concept

May 23, 2025·50 min read

Self-attention relates every token to every other token in a sequence. Covers queries, keys, values, contextual embeddings, masking, and computational cost.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Beam Search: Decoding and Sequence Generation

May 22, 2025·48 min read

Explains how beam search balances quality and compute in sequence generation, covering greedy decoding limits, length normalization, diverse beam search.

Open notebook

Data, Analytics & AISoftware EngineeringMachine LearningLanguage AI Handbook

Teacher Forcing: Training Seq2Seq with Ground Truth Context

May 21, 2025·45 min read

Teacher forcing trains sequence-to-sequence models with ground-truth context. Covers exposure bias, scheduled sampling, and alternatives based on REINFORCE.

Open notebook