Efficient, Long-Context, and Alternative Architectures

Covers O(n²) memory analysis, O(n²) compute analysis, attention matrix size, practical sequence limits, bottleneck visualization, motivation for efficiency.

Covers local attention windows, strided attention patterns, block-sparse attention, combining sparse patterns, sparse attention implementation.

Covers sliding window formulation, window size selection, dilated sliding windows, sliding window for long sequences, Mistral-style windowed attention.

Covers CLS token global attention, learned global tokens, global-local attention mixing, global token count, implementation strategies.

Covers Longformer attention pattern, global attention configuration, Longformer complexity, Longformer for documents, Longformer implementation.

Covers BigBird attention pattern, random attention benefits, BigBird theoretical guarantees, BigBird vs Longformer, BigBird applications.

Covers softmax attention reformulation, kernel feature maps, linear complexity attention, linear attention limitations, Performer and variants.

Covers GPU memory hierarchy, tiling for SRAM, online softmax computation, recomputation strategy, FlashAttention complexity, FlashAttention benefits.

Covers CUDA kernel basics, memory access patterns, FlashAttention-2 improvements, using FlashAttention in PyTorch, FlashAttention limitations.

Covers training sequence length limits, attention memory scaling, position encoding extrapolation, long-range dependency learning, evaluation challenges.

Covers linear position scaling, interpolation vs extrapolation, position interpolation implementation, fine-tuning for longer context, interpolation limitations.

Covers RoPE frequency analysis, high-frequency preservation, NTK-aware formula, dynamic NTK scaling, NTK vs linear interpolation.

Covers YaRN motivation, attention scaling factor, YaRN formula, YaRN training requirements, YaRN vs alternatives.

Covers attention sink phenomenon, StreamingLLM approach, sink token design, streaming inference, infinite context generation.

Covers memory network concepts, memory retrieval mechanisms, memory writing and updating, memory-augmented transformers, Memorizing Transformers.

Covers Transformer-XL approach, segment-level processing, recurrent state passing, relative position in recurrence, recurrent memory limitations.

Structured state spaces, selective state updates, Mamba-style sequence modeling, and linear-time claims.

Jamba-style hybrids, layer allocation, memory-throughput trade-offs, and architecture ablations.

Bytes, learned patches, dynamic segmentation, and the efficiency and robustness trade-offs of removing fixed tokenizers.

Discrete diffusion, denoising objectives, parallel refinement, controllability, and likelihood evaluation.

Generating multiple tokens per step, speculative blocks, verification, and quality-latency trade-offs.

Matched-compute experiments, hardware efficiency, memory scaling, long-context quality, and benchmark leakage.