Decoder Language Models and Capabilities

Covers GPT-1 architecture, GPT-1 pre-training, GPT-1 fine-tuning approach, GPT-1 transfer learning, GPT-1 historical significance.

Covers GPT-2 model sizes, GPT-2 architectural changes, zero-shot task performance, GPT-2 training data (WebText), GPT-2 generation quality.

Covers GPT-3 scale (175B), few-shot prompting discovery, in-context learning analysis, GPT-3 capabilities, GPT-3 limitations.

Covers ICL phenomenon, ICL vs fine-tuning, example selection strategies, ICL scaling behavior, ICL theoretical understanding.

Covers generation procedure, KV caching for efficiency, generation stopping criteria, generation speed optimization, generation code implementation.

Covers temperature scaling, temperature effects on distribution, temperature selection guidelines, temperature vs quality trade-off.

Covers top-k truncation, k selection strategies, top-k limitations, top-k implementation, combining with temperature.

Covers top-p formulation, cumulative probability threshold, nucleus sampling benefits, p selection guidelines, nucleus vs top-k.

Covers repetition in generation, repetition penalty formulation, frequency penalty, presence penalty, n-gram blocking.

Covers grammar-guided generation, JSON schema constraints, regex constraints, constrained beam search, constrained sampling.

Covers LLaMA design philosophy, LLaMA architectural choices, LLaMA training data, LLaMA efficiency, LLaMA significance.

Covers pre-norm with RMSNorm, SwiGLU FFN, RoPE implementation, component interactions, implementation details.

Covers GQA motivation, GQA formulation, KV head grouping, GQA memory savings, GQA vs MHA performance, GQA implementation.

Covers MQA extreme sharing, MQA memory benefits, MQA quality trade-offs, MQA for inference, MQA vs GQA.

Covers Mistral design choices, sliding window attention, Mistral efficiency, Mistral performance, Mistral vs LLaMA.

Covers Qwen architectural choices, Qwen training approach, Qwen multilingual capabilities, Qwen variants.

Covers Phi design philosophy, textbook-quality data, Phi training approach, Phi efficiency, small model capabilities.

Covers emergence definition, phase transitions, emergence examples, emergence mechanisms, emergence debate.

Covers ICL emergence curves, ICL vs fine-tuning scaling, ICL mechanism hypotheses, ICL as meta-learning.

Covers CoT emergence observations, CoT elicitation, CoT scaling behavior, CoT mechanism theories.

Covers discontinuous metrics, accuracy threshold effects, smooth underlying capabilities, re-examining emergence claims.

Covers inverse scaling phenomena, distractor tasks, sycophancy scaling, inverse scaling prize findings.

Covers grokking phenomenon, grokking in arithmetic, grokking mechanism theories, grokking phase transitions, practical implications.