Pre-training Data and Scaling

Covers Common Crawl, crawling strategies, robots.txt respect, crawl freshness.

Covers HTML parsing, boilerplate removal, content extraction, trafilatura and similar tools.

Covers language ID models, multilingual document handling, code-switching, language filtering.

Covers exact deduplication, near-duplicate detection, document vs substring dedup, dedup at scale.

Covers MinHash algorithm, Jaccard similarity estimation, MinHash LSH, MinHash implementation.

Covers heuristic filters, perplexity filtering, classifier-based filtering, filter thresholds.

Covers toxicity classifiers, toxicity thresholds, over-filtering risks, toxicity filter evaluation.

Covers PII detection methods, PII removal strategies, PII removal evaluation, privacy preservation.

Covers domain proportions, quality weighting, data mixing experiments, optimal mixing.

Covers synthetic data generation, quality verification, synthetic data diversity, distillation.

Tracing documents through collection, filtering, mixing, training, evaluation, and model release.

Licenses, terms, robots directives, creator consent, jurisdictional uncertainty, and defensible data decisions.

Data statements, source disclosure, influence estimation, attribution limits, and communicating uncertainty.

Generation, filtering, diversity control, verification, curriculum design, and provenance for synthetic corpora.

Feedback loops from model-generated data, tail loss, distribution drift, and mitigations for synthetic ecosystems.

Pre-training audits, sensitive-content review, documentation, approval gates, and post-release traceability.

Covers CLM objective formulation, autoregressive factorization, CLM loss computation, CLM for generation, CLM training data, CLM scaling properties.

Covers MLM objective formulation, masking strategies (15% rule), [MASK] token usage, MLM for understanding, MLM training dynamics.

Covers subword masking problems, whole word masking procedure, WWM implementation, WWM vs random masking, WWM for different tokenizers.

Covers span selection strategies, span length distribution, sentinel tokens, T5-style corruption, span corruption benefits.

Covers prefix LM formulation, prefix LM attention pattern, prefix LM for generation, prefix LM training, UniLM-style objectives.

Covers generator-discriminator setup, replaced vs original detection, RTD efficiency advantages, ELECTRA training procedure, RTD vs MLM comparison.

Covers token deletion, token shuffling, sentence permutation, document rotation, BART-style denoising, combining denoising tasks.

Covers power law definition, log-log linear relationships, power law fitting, power law universality, power law intuition.

Covers loss vs parameters, loss vs data, loss vs compute, Kaplan optimal allocation, Kaplan predictions.

Covers Chinchilla experiments, revised scaling coefficients, optimal tokens per parameter, Chinchilla vs Kaplan, Chinchilla implications.

Covers compute budget allocation, tokens vs parameters ratio, training efficiency, compute-optimal recipes, practical guidelines.

Covers data repetition effects, optimal repetition strategies, data augmentation scaling, synthetic data scaling.

Covers training vs inference compute, inference-optimal models, over-training for efficiency, deployment cost modeling.

Covers loss extrapolation, capability prediction, scaling law uncertainty, prediction reliability, practical forecasting.