Volume 2 of 3 · PDF edition

Volume 2: Neural Networks, Attention, and Transformers

The transition from structured prediction and benchmarks to embeddings, sequence-to-sequence learning, attention, transformers, BERT, and GPT.

The complete book remains free to read online. This paid edition is the carefully typeset PDF you keep, with every future update included.

Pages
319 pages
Chapters
34 chapters
Price
$14 one-time
Buy nowLearn more
History of Language AI, Volume Two: Neural Networks, Attention, and Transformers cover

Author and edition details

About the author and Volume 2 PDF edition

Michael Brenndoerfer, author of History of Language AI

Michael Brenndoerfer

Michael has spent more than a decade working across software engineering, data, AI, and business. He writes to understand difficult ideas more deeply and to share what he learns in a clear, practical way.

Report an error or suggest a correction
Edition
Volume 2 PDF 2026.08.2
Published
Last reviewed

Historical through-line

What this volume explains

  • See how shared benchmarks and structured learning made progress measurable.
  • Follow the sequence from word embeddings and neural translation to attention and transformer architectures.
  • Understand why pretraining changed the economics and capabilities of language AI.

Audience and prerequisites

Who this volume is for

Practitioners and students who want the historical path into neural language models and transformers.

Prerequisites: Readable on its own; Volume 1 supplies useful background for the statistical methods referenced here.

Free online preview

Start with “Transformer Architecture (2017)”

Vaswani et al.'s 'Attention Is All You Need' introduced the transformer, replacing recurrence with self-attention and establishing the architecture that would dominate all of NLP

Read the chapter

Exact contents

34 chapters across 3 parts

Structured Learning & Benchmarks

  1. 01Conditional Random Fields (2001)
  2. 02BLEU Metric (2002)
  3. 03Phrase-based SMT & MERT (2003)
  4. 04Neural Probabilistic Language Model (2003)
  5. 05Latent Dirichlet Allocation (2003)
  6. 06ROUGE & METEOR (2004)
  7. 07PropBank (2005)
  8. 08Freebase (2007)

Deep Learning Arrives

  1. 01IBM Watson on Jeopardy! (2011)
  2. 02Deep Learning for Speech Recognition (2012)
  3. 03Wikidata
  4. 04Word2Vec (2013)
  5. 05GloVe & Adam Optimizer (2014)
  6. 06Seq2Seq for MT (2014)
  7. 07Memory Networks (2014)
  8. 08Attention Mechanism (2015)
  9. 09Residual Connections (2015)
  10. 10Layer Normalization (2016)
  11. 11Subword Tokenization & FastText (2016)
  12. 12SQuAD (2016)
  13. 13Neural Information Retrieval
  14. 14Google Neural Machine Translation (2016)
  15. 15WaveNet (2016)

Transformers & Pretraining

  1. 01Transformer Architecture (2017)
  2. 02RLHF Foundations (2017)
  3. 03ELMo & ULMFiT (2018)
  4. 04BERT (2018)
  5. 05GPT-1 & GPT-2 (2018)
  6. 06GLUE & SuperGLUE (2018)
  7. 07XLNet, RoBERTa, ALBERT (2019)
  8. 08XLM (2019)
  9. 09T5 & Text-to-Text Framework (2019)
  10. 10Transformer-XL (2019)
  11. 11BERT for IR (2019)

Volume 2 PDF

Own this 319-page edition

Get the PDF and every future update and erratum for this volume. Your purchase is fully credited toward the complete edition.

  • Carefully typeset PDF
  • Future updates and errata included
  • Portable offline reading
  • Delivered by email and added to My books

Volume 2 of 3

$14

one-time

Version 2026.08.2 · 319 pages · secure checkout via Stripe

Delivered by email · Free updates included

Compare editions

This volume or the complete book?

Volume 2: Neural Networks, Attention, and Transformers

319 pages focused on Structured Learning & Benchmarks and Deep Learning Arrives and Transformers & Pretraining.

$14

The complete History of Language AI

All three volumes in one 968-page PDF. Three distinct volume purchases unlock it automatically; two volumes leave a $6 complete-edition upgrade.

$34

Compare all PDF editions

Also in the series

Explore the other volumes

Volume 1 · 324 pages

Signals, Symbols, and Statistical Methods

Signals & Symbols · The Statistical Turn

Explore Volume 1

Volume 3 · 338 pages

Scaling, Alignment, and Agentic AI

Scaling & Retrieval · Multimodal & Instruction Era · Open Models & Alignment · Agents & Long Context

Explore Volume 3

Version history

Kept current, not frozen in time

Each PDF purchase includes future editions. When the book changes, the updated copy appears in My books at no extra cost.

Current release

Edition 2026.08.2

Editorial improvements.

Earlier releases2

  1. 2026.08.1

    Editorial improvements throughout the book, with clearer explanations, refined presentation, and improved plots and visualizations.

  2. 2026.08.0

    Initial complete-book and three-volume PDF release with perpetual updates.