Neural Networks, Attention, and Transformers

Lafferty and colleagues introduced CRFs, revolutionizing structured prediction by modeling entire sequences jointly through conditional probability and feature functions, establishing that outputs are interdependent and should be predicted together rather than independently

IBM researchers introduced BLEU, revolutionizing machine translation evaluation by providing the first widely adopted automatic metric that correlated with human judgments, enabling rapid iteration and establishing automatic evaluation as fundamental to language AI development

Phrase-based statistical machine translation extended IBM word-based models to phrase-level learning, capturing idioms and collocations, while Minimum Error Rate Training optimized feature weights to directly maximize BLEU scores, establishing the dominant statistical MT paradigm

Bengio et al.'s first neural LM learned distributed word representations, foreshadowing modern embeddings and deep NLP

Latent Dirichlet Allocation introduced probabilistic topic modeling, enabling unsupervised discovery of thematic structure in large document collections

ROUGE and METEOR automatic evaluation metrics expanded beyond BLEU to better assess summarization and capture semantic similarity in MT evaluation

Added semantic role labels to the Penn Treebank, enabling statistical systems to learn 'who did what to whom'

Freebase launched as a collaborative knowledge base, providing structured data that would later feed retrieval and grounding systems for language models

IBM's Watson question-answering system defeated top human champions on the quiz show Jeopardy!, showcasing that AI could comprehend and answer natural-language questions at a human-expert level

Geoffrey Hinton and colleagues applied deep neural networks to speech recognition, significantly outperforming the then-dominant HMM-based models and dramatically reducing error rates in transcription

Wikidata emerged as a comprehensive collaborative knowledge base, becoming a crucial resource for grounding language models and enabling structured knowledge access

Mikolov's word2vec introduced efficient distributional word embeddings trained on large corpora, establishing vector similarity and the modern era of neural NLP representations

GloVe combined global co-occurrence statistics with local context, while the Adam optimizer enabled stable training of neural networks, both becoming foundational tools

Sutskever's sequence-to-sequence encoder-decoder framework revolutionized neural machine translation and established the template for text generation tasks

Weston et al. introduced neural models with an explicit external memory for QA, prefiguring retrieval-augmented methods

Bahdanau's attention mechanism introduced differentiable alignment in neural MT, enabling models to focus on relevant parts of input and dramatically improving translation quality

ResNet's residual connections from computer vision became standard in deep NLP architectures, enabling training of much deeper networks without degradation

Ba et al.'s layer normalization stabilized training of recurrent and deep networks, becoming a crucial component in transformer and modern LLM architectures

Byte Pair Encoding (BPE) enabled open-vocabulary modeling, while FastText provided robust word vectors with subword information, solving out-of-vocabulary problems

The Stanford Question Answering Dataset established reading comprehension as a flagship benchmark, driving research in language understanding and spawning many QA variants

Neural information retrieval learned semantic representations of queries and documents, enabling meaning-based matching beyond keyword overlap and transforming search systems with dual encoder architectures and dense retrieval methods

Google Translate switched from phrase-based methods to a neural machine translation system, an end-to-end LSTM-based encoder-decoder that produced far more fluent, natural translations than previous statistical models

DeepMind's WaveNet model generated raw audio waveforms for text-to-speech, producing remarkably natural-sounding speech and outperforming prior synthesis systems by modeling audio directly with a neural network

Vaswani et al.'s 'Attention Is All You Need' introduced the transformer, replacing recurrence with self-attention and establishing the architecture that would dominate all of NLP

Christiano et al.'s work on learning from human preferences established foundations for reinforcement learning from human feedback, later crucial for aligning language models

Context-sensitive embeddings from ELMo and transfer learning from ULMFiT demonstrated that pretraining on large corpora dramatically improved downstream tasks, launching the transfer learning era

Devlin et al.'s BERT with masked language modeling and bidirectional pretraining revolutionized NLP, causing leaderboard performance to jump overnight across all benchmarks

OpenAI's GPT models demonstrated that autoregressive pretraining could produce powerful generative models, with GPT-2 showing surprising zero-shot capabilities

The General Language Understanding Evaluation benchmarks established standardized multi-task evaluation, enabling systematic comparison of language understanding systems

Refinements to BERT including permutation language modeling (XLNet), optimized training (RoBERTa), and parameter efficiency (ALBERT) pushed pre-training performance further

Cross-lingual pretraining with translation language modeling enabled strong zero-/few-shot transfer across languages

Google's T5 unified all NLP tasks as text-to-text transformations, simplifying model architecture and training while achieving strong performance across diverse tasks

Transformer-XL introduced segment-level recurrence and relative positional encodings, enabling transformers to process longer sequences more effectively

BERT-based cross-encoder re-rankers revolutionized information retrieval, dramatically improving ranking quality and establishing neural reranking as standard practice