Encoder, Multilingual, and Translation Models

Covers BERT model sizes, BERT layer configuration, BERT embedding layers, BERT attention patterns, BERT output representations.

Covers pre-training data preparation, MLM implementation, NSP task design, pre-training hyperparameters, pre-training duration.

Covers classification fine-tuning, sequence labeling fine-tuning, question answering fine-tuning, fine-tuning hyperparameters, catastrophic forgetting.

Covers [CLS] token usage, layer selection strategies, pooling strategies, BERT as feature extractor, frozen vs fine-tuned representations.

Covers dynamic masking, NSP removal, larger batches, more data, RoBERTa training recipe, RoBERTa vs BERT performance.

Covers factorized embeddings, cross-layer parameter sharing, sentence order prediction, ALBERT efficiency, ALBERT performance trade-offs.

Covers generator training, discriminator training, RTD objective, ELECTRA sample efficiency, ELECTRA scaling, ELECTRA fine-tuning.

Covers disentangled attention formulation, enhanced mask decoder, DeBERTa position encoding, DeBERTa improvements, DeBERTa-v3 advances.

Covers T5 encoder-decoder design, T5 attention patterns, T5 model sizes, T5 relative positions, T5 implementation.

Covers span corruption procedure, sentinel tokens, corruption rate, T5 pre-training data, T5 training scale.

Covers task prefixes, classification as generation, NER as generation, QA as generation, task formatting examples.

Covers BART encoder-decoder, BART attention configuration, BART vs T5 comparison, BART model sizes.

Covers token masking, token deletion, text infilling, sentence permutation, document rotation, objective combinations.

Covers mT5 training data, language sampling, cross-lingual transfer, mT5 vs T5 performance, multilingual tokenization.

Writing systems, morphology, word order, language families, and why English-centric assumptions fail.

Fertility, script coverage, shared vocabularies, byte models, and unequal token costs across languages.

Sampling temperatures, capacity allocation, transfer-interference trade-offs, and multilingual mixture design.

Shared representations, zero-shot transfer, alignment objectives, adapters, and transfer diagnostics.

Data scarcity, community participation, transliteration, active learning, and responsible evaluation.

Language identification, mixed scripts, code-switched generation, evaluation, and deployment failure modes.

Translationese, construct validity, cultural knowledge, local harms, and evaluation led by native speakers.

Word alignment, phrase tables, language models, log-linear decoding, and the ideas inherited by neural MT.

Encoder-decoder translation, attention, transformer MT, training objectives, and decoding.

Bitext mining, alignment, filtering, back-translation, synthetic parallel data, and contamination checks.

Multilingual transfer, mixture-of-experts translation, zero-shot directions, and capacity bottlenecks.

Learned metrics, adequacy, fluency, terminology, human evaluation, and metric failure modes.

Terminology consistency, discourse context, pronouns, document memory, and long-form evaluation.

Cascaded and end-to-end systems, latency, speaker preservation, prosody, and multilingual safety.