WordPiece › WordPiece in BERT

BERT uses a WordPiece vocabulary of approximately 30,000 tokens. The vocabulary was trained on a large English corpus (BookCorpus plus English Wikipedia). Because the training data was English-only, the vocabulary is optimized for English morphology, though BERT models in other languages and multilingual BERT use vocabularies trained on corresponding corpora.

The choice of vocabulary size was a deliberate engineering decision. At 30,000 tokens, the embedding matrix (which maps each token to a dense vector) contains 30,000 rows. For BERT-base with 768-dimensional embeddings, this is about 23 million parameters just for the embedding table, which is roughly 25% of BERT-base's total parameter count. Going much larger would make the embedding table dominate the model. Going much smaller would produce more out-of-vocabulary tokens and longer sequences. The 30,000-token vocabulary represents a practical balance between coverage and compactness.

The BERT tokenizer adds several special tokens that have fixed positions and meanings. These tokens are not just implementation details: they carry semantic meaning within BERT's pre-training framework.

• [CLS]: Added at the beginning of every input sequence. The BERT model uses the representation of this token as a summary of the entire sequence for classification tasks. During pre-training, BERT learns to encode sequence-level information into the [CLS] position via the next-sentence prediction objective.
• [SEP]: Added at the end of each segment in the input. For tasks that take two sentences (like textual entailment), [SEP] separates the two segments and marks where one ends and the other begins.
• [PAD]: Used to pad shorter sequences to a fixed length in batched processing. BERT uses an attention mask to ignore these padding tokens, so their presence does not affect the model's computations beyond memory layout.
• [MASK]: Used during masked language model pre-training to replace tokens that the model must predict. Approximately 15% of tokens are selected for masking, and of those, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged.
• [UNK]: The fallback for any word that cannot be tokenized. In practice this is very rare since single characters are always in the vocabulary.

When tokenizing a sentence for BERT, the full process is:

1. Lowercase the text (for the uncased BERT model) or preserve case (for the cased model)
2. Run basic whitespace and punctuation splitting to produce candidate words
3. Apply WordPiece tokenization to each word
4. Prepend [CLS] and append [SEP]
5. Map each token to its integer ID in the vocabulary

The result is a sequence of integer IDs that BERT processes as its input.

BERT's maximum sequence length of 512 tokens is closely tied to the tokenization. Because subword tokenization expands words into multiple tokens, a sentence of 100 words might produce 130-150 tokens after WordPiece splits compound words and rare morphological forms. This expansion factor is typically 1.1 to 1.5 for English text, meaning the 512-token limit corresponds to roughly 300-450 words. For longer documents, practitioners typically truncate or use chunking strategies, which is an active area of research for long-document understanding.

The bert-base-uncased tokenizer lowercases all input before tokenization. This halves the effective vocabulary space for alphabetic characters (no need for both A and a), allowing the vocabulary budget to be spent on more subword units instead. The trade-off is that case information is lost, which hurts performance on tasks where capitalization carries semantic weight (such as named entity recognition, where "Apple" the company and "apple" the fruit are different). The bert-base-cased variant preserves case and uses a separately trained vocabulary; it performs better on such tasks at the cost of a slightly higher unknown token rate.