WordPiece › Handling Unknown Characters

One practical concern with any subword tokenizer is what happens when it encounters a character that was not present in the training corpus. BPE can produce the [UNK] token for entire unknown words, but in theory a BPE vocabulary trained with single characters as the initial alphabet will always be able to tokenize any text using individual characters as fallback.

WordPiece has the same property: if the vocabulary contains all characters from the training corpus, any new word can always be tokenized character by character, with each character after the first receiving the ## prefix. In the worst case, every character in an unknown word becomes its own token. This character-by-character fallback is not ideal, because individual characters carry very little semantic information on their own. A sequence of character tokens conveys the spelling of a word without providing much help in understanding its meaning. However, it is always preferable to dropping the word entirely, and it allows the model to at least process the input without crashing.

The character fallback has an interesting side effect for sequences from scripts that are completely different from the training language. Suppose BERT's English vocabulary encounters a Greek word. Greek letters are Unicode characters, and if BERT's training corpus included any Greek-script text (even in citations or proper nouns), those characters will be in the vocabulary. Each Greek character becomes its own token. The resulting token sequence conveys the spelling of the Greek word but is extremely long relative to the word length. This makes Greek-script text much more costly in terms of sequence length, which can exhaust BERT's 512-token budget faster than English text.

BERT's vocabulary was trained on English text, so it handles standard ASCII well. It also includes many common characters from other scripts, but encounters difficulty with languages using scripts very different from Latin. This is one reason why multilingual BERT (mBERT) was trained on 104 languages simultaneously, with a larger shared vocabulary of approximately 120,000 tokens to accommodate all those scripts. Even mBERT, however, allocates vocabulary budget unevenly across languages, with high-resource languages getting more tokens and low-resource languages falling back to shorter subword pieces more often. This imbalance is a known weakness of shared multilingual vocabularies.

For truly unseen characters, BERT falls back to [UNK]. This is rare for any language that uses the Latin script but can be frequent for languages using uncommon Unicode ranges that were underrepresented in BERT's training data. Some emoji sequences, mathematical symbols from obscure Unicode planes, and private-use area characters commonly produce [UNK] tokens from standard BERT tokenizers.