Whole Word Masking › WWM for Different Tokenizers
Language AI HandbookAll chapters Whole Word Masking › WWM for Different TokenizersOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Different tokenizers use different conventions for marking subword boundaries, and implementing WWM correctly requires adapting to each convention. The underlying principle is the same across all tokenizer families: identify which tokens belong to the same word and treat them atomically during masking. But the detection mechanism differs based on how the tokenizer encodes word boundary information.
Understanding these conventions matters for implementing WWM from scratch and for debugging unexpected tokenizer outputs. If your tokenizer produces tokens in an unusual format (custom special characters, byte-level encoding, etc.), you need to verify that your boundary detection logic correctly identifies word groups before applying masking.
WordPiece (BERT)
WordPiece uses ## to prefix continuation tokens. A word starts with a token lacking ##, and subsequent ##-prefixed tokens continue it. This convention makes boundary detection straightforward: a token is the start of a new word if and only if it does not begin with ##.
def is_wordpiece_continuation(token):
"""Check if token continues a previous word (WordPiece)."""
return token.startswith("##")Tokens: ['un', '##ha', '##pp', '##iness'] Continuations: [False, True, True, True]
The WordPiece convention is clean and explicit. Every continuation token carries its ## marker, so boundary detection requires no lookahead and no ambiguity. The edge cases are special tokens like [CLS], [SEP], and [PAD], which do not follow the ## convention but also should not be masked. Production implementations filter these out before applying WWM grouping.
One subtlety: tokenizers applied to pre-tokenized text (text that has already been split into words by whitespace or another tokenizer) can produce different results than tokenizers applied to raw text. If you pre-tokenize first and then apply WordPiece to each word separately, the resulting tokens may not have the same ## prefix structure as if you applied WordPiece to the full raw string. Always verify that your tokenization pipeline produces the expected token format before implementing WWM on top of it.
SentencePiece (T5, LLaMA)
SentencePiece uses the special character ▁ (U+2581, a lower one-eighth block) to mark word boundaries. Unlike WordPiece, this character appears at the start of words that follow a space, not as a continuation marker. Tokens without the ▁ prefix are continuations of the previous word:
from transformers import T5Tokenizer
t5_tokenizer = T5Tokenizer.from_pretrained("t5-small")
def is_sentencepiece_word_start(token):
"""Check if token starts a new word (SentencePiece)."""
return token.startswith("▁")
def identify_word_groups_sentencepiece(tokens):
"""Group tokens into words for SentencePiece tokenizers."""
word_groups = []
current_start = 0
for i, token in enumerate(tokens):
if i > 0 and is_sentencepiece_word_start(token):
word_groups.append((current_start, i))
current_start = i
if current_start < len(tokens):
word_groups.append((current_start, len(tokens)))
return word_groupsT5 tokens: ['▁Transformation', 'al', '▁change'] Word: ['▁Transformation', 'al'] Word: ['▁change']
The SentencePiece convention is logically inverted from WordPiece: instead of marking continuations, it marks new starts. A token without ▁ is a continuation of the previous word. This design reflects SentencePiece's unified approach to treating text as a sequence of Unicode characters, where spaces are just another character that happens to signal word boundaries in most languages.
For SentencePiece-tokenized text, the first token in a sequence (or the first token after a special prefix token) typically lacks the ▁ marker regardless of whether it starts a word in the traditional sense. This can create an edge case for the first word in a sequence, which you may need to handle explicitly in your grouping logic.
BPE with GPT-2 Style
GPT-2's BPE tokenizer uses a different convention: it adds Ġ (representing a space character in the tokenizer's byte-level encoding) at the start of tokens that begin a new word. Tokens without the leading Ġ are continuations:
from transformers import GPT2Tokenizer
gpt2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
def identify_word_groups_gpt2(tokens):
"""Group tokens into words for GPT-2 BPE tokenizer."""
word_groups = []
current_start = 0
for i, token in enumerate(tokens):
# Ġ marks word boundary (space before token)
if i > 0 and token.startswith("Ġ"):
word_groups.append((current_start, i))
current_start = i
if current_start < len(tokens):
word_groups.append((current_start, len(tokens)))
return word_groupsGPT-2 tokens: ['Transform', 'ational', 'Ġchange'] Word: ['Transform', 'ational'] Word: ['Ġchange']
GPT-2's byte-level BPE encodes text at the byte level, using a bijective mapping from bytes to Unicode characters to avoid the need for unknown tokens. The Ġ character (Unicode U+0120) represents the byte 0x20, which is a space. So Ġ at the start of a token indicates that a space preceded this token in the original text, which means this token starts a new word. This is logically equivalent to the SentencePiece ▁ approach but uses a different Unicode codepoint.
Universal WWM Function
We can create a universal function that detects the tokenizer type and applies the appropriate grouping logic. This is useful when building tools that need to work across multiple model families without requiring users to specify the tokenizer convention manually.
def identify_word_groups_universal(tokens, tokenizer):
"""
Identify word groups for any common tokenizer type.
Automatically detects WordPiece, SentencePiece, or BPE conventions.
"""
if hasattr(tokenizer, "wordpiece_tokenizer"):
# WordPiece (BERT-style): ## prefix for continuation
return identify_word_groups(tokens)
# Check first few tokens for convention clues
sample = " ".join(tokens[: min(5, len(tokens))])
if "▁" in sample:
# SentencePiece: ▁ prefix for word start
return identify_word_groups_sentencepiece(tokens)
elif "Ġ" in sample:
# GPT-2 BPE: Ġ prefix for word start
return identify_word_groups_gpt2(tokens)
else:
# Fallback: treat each token as separate word
return [(i, i + 1) for i in range(len(tokens))]BERT: ['un', '##ha', '##pp', '##iness'] Groups: [(0, 1), (1, 2), (2, 3), (3, 4)] T5: ['▁un', 'h', 'app', 'iness'] Groups: [(0, 4)] GPT-2: ['un', 'h', 'appiness'] Groups: [(0, 1), (1, 2), (2, 3)]
Each tokenizer produces different subword splits for "unhappiness", but the universal function correctly groups them into a single word for all three. BERT uses ## prefixes to mark the continuation pieces. T5 produces a different subword segmentation using ▁. GPT-2's byte-level BPE produces yet another segmentation, but the Ġ detection correctly identifies the word boundary. In each case, the result is a single group spanning all the subword tokens of "unhappiness", which is exactly what we need for whole word masking.
The fallback case (treating each token as a separate word) is safe but conservative: it effectively disables whole word masking for unknown tokenizer formats. A better fallback for production systems might raise an exception to alert the user that their tokenizer format is not recognized, preventing silent degradation to per-token masking.