WordPiece › The ## Prefix Convention

WordPiece uses a simple but important convention to distinguish between tokens that start a word and tokens that appear in the middle or end of a word. Tokens that can begin a word carry no special marker. Tokens that are continuation pieces (meaning they follow some preceding token within the same word) are prefixed with ##.

This matters because "ing" and "##ing" are different tokens in the vocabulary. The token "ing" might appear as an independent word in some context (rare in English, but the point stands). The token "##ing" unambiguously means "a suffix that attaches to the previous token as part of the same word." Without this distinction, a model would have no way to tell whether the token "ing" was a standalone word or the suffix of a longer word. The ## prefix bakes word boundary information directly into the token identity.

Think of the ## prefix as a grammatical marker analogous to a hyphen in a dictionary entry. When a dictionary entry shows "re-" as a prefix or "-ing" as a suffix, it is communicating that these pieces attach to other words, not that they stand alone. The ## convention does the same thing for subword tokens: it signals attachment to the preceding piece of the same word.

When WordPiece tokenizes the word "playing", it might produce:

When it tokenizes "playing a game", the full output might be:

The ## prefix tells any downstream model (like BERT) that "##ing" is part of the same word as "play", not an independent token. BERT uses this information when determining word boundaries and when performing tasks like named entity recognition that require knowing which tokens belong to the same word. For example, when BERT is used for sequence labeling tasks like NER, the model typically assigns a label to the first token of each word and ignores or copies the label for ## continuation tokens, since a multi-token word like "un##play##ed" should receive a single entity label rather than one per subword.

The convention also matters during decoding. To reconstruct the original text from WordPiece tokens, you concatenate tokens that start with ## directly to the previous token, and insert a space before tokens that do not start with ##. So ["play", "##ing", "a", "game"] reconstructs as "playing a game".

One limitation of the ## convention appears in languages without clear word boundaries. In Japanese, for instance, text is not separated by spaces, so the notion of "word-initial" versus "word-internal" is less clear. In these settings, pre-tokenization strategies that impose pseudo-word boundaries are needed before WordPiece can operate effectively. This is one reason why multilingual models and language-specific models sometimes use different tokenization approaches that do not rely on the ## convention.