How Language Models Work (in Plain English) › Breaking Text into Tokens

Before we go further, let's clarify what models work with. When you type a sentence, the model doesn't see individual letters. Instead, it breaks text into tokens (chunks that might be whole words, parts of words, or even punctuation).

For example, the sentence "The AI agent is helpful!" might be split into tokens like this:

Notice that spaces are often attached to words, and common words usually get their own token. Rarer or longer words might be split into pieces. For instance, "unhelpful" might become three separate tokens: "un", "help", and "ful".

Why use tokens instead of letters or whole words? It's a practical compromise that balances efficiency with flexibility. If models worked with individual letters, they'd need to learn spelling from scratch (a huge waste of computational power). On the other hand, using whole words would create a massive vocabulary with millions of entries, including countless rare words that appear only once or twice. Tokens strike a sweet spot: a vocabulary of around 50,000 to 100,000 tokens can represent virtually any text efficiently, breaking rare words into familiar pieces while keeping common words intact.

You don't need to worry about tokenization most of the time. Just know that when we say the model "predicts the next word," it's really predicting the next token. For simplicity, we'll keep saying "word" in this chapter, but technically it's "token."