How Language Models Work (in Plain English) › Glossary

Token: A chunk of text that a language model processes, which might be a whole word, part of a word, or punctuation. Models break text into tokens before processing it, typically using a vocabulary of 50,000 to 100,000 tokens.

Training: The process where a language model learns to predict text by seeing billions of examples drawn from many sources. During training, the model's parameters are adjusted to make better predictions.

Parameters: The internal values (like millions of tiny knobs) that encode everything a language model has learned. Larger models have more parameters and can typically handle more complex tasks.

Prompt: The input text you give to a language model. The model uses this as context to predict what should come next, generating its response token by token.

Hallucination: When a language model generates plausible-sounding but incorrect information. This happens because models predict based on patterns rather than looking up verified facts.

Neural Network: The mathematical structure underlying language models, consisting of layers that process input tokens. Modern language models use a specific type called transformers.

Attention: A mechanism in transformer models that helps them focus on relevant parts of the input when making predictions. For example, it helps the model connect "it" to "cat" in the sentence "The cat sat on the mat and then it moved."