Nucleus Sampling › The Problem with Fixed-k Sampling

Top-k sampling works by selecting the $k$ tokens with the highest probabilities and redistributing probability mass among them. This works well when the model's confidence is consistent, but language is anything but consistent.

Consider two scenarios:

1. High certainty: The model predicts "The capital of France is ___" and assigns 95% probability to "Paris". Here, sampling from the top 50 tokens includes 49 tokens that collectively share only 5% of the probability mass, many of which would produce nonsensical completions.

2. Low certainty: The model predicts "I had a wonderful ___" where "day", "time", "experience", "meal", "trip", and dozens of other tokens are all reasonable. With top-k=10, we might exclude perfectly valid continuations.

The core issue is that $k$ is a hyperparameter that cannot adapt to the context. What we really want is to keep all tokens that represent "reasonable" choices, and the most principled way to define "reasonable" is through probability mass.

This problem is more subtle than it first appears. When you set k=50 and hope for the best, you're making an implicit assumption: that every generation step involves roughly the same number of plausible continuations. That assumption is almost always wrong. A story generation task might have high-certainty moments when a character's name is being repeated and low-certainty moments when the next plot event is being chosen. A code generation task might have near-deterministic steps when closing a parenthesis and uncertain steps when choosing between algorithmic approaches. A fixed $k$ handles none of these variations well.

There is also an asymmetry in how top-k fails. When $k$ is too large, you include harmful tail tokens and generation quality degrades. When $k$ is too small, you exclude legitimate options and generation becomes repetitive or stilted. These failure modes pull in opposite directions, and neither extreme is acceptable for open-ended generation. Practitioners often discover this by trying multiple values of $k$ and noticing that the right value changes between task types, prompt styles, and even between sentences in the same passage.

The deeper philosophical issue is that top-k conflates two separate things: the threshold for "plausibility" and the count of plausible items. In most generation contexts, you care about plausibility as measured by probability, not about having exactly $k$ options. When the model's distribution is uniform across 50 tokens, all 50 deserve consideration. When the distribution puts 95% on one token, you probably want just that one. Nucleus sampling honors this by letting the distribution itself determine how many tokens to include.