Top-k Sampling › Limitations of Top-k Sampling

While top-k is widely used and practically effective, it has notable limitations that have driven research into more sophisticated decoding strategies. Understanding these limitations is important both for knowing when top-k might fail and for appreciating the motivation behind alternatives like nucleus sampling, beam search, and minimum Bayes risk decoding.

The core tension in top-k sampling is that it uses a fixed count to define the sampling boundary, but the appropriate boundary is fundamentally probabilistic and context-dependent. The model itself does not think in terms of how many tokens should be reasonable; it thinks in terms of probabilities. By imposing a count-based cutoff, we are translating a probabilistic judgment into a structural one, and this translation inevitably loses information.

A second important limitation is that top-k sampling is memoryless at the selection level: it makes the same kind of decision at every generation step, regardless of what has been generated so far. Some researchers have argued that good text generation requires the system to track the "narrative direction" of the text and bias future sampling accordingly. Standard top-k has no such mechanism; it looks only at the current context and applies the same fixed-$k$ rule every time.

The basic limitation is that $k$ is fixed regardless of context. Consider two scenarios:

• High-confidence context: "The president of the United States in 2020 was Donald", where very few tokens are reasonable next.
• Open-ended context: "I think the best way to", where many tokens could reasonably follow.

A fixed $k$ treats both identically. If $k=50$, the first case includes many implausible tokens, since the model might only need $k=5$ to cover all reasonable options. In the second case, $k=50$ might not be enough to capture the full range of plausible continuations, artificially restricting diversity when the model sees many good options.

This context-blindness of fixed-$k$ means that the effective quality of the sampling boundary varies wildly across prompts. For some contexts, $k=50$ is overly generous and includes garbage. For others, it is overly restrictive and excludes good options. You cannot tune a single $k$ to be optimal across all possible contexts, because the contexts themselves vary in how concentrated or dispersed their distributions are.

No single $k$ value works optimally across all situations. Lower $k$ improves quality but reduces diversity. Higher $k$ increases diversity but risks including low-quality tokens. This creates a trade-off when the goal is text that is both high quality and interestingly varied.

The challenge is amplified by the fact that quality and diversity are difficult to measure automatically. Human judgment is the gold standard, but it is expensive and slow. Automatic metrics like perplexity measure how "expected" the text is by the model, which correlates with quality but not with diversity. Metrics like distinct-n (the fraction of distinct n-grams in generated text) measure diversity but not quality. There is no single number that captures the balance well, which makes empirical $k$ tuning difficult.

A more subtle limitation is the hard boundary effect. A token ranked $k$ and a token ranked $k+1$ might have nearly identical probabilities, but one is included and the other is completely excluded. This sharp boundary creates a kind of "exposure bias" where the model never has to deal with its own slightly-below-cutoff predictions during generation. During training, the model saw all tokens including low-probability ones. During inference with top-k, it never generates from that region of its distribution. This mismatch between training and inference can lead to subtle degradation in long-form generation.

Nucleus sampling (top-p), introduced by Holtzman et al. in 2019, addresses the fixed-$k$ limitation by instead choosing the smallest set of tokens whose cumulative probability exceeds a threshold $p$. This allows the sampling set to shrink when the distribution is peaked and expand when it is flat, automatically adapting to each context. The next chapter explores this approach in detail, including the mathematical relationship between top-k and top-p and how to choose $p$ analogously to how you choose $k$.

A third practical limitation of top-k sampling (shared with most sampling methods) is its tendency to produce repetitive text in certain regimes. When the context strongly predicts specific tokens, those tokens dominate the top-$k$ set and keep getting sampled. The model then sees those tokens in the context, which further increases their probability at the next step, creating a self-reinforcing loop.

This repetition problem is mitigated by repetition penalties, which downweight the logits of tokens that have already appeared in the generated sequence. The penalty is applied before top-k selection, reducing the probability of repeated tokens and encouraging more diverse output. Systems like Hugging Face's generate() support a repetition_penalty parameter for this purpose. In practice, combining top-k with a mild repetition penalty (around 1.1 to 1.3) substantially improves the quality of long-form text generation.