Top-k Sampling › Key Parameters

When using top-k sampling for text generation, the following parameters have the greatest impact on output quality:

• k (top_k): The number of highest-probability tokens to keep. Values of 40-100 work well for most applications. Lower values (10-30) produce more focused, deterministic output suited for factual content. Higher values (100-200) allow more creative diversity but risk occasional incoherent tokens.

• temperature: Scales the logits before computing probabilities. Values below 1.0 sharpen the distribution, making the top tokens more dominant. Values above 1.0 flatten the distribution, spreading probability more evenly. Common values range from 0.7 to 1.2. Temperature is typically applied before top-k selection.

• do_sample: Boolean flag in Hugging Face's generate() method. Must be True to enable any sampling strategy including top-k. When False, the model uses greedy decoding regardless of other parameters.

• max_new_tokens: Limits the number of tokens to generate. Longer generations may accumulate sampling noise, so lower $k$ values can help maintain coherence over extended outputs.

• repetition_penalty: An additional parameter (not part of the core top-k algorithm, but commonly combined with it) that downweights the logits of previously generated tokens. Values slightly above 1.0, such as 1.1 or 1.2, reduce repetition without drastically altering the distribution.