GSM8K › Key Evaluation Parameters

The key parameters for GSM8K evaluation are:

• num_shots: Number of few-shot examples to include in the prompt. More examples give stronger guidance but increase prompt length and computational cost. The typical sweet spot is 4-8 examples, balancing demonstration quality against context window utilization. Above 8 examples, gains become marginal on most problems.
• sample_size: Number of test examples to evaluate. Larger samples yield more reliable accuracy estimates but require more computation time. Evaluating the full 1,000-problem test set is standard for publication-quality comparisons; smaller samples (100-200) suffice for rapid iteration during development.
• num_samples (for self-consistency): Number of reasoning paths to sample and aggregate. Higher values improve accuracy through better consensus but increase computational cost linearly. Diminishing returns typically appear after 5-10 samples, with accuracy improvements falling below 0.5% per additional sample beyond that point.
• temperature: The sampling temperature used when generating diverse reasoning chains for self-consistency. Too low (approaching greedy decoding) produces insufficiently diverse paths that all fail at the same steps; too high produces incoherent chains. Values between 0.5 and 0.8 typically work well for self-consistency on GSM8K.
• random_state: Seed value for reproducibility. Controlling randomness ensures that performance comparisons across different model configurations reflect true capability differences rather than prompt selection variance.