Preference Evaluation › Key Parameters

The preference evaluation system involves several parameters whose settings materially affect the quality of results:

• k_factor: Controls the sensitivity of Elo rating updates. Higher values (e.g., 32) allow ratings to respond quickly to new information but introduce volatility; lower values (e.g., 16) stabilize ratings but slow adaptation to real improvements. Selection should consider the expected rate of model improvement and the cost of rating volatility versus the risk of missing real changes in quality. In practice, dynamic K-factors that decrease as models accumulate more matches offer the best of both worlds.

• initial_rating: The starting Elo rating for all models (typically 1500), serving as the reference point for relative skill comparisons. While the absolute value is arbitrary, consistency within an evaluation ecosystem is needed for real comparisons. If new models always enter at 1500 regardless of their true strength, initial ratings create temporary distortions that take many matches to correct.

• scale: Conversion factor between Bradley-Terry beta parameters and Elo ratings ($400 / \ln(10) \approx 173.7$), mapping log-odds differences to the standard Elo scale. This parameter determines how much probability changes per Elo point and should be calibrated to match the empirical variance observed in your specific domain. A domain where human annotators have high variability will benefit from a larger scale factor that spreads ratings more widely.

• n_bootstrap: Number of resampling iterations for bootstrap confidence intervals (commonly 1000), balancing computational cost with statistical precision. Higher values give smoother estimates of the sampling distribution but increase computation time linearly. For preliminary results, 500 bootstraps suffices; for publication-quality confidence intervals, 2000 or more is preferable.

• alpha: Significance level for confidence intervals (default 0.05 for 95% intervals), determining the width of uncertainty bounds around rating estimates. Lower alpha values give more conservative bounds appropriate for high-stakes deployment decisions where false positives (incorrectly declaring a new model better) carry significant costs.