Benchmark Saturation › Key Parameters

The saturation detection analysis involves several design parameters that require calibration for different contexts.

• saturation_threshold: The accuracy level above which a benchmark is considered potentially saturated (default: 0.90). Higher values delay saturation classification until models achieve near-perfect scores. Setting this threshold requires balancing between catching saturation early and avoiding false positives from temporary plateaus. If set too high (e.g., 0.99), detection may occur too late to be useful; if set too low (e.g., 0.80), healthy benchmarks may be flagged prematurely.

• slope_threshold: The minimum rate of improvement per quarter required to consider the benchmark still informative (default: 0.02). Values below this indicate stagnation. This parameter captures the velocity of progress. Even if accuracy is high, significant positive slope indicates the benchmark still discriminates; conversely, near-zero slope with moderate accuracy suggests the field has converged on similar solutions.

• window: The size of the rolling window for calculating statistics (default: 4 quarters). Larger windows smooth out noise but delay saturation detection. The window size should reflect the typical release cycle of models in the field. For fast-moving areas like large language models, shorter windows (2-3 quarters) may be appropriate; for stable fields, longer windows (1-2 years) prevent false alarms from temporary slowdowns.

• adaptation_rate: The rate at which dynamic benchmarks increase difficulty when models exceed performance thresholds (default: 0.3). Controls how aggressively the benchmark evolves to maintain discriminative power. Higher values create more challenging benchmarks faster but risk overshooting into territory where results are too difficult to interpret; lower values extend lifespan more gradually but may still allow periodic saturation.