Datasets, Benchmarks, and Experimental Design › Ablations, Baselines, and Statistical Power

We now have a training set and the right held-out unit. The remaining questions are: what should you compare against, what does an ablation establish, and how many independent replicates do you need before a difference means anything?

A baseline defines a comparison, not automatically an isolated causal contribution. Specify its inputs, fitting and tuning procedure, and resource scope before interpreting the contrast.

• Simple dynamics baseline. Predict the next state from a scaled last action or fit a low-order linear model. The contrast measures improvement over that specified predictor; it does not separate capacity, nonlinearity, optimization, and feature effects unless those are controlled.
• Action-agnostic predictor. Fit a matched predictor that omits actions. Its contrast measures the incremental predictive value of supplied action information under the protocol. It does not by itself establish how an already-trained conditioned model uses actions. Controlled action changes or interventions address that different question, with support and identification caveats.
• Well-tuned feedback or model-free comparator. Compare control systems under a declared interaction budget. Architecture, optimization, and representation may still differ, so this is a whole-system comparison. Matched component ablations or controlled model substitutions can support more specific contribution claims; identified observational analysis or a justified structural derivation can provide other routes.
• True-dynamics oracle. Give a planner the exact simulator as a privileged diagnostic. Its achieved return is not generally an upper bound when search is approximate, the horizon is short, or information access differs. A solved optimum under the same constraints is a different object and can provide a bound when its assumptions are satisfied.

Giving a planner true state or true dynamics changes its information access and must be disclosed. It can help diagnose model-versus-search limitations, but the diagnosis is conditional on the planner and evaluation protocol. For example, a one-step greedy planner can take an immediate reward of 1 instead of investing now for reward 10 on the next step, even with exact dynamics. Suppose the model also predicts rewards and incorrectly assigns the investment an immediate reward of 2 instead of its true reward of 0. The biased reward prediction induces investment and an actual two-step return of 10. Dynamics bias alone cannot change this one-step ranking if the current-state reward is known exactly. The true-model planner's realized two-step return of 1 is not an upper bound on that result; the true constrained optimum is a separate calculation. Do not present privileged access unavailable at deployment as an ordinary deployable competitor.

A useful heuristic when reading your own results draft: for each row of a results table, ask what a skeptical reader would say if this baseline were absent. If the claim depends on that comparison, keep the baseline in the main table. If removing it leaves the claim unchanged, the comparison can move to an appendix. This keeps the main table focused on comparisons that constrain the claim.

Two questions get conflated under the heading of "fair comparison":

1. Declared resource caps. Each method gets the specified limits relevant to the question, such as environment interactions or training wall-clock time. This asks what each attained under those caps. Equal gradient-step counts do not imply equal compute when batch sizes or per-step work differ.
2. Reasonably tuned performance. Each method is reported at the best validated configuration found within a declared search procedure. This asks what that search attained, not what the method could achieve under every possible training and tuning procedure.

Both questions are legitimate. Matching parameter counts alone does not establish equal training compute, but it can answer a parameter-count-constrained question. For weight-memory constraints, also declare parameter precision and storage. For total deployment-memory constraints, measure or bound runtime state, caches, activations and buffers under the stated input and batch scope. Disclose training cost separately. You need not equalize every resource simultaneously; name the constraint being compared. Unequal tuning effort is part of the attained system contrast, not evidence of an isolated algorithmic effect.

The minimum disclosure is: what was searched, over what space, how it was selected, and how many configurations were tried for each method. Searching 200 configurations consumes compute and can make the winning validation estimate optimistic. A final test independent of that selection need not inherit the validation winner's bias. Report the search and preserve the fitting, selection, and testing boundaries.

One implication that papers sometimes miss: if you tuned method A over 200 configurations and method B over 20, then even with equal compute budgets the comparison is asymmetric in search effort. The honest fixes are to give both methods search budgets of the same order, to report the best-config performance and the default-config performance side by side, or to explicitly declare that the comparison is between one method's tuned configuration and another method's default configuration. Any of those is fine; silently mixing them is not.

The word "ablation" covers two experiments that answer different questions.

• Removal ablation. Take a trained model and delete or mask a component at inference. This answers: how much does the trained system rely on this component?
• Retrained ablation. Train without the component from scratch using a declared training and tuning procedure. This estimates its contribution to the performance achieved by that procedure, not everything either architecture could learn under all procedures.

These can disagree. A component can help optimization during training yet be dispensable at inference. Conversely, a trained system can rely on a component while a separately retrained system compensates for its absence. Label the intervention and training procedure clearly; these possibilities do not establish a frequency of disagreement.

A second issue is confounding. If removing a component also changes the parameter count, training budget, or optimization difficulty, the ablation measures that combination. A one-factor-at-a-time design that leaves combinations unobserved cannot generally identify their interaction without additional assumptions. For two binary components, evaluating all four combinations is a simple factorial design for estimating the interaction as a difference of differences. Replication is still needed to estimate uncertainty. Additional combinations and replicates consume runs; state which interaction the design can identify rather than assuming every ablation reveals a universal mechanism.

A third issue is identifiability. The effect of a component depends on the rest of the system. An ablation does not automatically identify a universal causal mechanism; it identifies the role of that component in this architecture, at this scale, on this data. Writing "the recurrence is essential for memory" after one ablation is a much stronger claim than the experiment supports. A more defensible phrasing would be: "In our architecture, at this scale, on this suite, removing recurrence at inference degraded held-out error by X, and retraining without recurrence degraded it by Y." The narrower statement travels well; the broader one does not.

Finally, evaluating all subsets of six binary components gives $2^6=64$ candidate configurations and consumes a search budget. Selecting the largest noisy selection-set estimate can make that estimate optimistic; the candidate count alone does not determine the magnitude of the bias. A final test independent of that selection need not inherit it. Keep the final test set separate, and report the full search, including candidates other than the winner.

Consider an illustrative experiment with:

• 12 independent training runs per method,
• 30 evaluation episodes per run,
• 200 transitions per episode,
• 20 tasks.

How many independent replicates do you have? It depends entirely on what you are claiming.

Ten thousand transitions evaluated under one learned model are not ten thousand training replicates. They provide within-run observations, with a dependence structure determined by their collection. A transition-level interval cannot substitute for an interval over training procedures. It omits run variability if that variability is present; whether the omission dominates precision is an empirical question. The illustrative ranges in the table are not literature frequency estimates or required seed counts.

There are two kinds of uncertainty, and they are not interchangeable:

• Conditional uncertainty. Given a fixed fitted model, how variable are the evaluation outcomes? Independently sampled evaluation episodes estimate that variation. More such episodes can reduce Monte Carlo error for the conditional mean under finite-variance assumptions, but their cost depends on the environment and protocol.
• Procedure-level uncertainty. How variable is the outcome across the sources randomized in the target procedure? State whether the dataset, initialization, minibatch order, and model-selection process are fixed or varied. Independent fits conditional on one dataset estimate a different uncertainty from fresh-data fits that rerun selection.

If independent training runs are evaluated on identical fixed cases, inference on the run-level differences is conditional on those cases unless the analysis also models case sampling. To target fresh cases as well as fresh runs, describe the case population and sampling hierarchy. Shared cases create a crossed dependence structure rather than separate nested cases for every run. Accounting for case sampling can add a variance component, but does not guarantee that every estimated interval is wider than its fixed-case counterpart.

Pairing can reduce variance in a comparison when matched outcomes have positive covariance. The variance identity states that condition and also shows when pairing can hurt.

Let the experimental unit be the independent replicate, and suppose for replicate $i$ you observe $Y_{A,i}$ under method A and $Y_{B,i}$ under method B, on the same cases. Define the paired difference

$$D_i = Y_{A,i} - Y_{B,i},$$

where:

• $i$: the index over the independent replicate (for example, the training run)
• $Y_{A,i}$: the metric for replicate $i$ under method A
• $Y_{B,i}$: the metric for the same replicate $i$ under method B, evaluated on the same matched cases

For an error metric, positive $D_i$ means method A has larger error. For a return metric, positive instead means A has larger return; the algebra does not decide which direction is desirable. Write $\bar D=n^{-1}\sum_{i=1}^n D_i$ for the average of $n$ replicate differences. Variance measures how outcomes vary across repetitions; covariance measures the shared movement of two outcomes. If the $D_i$ are independent and identically distributed with finite variance $\sigma_D^2$, then

$$\operatorname{Var}\!\left(\bar D\right) = \frac{\sigma_D^2}{n}.$$

Uncorrelated equal-variance differences also suffice for this variance identity. Independence is a stronger assumption used in the subsequent interval analysis; the variance identity alone does not establish its distributional validity.

To expand $\sigma_D^2$ in terms of the individual outcomes, assume both $Y_A$ and $Y_B$ have finite second moments. Finite variance of their difference alone does not ensure that the marginal variances and covariance exist. Under the marginal-moment condition,

$$\sigma_D^2 = \operatorname{Var}(Y_A) + \operatorname{Var}(Y_B) - 2\operatorname{Cov}(Y_A, Y_B),$$

so, retaining the preceding assumptions for the replicate differences and these finite marginal second moments,

$$\operatorname{Var}\!\left(\bar D\right) = \frac{\operatorname{Var}(Y_A) + \operatorname{Var}(Y_B) - 2\operatorname{Cov}(Y_A, Y_B)}{n}.$$

Three consequences follow.

1. Positive covariance helps. At the same $n$ and marginal outcome variances, positive covariance makes the variance of the paired mean difference smaller than an independent-sample comparison. Matching can produce this benefit, but does not establish the covariance's sign or size. Estimated interval widths also depend on the interval procedure and degrees of freedom.
2. Negative covariance hurts. If the methods trade off, for example when one succeeds precisely where the other fails, pairing can be worse than not pairing.
3. Matching must be defined. A seed number alone does not guarantee corresponding draws across different implementations. Feedback policies can follow different state-action paths while still sharing initial conditions, task difficulty, or exogenous disturbances. Path divergence does not force outcome covariance to zero. Measure the covariance or the paired-difference variability rather than assuming the benefit vanishes. Valid interval inference additionally needs the stated independence and distributional assumptions across replicate pairs.

Pairing removes shared outcome variation, which can include variation caused by shared training-data draws or coupled initialization. If $Y_{A,i}=U_i+\epsilon_{A,i}$ and $Y_{B,i}=U_i+\epsilon_{B,i}$, the shared $U_i$ cancels in $D_i$ even if it came from training. Unshared variation remains. More independent runs are needed to resolve that residual uncertainty when it is large; matched evaluation cases alone do not guarantee a small residual.

We now run the second experiment. The design is deliberately structured the way a real comparison should be:

• 12 independent runs. Each run simulates a fresh cohort with its own RNG stream, so the intercepts, noise, and actions differ.
• Within each run, a session-grouped split gives 28 training sessions and 12 held-out sessions.
• Both models are fit on the same training rows.
• Both models are evaluated on exactly the same held-out rows, which is what makes the comparison paired.
• The run-level metric is the mean absolute error over those rows, so each run contributes one number per method.
• We average matched cases within a run first, then treat the run-level differences as the replicates.

The two methods are deliberately from different families: an affine model using only the current state and action, and a k-NN model using a three-step history window. Their disagreement is what we will measure.

Seeds help repeatability when the relevant random streams and other nondeterminism are controlled. A seed alone is not a characterization of variation across draws. Independent runs or a justified stochastic model are needed for that uncertainty. The 12 fits in this executable example are a demonstration choice, not a standard or an assurance of adequate power. A borderline interval with few replicates warrants an unresolved conclusion rather than a categorical presence-or-absence claim.

The right time to decide how many runs you need is before you run them. For a two-sided paired comparison, a standard approximate calculation is

$$n \;\approx\; \left(\frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)\,\sigma_D}{\delta}\right)^{2},$$

where $\alpha$ is the Type I error rate, $1 - \beta$ the desired power, $\delta$ the smallest difference worth detecting, and $\sigma_D$ the assumed standard deviation of the paired differences. The symbol $z_p$ is the $p$ quantile of a standard normal distribution. At $\alpha=0.05$ and power $0.80$, the two quantiles are approximately $1.960$ and $0.842$. This is a normal approximation for planning. It is not an exact small-sample guarantee, it is not post-hoc observed power, and it is not a universal seed recommendation.

We can drive it with the pilot $\sigma_D$ estimated from the 12 runs above, while keeping in mind that a pilot estimate from a small number of runs is itself uncertain.

Practical guidance that follows:

• Round up, always. Fractional runs do not exist.
• Explore plausible SD values. A larger assumed $\sigma_D$ gives a larger planned $n$ at fixed target difference, error rate, and power. This is conservative relative to the pilot value, not a certified lower bound under the unknown population distribution.
• Distinguish exact inference from approximate planning. For iid Gaussian differences with positive population variance and $n\geq2$, the Student-$t$ interval has exact nominal coverage. Small samples can make robustness to nonnormality poor and make the normal planning formula inaccurate. A noncentral-$t$ calculation under the Gaussian model, or simulation under a justified alternative model, can refine planning.
• Check heavy-tailed scenarios. Heavy tails can make finite-sample normal planning inaccurate in either direction. Infinite variance invalidates a formula requiring $\sigma_D$. Simulate plausible distributions and assess Type I error and power rather than assuming the approximation always underestimates the requirement.
• Separate episodes from runs. More independent evaluation episodes can reduce within-run measurement noise and improve procedure-level precision, but cannot remove a nonzero between-run component. When that component dominates, extra episodes buy little compared with extra independent runs.

Several statistical reflexes are wrong, and each has a specific correct counterpart.

Overlapping confidence intervals do not settle whether a paired difference is zero. Two marginal intervals can overlap substantially while the paired difference is confidently nonzero, because overlap ignores the covariance. Always compute the interval on the difference, not by eyeballing two separate intervals.

Absence of statistical significance is not equivalence. Failing to reject zero leaves a range of effects compatible with the estimate and uncertainty; it neither establishes equivalence nor measures power. Prespecify meaningful equivalence bounds. A symmetric choice is $[-\Delta_{\text{eq}},+\Delta_{\text{eq}}]$, although asymmetric bounds are possible. Two one-sided tests (TOST), each at level $\alpha$, reject the two boundary nulls when the corresponding $100(1-2\alpha)\%$ two-sided interval lies strictly inside those bounds. At $\alpha=0.05$, that is a 90% interval, not the 95% interval used above to assess a difference from zero. A 95% interval wholly inside the same bounds is sufficient but more conservative. State the test, interval level, and practical bounds together. Lakens's equivalence-test primer explains this distinction.

Statistical and practical significance are different questions. With enough runs, a difference of 0.001 state units can be statistically significant. That does not make it worth deploying. Report the effect size in interpretable units alongside the interval.

Multiplicity matters. Ten independent true-null tests, each with actual Type I error probability 0.05, have probability $1-0.95^{10}\approx0.4013$ of at least one rejection. Nominal level 0.05 alone does not imply that each test attains that probability; dependence also changes the calculation. Define the confirmatory family and error target. Holm controls familywise error, the probability of at least one false rejection. Benjamini-Hochberg controls false discovery rate under its stated dependence assumptions, an expected proportion among rejected hypotheses. These are different guarantees. Prespecifying one primary comparison limits that primary family but does not automatically control additional claims. Labeling other analyses exploratory makes their status clear; disclosure alone does not change their statistical error probabilities.

Prespecify the primary metric, comparison, split, and stopping rule before confirmatory evaluation. Outcome-informed choices can compromise an unadjusted test interpretation. Not every later action is such a choice: transparently correcting a verified implementation error under a declared protocol is different from selecting the most favorable observed result.