PILCO, PETS, Ensembles, and MBPO › Worked example: fixed-controller branch evaluation under model bias
World Models HandbookAll chapters PILCO, PETS, Ensembles, and MBPO › Worked example: fixed-controller branch evaluation under model biasOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We will use a one-dimensional system to isolate multi-step prediction error. The true dynamics and biased model receive the same initial states, action sequences, and process-noise draws. These common random numbers remove sampling differences from the model-versus-truth comparison; the remaining discrepancy comes from the action-gain bias.
import numpy as np
A_true, B_true = 0.9, 0.5
A_hat, B_hat = 0.9, 0.6
horizon = 12
n_trials = 200
rng_rmse = np.random.default_rng(0)
initial_states = rng_rmse.normal(size=n_trials)
actions = rng_rmse.normal(size=(n_trials, horizon))
process_noise = 0.05 * rng_rmse.normal(size=(n_trials, horizon))
s_true = initial_states.copy()
s_biased = initial_states.copy()
s_exact = initial_states.copy()
biased_sq_error = np.zeros(horizon)
exact_sq_error = np.zeros(horizon)
for t in range(horizon):
eps_t = process_noise[:, t]
a_t = actions[:, t]
s_true = A_true * s_true + B_true * a_t + eps_t
s_biased = A_hat * s_biased + B_hat * a_t + eps_t
s_exact = A_true * s_exact + B_true * a_t + eps_t
biased_sq_error[t] = np.mean((s_biased - s_true) ** 2)
exact_sq_error[t] = np.mean((s_exact - s_true) ** 2)
rmse = np.sqrt(biased_sq_error)
rmse_exact = np.sqrt(exact_sq_error)The exact-model control line is zero because it receives the same disturbances as the true system. This is a diagnostic use of common random numbers, not a claim that a predictive distribution has zero aleatoric variance.
step 1: biased-model RMSE = 0.1043 step 2: biased-model RMSE = 0.1308 step 3: biased-model RMSE = 0.1524 step 4: biased-model RMSE = 0.1720 step 5: biased-model RMSE = 0.1906 step 6: biased-model RMSE = 0.1969 step 7: biased-model RMSE = 0.2177 step 8: biased-model RMSE = 0.2192 step 9: biased-model RMSE = 0.2189 step 10: biased-model RMSE = 0.2201 step 11: biased-model RMSE = 0.2241 step 12: biased-model RMSE = 0.2226 Action-gain bias: B_hat - B_true = 0.10 Final-step RMSE (12 steps): 0.2226
The biased curve need not increase at every sampled step. In this stable system, the error obeys a stable recurrence driven by the random term , so its population variance approaches a finite limit. Different systems can amplify, cancel, oscillate, or saturate model error. Dimensionality alone does not determine error propagation: stability, coupling, error direction, and controller sensitivity matter. The task objective can change how consequential an error is and, when it changes action selection, which trajectories are encountered.
Coherent fixed-controller branch evaluation
The next toy is deliberately narrower than policy training. No policy is learned, no replay mixture is optimized, and this is not a literal MBPO implementation. We evaluate a grid of fixed proportional controllers, , under the true and biased simulators. Each branch starts from the same bank of states, runs coherently for consecutive steps, and receives the same process-noise draws in both simulators and for every candidate gain.
rng_branches = np.random.default_rng(43)
n_branches = 4096
max_branch = 12
branch_lengths = np.arange(1, max_branch + 1)
Ks = np.linspace(0.0, 2.0, 41)
branch_starts = rng_branches.normal(size=n_branches)
branch_noise = 0.05 * rng_branches.normal(size=(max_branch, n_branches))
def cumulative_branch_scores(A, B):
"""Score fixed feedback gains on coherent branches with shared randomness."""
scores = np.empty((max_branch, len(Ks)))
for j, K in enumerate(Ks):
s = branch_starts.copy()
cumulative_cost = np.zeros(n_branches)
for t in range(max_branch):
a = -K * s
s = A * s + B * a + branch_noise[t]
cumulative_cost += s**2 + 0.05 * a**2
scores[t, j] = -np.mean(cumulative_cost)
return scores
true_branch_scores = cumulative_branch_scores(A_true, B_true)
model_branch_scores = cumulative_branch_scores(A_hat, B_hat)
score_rmse = np.sqrt(
np.mean((model_branch_scores - true_branch_scores) ** 2, axis=1)
)
true_best_gain = Ks[np.argmax(true_branch_scores, axis=1)]
model_best_gain = Ks[np.argmax(model_branch_scores, axis=1)]k= 1: true-best K=1.50, model-best K=1.30, score RMSE=0.0516 k= 3: true-best K=1.55, model-best K=1.35, score RMSE=0.0915 k=12: true-best K=1.55, model-best K=1.35, score RMSE=0.1234
Every model branch advances from its own current model state for exactly steps; there is no temporally lagged "real" state and no periodic reset inside a branch. Common random numbers make comparisons insensitive to candidate-evaluation order and reduce Monte Carlo noise in model-versus-truth differences. In this seeded toy, the plotted score discrepancy grows over the displayed horizons and then begins to level as the stable closed-loop dynamics contract.
The printed gains are evaluation results for fixed controllers, not evidence about policy training. The example supports only the local conclusion that longer coherent branches expose cumulative controller scores to more consequences of this particular model bias. It does not establish that three steps are negligible, that a short branch always selects a controller closer to the true optimum, or that changing one bias parameter moves an optimal monotonically.
For integer branch lengths, a candidate should be compared with neighboring values and validated on the downstream task. Equality of marginal benefit and marginal cost is only an intuition for a differentiable continuous relaxation; boundary optima and non-monotone finite differences are possible.