Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sim-to-Real Deployment and Continual Correction › Ablations: Damping, Gain, Delay, and Saturation

Michael BrenndoerferAugust 10, 202610 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 71Sim-to-Real Deployment and Continual Correction › Ablations: Damping, Gain, Delay, and Saturation

Reading section 6 of 13 · Ablations: Damping, Gain, Delay, and Saturation · 10 minSection 6 of 13 · 10 min

Section 6 of 13 · 10 min read

Sim-to-Real Deployment and Continual Correction › Ablations: Damping, Gain, Delay, and SaturationOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We compare a nominal-plant check, a parameter-shifted deployment baseline, and two changes to that deployment baseline. The last two retain the shifted parameters: their costs should be compared with the parameter-shift baseline to study the additional factor.

Nominal plant: plant dynamics match the nominal simulator. Predictors adapted to the different deployment plant need not match this plant.
Parameter shift: the deployment plant, no delay, full actuator authority. Pure parametric error.
Sensor delay: the same deployment plant with a three-step observation delay. This changes the controller's information, not the physical transition.
Tight saturation: the same deployment plant with $u_{\max}=0.5$ instead of $1.0$ . Both planner and plant receive this bound, so the experiment measures reduced control authority, not an unmodeled clipping limit.

A note on process noise. These loops use zero process noise, fixed initial conditions and seeded planner sampling. Across-model differences therefore concern the predictors and the actions selected using them under these conditions. We do not run a noisy tracking ablation, so neither the tracking-cost change nor preservation of the model ranking under noise is established. The clean experiment isolates a comparison; it is not a performance estimate for noisy hardware.

In[17]:

Code

MODELS = {
    "Nominal sim model": nominal_predict_batch,
    "DR (parameter-blind)": blind_predict_batch,
    "DR (parameter-conditioned)": make_cond_batch(theta_hat),
    "Nominal + residual": make_residual_batch(W_res),
}

REGIMES = {
    "Nominal plant": {"params": NOMINAL, "delay": 0, "u_max": 1.0},
    "Parameter shift": {"params": DEPLOY, "delay": 0, "u_max": 1.0},
    "Sensor delay (3 steps)": {"params": DEPLOY, "delay": 3, "u_max": 1.0},
    "Tight saturation": {"params": DEPLOY, "delay": 0, "u_max": 0.5},
}

T_CL = 200
ref_seq = reference_traj(T_CL)
ENV_P, ENV_V = 0.9, 3.0

closed_loop = {}
for rname, cfg in REGIMES.items():
    closed_loop[rname] = {}
    for mname, pred in MODELS.items():
        states, applied = close_loop(
            pred,
            cfg["params"],
            ref_seq,
            delay_steps=cfg["delay"],
            u_max=cfg["u_max"],
        )
        err = states[1:, 0] - ref_seq
        closed_loop[rname][mname] = {
            "mse": float(np.mean(err**2)),
            "violation": float(
                np.mean(
                    (np.abs(states[1:, 0]) > ENV_P)
                    | (np.abs(states[1:, 1]) > ENV_V)
                )
            ),
            "states": states,
            "applied": applied,
        }

Out[18]:

Console

Mean squared position tracking error (lower is better)
regime                             Nominal sim model        DR (parameter-blind)  DR (parameter-conditioned)          Nominal + residual
Nominal plant                                0.01459                     0.01607                     0.01756                     0.01756
Parameter shift                              0.02134                     0.02134                     0.02558                     0.02558
Sensor delay (3 steps)                       0.01564                     0.01568                     0.01672                     0.01672
Tight saturation                             0.03752                     0.03705                     0.03870                     0.03870

Fraction of steps outside the position/velocity envelope
regime                             Nominal sim model        DR (parameter-blind)  DR (parameter-conditioned)          Nominal + residual
Nominal plant                                  0.000                       0.000                       0.000                       0.000
Parameter shift                                0.000                       0.000                       0.000                       0.000
Sensor delay (3 steps)                         0.000                       0.000                       0.000                       0.000
Tight saturation                               0.000                       0.000                       0.000                       0.000

Read the tables as separate measurements. Tracking MSE evaluates reference following; the envelope fraction evaluates breaches of the declared synthetic position and velocity limits. Every envelope fraction is zero in these runs, so this table does not distinguish the models and does not demonstrate model exploitation or certify safety. A controller could improve tracking while violating a limit in another experiment, but that event is not observed here.

Model-based planning can prefer actions whose predicted consequences are wrong, including actions with understated risk. Whether an optimizer selects them depends on its objective, constraints and search. This experiment does not demonstrate that mechanism: no declared envelope is breached. Keeping tracking and envelope metrics separate would let a broader test detect a tradeoff rather than bury it in one aggregate score.

The measured rankings differ from the simple expectation that a more accurate predictor must track better. In the parameter-shift regime, the nominal and parameter-blind models have similar tracking MSE, while the conditioned and residual models have higher tracking MSE despite fitting the deployment dynamics accurately. Two columns coincide because those fitted predictors are essentially identical. This is evidence about the combined predictor and fixed sampled MPC, not evidence that residual fitting failed to generalize. Finite search, a short prediction horizon, and the planner's effort-penalized objective are possible contributors, but this experiment does not isolate their causal contributions. Do not infer a universal ranking from this single reference, initial condition, and planner seed.

Open-loop rollout error and why it ranks models differentlyLink Copied

Closed-loop tracking error is not the same as prediction error, and the two can rank the models differently. To see this, we roll every predictor recursively from the same initial state under the same fixed action sequence from the held-out test trajectory. This is pure open-loop prediction with no feedback to hide mistakes. Recursion is the key word: each prediction is fed back in as the next state, so a small bias at one step becomes an input to the next, and the compounding described by the error bound becomes directly visible.

In[19]:

Code

def recursive_rollout(predict_batch, s0, u_seq):
    """Recursively feed the model's own prediction back in as the next state."""
    S = np.asarray(s0, dtype=float)[None, :].copy()
    preds = [S[0].copy()]
    for t in range(len(u_seq)):
        S = predict_batch(S, np.array([u_seq[t]]))
        preds.append(S[0].copy())
    return np.array(preds)


# Untouched final test trajectory on the deployment plant.
test_u = exploration_actions(np.random.default_rng(999), 200)
test_states, _ = rollout(DEPLOY, test_u, s0=np.array([0.3, -0.2]))

# Explicit reporting scales: one metre of position error and one metre per
# second of velocity error each contribute one unit to the dimensionless norm.
# These are metric conventions, not safety tolerances or fitted parameters.
ERROR_P_SCALE = 1.0  # metres
ERROR_V_SCALE = 1.0  # metres per second
STATE_ERROR_SCALE = np.array([ERROR_P_SCALE, ERROR_V_SCALE])


def normalized_state_error(predicted, observed):
    """Dimensionless joint position/velocity error under declared fixed scales."""
    return np.linalg.norm((predicted - observed) / STATE_ERROR_SCALE, axis=1)


open_loop = {}
for mname, pred in MODELS.items():
    preds = recursive_rollout(pred, test_states[0], test_u)
    open_loop[mname] = normalized_state_error(preds, test_states)

Out[20]:

Console

Recursive open-loop normalized state error [dimensionless] on held-out deployment trajectory
model                               1 step    10 steps    50 steps   200 steps
Nominal sim model                   0.0032      0.0233      0.1542      0.3498
DR (parameter-blind)                0.0031      0.0277      0.1487      0.3698
DR (parameter-conditioned)          0.0000      0.0000      0.0000      0.0000
Nominal + residual                  0.0000      0.0000      0.0000      0.0000

The table reports a dimensionless joint state error, $e_t = \sqrt{((\hat{p}_t-p_t)/p_{\text{scale}})^2 + ((\hat{v}_t-v_t)/v_{\text{scale}})^2}$ , with $p_{\text{scale}}=1\,\text{m}$ and $v_{\text{scale}}=1\,\text{m/s}$ . The scales specify how position and velocity are weighted; they are not safety limits. Adding their unscaled squared errors would mix incompatible units. This convention preserves the numerical values of the earlier raw norm but makes its meaning explicit, and the monitoring and held-out gates below use the same convention. Other scales could change rankings and gate decisions; no scale-sensitivity study is included here.

The fitted conditioned and residual predictors produce errors that round to zero at each displayed horizon, consistent with the exactly identifiable noiseless plant. The nominal and parameter-blind predictors have small one-step error and much larger long-horizon error on this trajectory; neither curve is guaranteed to grow monotonically, and their long-horizon values are close. This is not a measured instance of the parameter-blind model necessarily diverging fastest. Prediction error and tracking MSE ask different questions and can rank the same predictors differently. Part XI: Evaluation and Understanding discusses why one-step fit alone is insufficient for a planning model.

The curve adds intermediate horizons to the table, showing how the error evolves along this fixed trajectory.

Out[21]:

Visualization

Log-scale normalized state error curves; the two fitted model curves overlap at the display floor. — Recursive dimensionless state error on one held-out deployment trajectory. Position and velocity errors are scaled by 1 m and 1 m/s, respectively. Fitted conditioned and residual predictors overlap near numerical precision; nominal and parameter-blind errors are larger at long horizons. A 1e-6 display floor permits the log axis and does not alter reported errors.

Out[22]:

Visualization

Grouped bar chart of closed-loop tracking error by regime and model. — Position tracking MSE under four synthetic regimes with a fixed sampled MPC and matched candidate-action draws. Conditioned and residual costs coincide to the displayed precision. Tight saturation raises all four costs; these runs do not establish a general ranking of model-fitting methods or a safety improvement.

The delay trapLink Copied

We sweep observation delay from zero to eight steps and rerun the same controller for the nominal, parameter-blind, and residual predictors. This changes the information supplied to the controller without refitting any predictor. The nominal curve decreases in this particular sweep; delay does not have a guaranteed monotone relationship with closed-loop tracking MSE. Neither a lower MSE at positive delay nor a worse MSE from an accurate model establishes why the controller behaves that way. A causal explanation would require further controller and task ablations.

In[23]:

Code

DELAYS = [0, 1, 2, 3, 5, 8]
delay_sweep = {}
for mname in [
    "Nominal sim model",
    "DR (parameter-blind)",
    "Nominal + residual",
]:
    pred = MODELS[mname]
    errs = []
    for d in DELAYS:
        states, _ = close_loop(pred, DEPLOY, ref_seq, delay_steps=d, u_max=1.0)
        errs.append(float(np.mean((states[1:, 0] - ref_seq) ** 2)))
    delay_sweep[mname] = np.array(errs)

# Reference only: nominal predictor and exact current-state observations.
zero_delay_states, _ = close_loop(
    nominal_predict_batch, DEPLOY, ref_seq, delay_steps=0, u_max=1.0
)
zero_delay_reference_mse = float(
    np.mean((zero_delay_states[1:, 0] - ref_seq) ** 2)
)

Out[24]:

Visualization

Tracking MSE curves versus delay with a nominal zero-delay reference line. — Tracking MSE across observation delays for three fixed predictors. The dashed line is the nominal predictor at zero delay, a reference rather than a bound. MSE decreases with delay in these runs; the sweep does not demonstrate that residual correction removes delay or that delay universally improves control.

The residual model has higher tracking MSE than the nominal model at zero delay, even though its fitted dynamics reproduce the deployment plant to numerical precision. Its delay curve decreases rather than rising in parallel with the other curves. There is no evidence here that its dynamics fit failed to generalize. The controller is also feeding an aged state into a one-step predictor trained on synchronized current states. An accurate transition from the aged state is not automatically an estimate of the current state; that timing mismatch is distinct from parameter error. The plotted MSE alone does not quantify how much each source contributes.

A possible remedy for known delay is to propagate the delayed observation through the known intervening action history, or to use a state estimator with time-stamped observations and actions. In a noiseless known model this can reconstruct current state; with uncertainty it needs an uncertainty-aware treatment. That estimator is not implemented or evaluated here. More synchronized adaptation data can improve a dynamics fit, but it does not by itself add the missing timing and history inputs to this controller. A plateau in improvement is not a diagnosis on its own: limited excitation, a misspecified model, optimization, and observation timing all need separate checks.