Sim-to-Real Deployment and Continual Correction

A better fitted dynamics model need not give better tracking with a fixed planner. The synthetic experiment in this chapter demonstrates that distinction while separating parameter mismatch, delayed observations and actuator limits. We develop a reproducible deployment workflow: collect informative adaptation data, monitor prediction errors, and evaluate proposed corrections before promotion. These executable results are not hardware evidence or a safety certificate.

Simulation-to-reality transfer can expose parametric mismatch, structural mismatch, observation differences, and task or interface errors. Parametric mismatch means the chosen model family can represent the relevant plant but its numerical values are wrong, such as damping after a payload change. Identifying those values requires observations that distinguish the parameters; sufficient excitation can provide that information. Good conditioning reduces sensitivity to measurement noise and numerical error, rather than being a logical requirement for exact noiseless recovery. Structural mismatch means the relevant true map is outside the fixed hypothesis class. Retuning parameters cannot make a member of that class equal an excluded map over the whole domain. This does not imply a positive approximation-error floor or a permanent observed residual: a nonclosed class can approach the map arbitrarily closely, and finite observations can miss the discrepant region.

The remedies address different parts of this problem. Domain randomization supplies training variation within a specified family; it does not automatically add an omitted mechanism. Residual models fit discrepancies relative to a nominal predictor, with generalization depending on their structure and adaptation data. Monitoring checks for departures from a reference error pattern; detection depends on distinguishable observations and a suitable decision rule, and does not by itself identify the cause. Continual correction adds a controlled update process, including rejection, versioning and rollback.

The rest of Part XII has already given you the tools to talk about what goes wrong. In Failure Modes and Model Exploitation we saw how optimization against an imperfect model can exploit and amplify accessible errors. In Robustness, Calibration, and Safe Control we developed uncertainty calibration and runtime assurance. We will not re-derive those here. Instead we will take a small, fully executable system, deliberately break it in controlled ways, and watch which remedies help, which do not, and why. Controlled comparisons help attribute effects to specified changes. Our delay and actuator-authority conditions add one factor at a time to an already parameter-shifted deployment baseline; they are not single-discrepancy comparisons against the nominal plant. Hardware can combine multiple discrepancies, motivating matched ablations rather than attribution from an aggregate score.

This section defines the reality gap precisely. We sort the sources of mismatch by how they behave under different remedies, and we distinguish two kinds of model error that organize the rest of the chapter.

Jakobi, Husbands, and Harvey's 1995 work used the term in its title, Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics. This is an attribution of that published use, not a claim of priority for inventing the term. For deployment, the practical question is task-specific: which distinctions must this simulator preserve for the controller and evaluation conditions? A simulator can support one task yet fail on another using the same physical system.

That reframing matters because it tells you where to look. A simulator can be extremely accurate about contact dynamics and completely silent about motor heating. It can model the arm perfectly and forget that the camera exports frames on a different clock than the controller. The gap is not a scalar error bar; it is a structured mismatch, and different parts of it have different fixes. A scalar error bar suggests a single knob to turn, whereas a structured mismatch tells you to go find the specific component of the system whose behavior is missing from the model and decide whether to model it, randomize over it, or monitor for it.

It helps to sort the sources by the part of the system they affect. Residual patterns can suggest tests, but do not uniquely identify these categories:

• Parameter mismatch. Wrong values in a chosen model family: masses, inertias, friction coefficients, damping, gains, gear ratios, battery-voltage curves or motor constants. Estimation can correct these when the experiment identifies them. They need not produce signed bias: a damping error gives a velocity-dependent residual whose mean can be zero over symmetric motion.
• Unmodeled structure. Missing or incorrectly represented mechanisms, such as deadband, backlash, stiction, hysteresis, cable stretch or contact stick-slip, relative to the chosen class. Correction can require a different functional form, state or input. Its residual need not depend on state: an omitted constant force is a structural discrepancy in a class that lacks a constant term.
• Timing and latency. Sensor delay, actuator delay, jitter, unsynchronized clocks, overruns or dropped frames. Sensor delay can leave the physical transition valid while making an observation-only prediction insufficient. Actuator delay changes which command reaches the plant and may require action history or additional state.
• Saturation and rate limits. Delivered commands differ from requested commands when limits bind. Limits may depend on speed or temperature for a particular actuator and operating range; that dependence is not universal. Gentle tests may never activate the limit. Once it does activate, the resulting state error can persist after the actuator leaves saturation.
• Observation channel differences. Noise statistics, quantization, offsets, occlusion or modality dropout. These affect what the controller knows even when the modeled plant dynamics are accurate.
• Distribution and task mismatch. Deployment reference trajectories, initial conditions or loads can differ from those used in model fitting and evaluation. Good fit on the simulated population does not establish performance under that new population. This can coexist with omitted physical mechanisms; the categories are a guide to designing checks, not mutually exclusive diagnoses.

One note on terminology before the formal distinction. Throughout this chapter the plant is the physical system being controlled, the model is the mathematical object used to predict its behavior, and the controller is the rule that maps observations into commands. These three words appear constantly, and keeping them separate is what makes the rest of the argument readable. A great deal of confusion in practice comes from sliding between them, for instance by saying "the model is wrong" when the model is right and the observation feeding the controller is stale, or by saying "the controller is unstable" when the controller is fine and the model it plans against is systematically biased.

Let us be precise, because this distinction organizes the whole chapter.

Suppose the true plant is a damped oscillator whose state consists of position $p$ and velocity $v$. The continuous-time dynamics are:

$$\begin{aligned} \dot{p} &= v \\ \dot{v} &= -k\,p - c\,v + b\,u_{\text{app}} \end{aligned}$$

where:

• $p$: position in metres, the first component of the state $s_t = (p_t, v_t)$
• $v$: velocity in metres per second, the second component of the state
• $\dot{p}$: the time derivative of position, which equals $v$ by definition
• $\dot{v}$: acceleration, or equivalently the time derivative of velocity
• $k$: mass-normalized stiffness, in $\mathrm{s^{-2}}$; for $k>0$, the term $-kp$ produces restoring acceleration
• $c$: mass-normalized viscous damping, in $\mathrm{s^{-1}}$; for $c\geq0$, its contribution dissipates mechanical energy
• $b$: dimensionless control gain, since the applied command already has acceleration units
• $u_{t}$: the scalar command issued by the controller at time $t$, in metres per second squared (the units of acceleration), which plays the role of the book's action $a_t$. We write the raw issued command as $u_t$ and the post-clip applied command as $u_t^{\text{app}} = \mathrm{clip}(u_t, -u_{\max}, u_{\max})$. The distinction between $u_t$ and $u_t^{\text{app}}$ matters because the model may predict a command the actuator cannot deliver. The physical acceleration is always written as $\dot{v}$ to avoid collision with the control symbol.

The oscillator has linear second-order dynamics in the applied input. Clipping from issued to applied command makes the overall commanded system nonlinear. A damped mechanical mode is a useful teaching example for inertia, restoring behavior and dissipation; it is not a claim that every controlled plant has such a mode. The parametric-versus-structural distinction does not depend on the simplicity of the plant; it depends on whether the discrepancy between the true plant and the model is confined to the constants or spills into the functional form.

A plant with $c^\star=0.62$ when the model uses $c=0.4$ has parametric mismatch if the remaining modeled structure is correct. Sensor delay is a different layer: $o_t=s_{t-3}$ leaves the physical transition $s_{t+1}=f(s_t,u_t)$ valid, but changes the controller's information. A predictor receiving only $(o_t,u_t)$ may lack enough history to determine the next state. Residual fitting can improve that conditional prediction, but exact current-state reconstruction needs additional assumptions and information, such as a known deterministic transition and the intervening applied actions. The delay sweep below tests a controller without that reconstruction, not the best possible time-aware estimator.

When a monitor fires, first test candidate explanations. If the discrepancy is identifiable parameter mismatch, an informative episode can support an update within the current class. If the class is misspecified, consider a different functional form, state or input. Neither category determines how difficult the correction is: an unexcited parameter can be unidentifiable, while an omitted constant term can be straightforward to estimate. The monitor alone does not make that distinction.

A rollout bound makes the distinction between local defects and propagated error explicit. Compare the true recursion $s_{t+1}=f^\star(s_t,u_t)$ with $\hat{s}_{t+1}=f(\hat{s}_t,u_t)$ under the same issued action sequence. Fix one norm and an operating region containing both trajectories. Suppose $\|f(x,u)-f^\star(x,u)\|\leq\epsilon$ throughout that region and $f^\star$ is $L$-Lipschitz in state with action held fixed. The triangle inequality gives

$$\begin{aligned} e_{t+1} &=\|\hat{s}_{t+1}-s_{t+1}\|\\ &\leq\|f(\hat{s}_t,u_t)-f^\star(\hat{s}_t,u_t)\| +\|f^\star(\hat{s}_t,u_t)-f^\star(s_t,u_t)\|\\ &\leq\epsilon+Le_t,\\ e_H&\leq L^He_0+\epsilon\sum_{j=0}^{H-1}L^j. \end{aligned}$$

where:

• $e_t=\|\hat{s}_t-s_t\|$ is endpoint error, not a sum of errors over time.
• $e_0$ is the initial-state discrepancy.
• $\epsilon\geq0$ is a uniform one-step defect bound in the same norm.
• $L\geq0$ bounds the true map's change with state while holding the action fixed.
• $H$ counts transitions; at $H=0$ the sum is empty.

For the example below, both rollouts start from the same state, so $e_0=0$. Define the resulting upper bound as $B_H=\epsilon\sum_{j=0}^{H-1}L^j$. It is a conditional guarantee for each trajectory satisfying the premises, not a prediction that a trajectory attains the bound. If either rollout leaves the operating region, this argument no longer supplies the stated guarantee.

The unamplified final term comes from the newest defect. A defect introduced one transition earlier can be multiplied once by $L$; an older one can be multiplied twice, and so on. These factors permit contraction when $L<1$ as well as expansion when $L>1$. The inequalities need not be jointly tight.

For $0\leq L<1$, $B_H$ approaches $\epsilon/(1-L)$. At $L=1$, it is $\epsilon H$. For fixed $L>1$ and $\epsilon>0$, $B_H=\epsilon(L^H-1)/(L-1)$ grows asymptotically as $\epsilon L^H/(L-1)$. Actual error may remain much smaller, including zero for identical maps. These open-loop bounds hold actions fixed; independently replanning controllers produce different action sequences and need a separate closed-loop analysis.

Use a dimensionless state norm for the oscillator: $\|( \delta p,\delta v)\|=\sqrt{(\delta p/(1\,\mathrm m))^2+(\delta v/(1\,\mathrm{m/s}))^2}$. The plot is a generic illustration with $\epsilon=0.01$ in this norm, not an estimated Lipschitz constant or defect bound for the fitted models below.

We plot the same bound for three values of $L$ at a fixed defect bound. Only the assumed amplification factor changes; the figure does not classify the fitted plant as contractive or expansive.

Domain randomization trains across a declared distribution of environments or plants rather than a single nominal instance. The aim is to cover relevant deployment variation. Successful simulated performance across that family does not by itself guarantee physical transfer: omitted mechanisms, observations, tasks and interfaces can still differ. Whether broad training or deployment-time identification is preferable depends on what can be measured and changed.

Tobin et al. (2017) trained object localization from randomized synthetic images and used it for real robotic grasping. Sadeghi and Levine (2017) trained an indoor quadrotor visual collision-avoidance policy using randomized simulated imagery without real training images. Peng et al. (2018) varied dynamics, including mass, friction and control gains, for a robot-arm object-pushing task. Kumar et al. (2021) combined randomized locomotion training with an adaptation module in RMA. That module estimates task-relevant latent environmental context from proprioceptive state-and-action history, rather than directly identifying a physical parameter vector. They demonstrated physical quadruped deployment, including unseen terrains. These are task-specific empirical demonstrations, not universal guarantees. Successful transfer also does not prove that every property of the physical deployment was inside the declared simulated support.

Randomization can be applied to at least three distinct objects. These operations have different properties and should not be treated as interchangeable.

A policy trained across variation. Randomization supplies policy-training environments. A fixed policy can learn useful behavior across the sampled family without an explicit parameter-identification module. Average performance under that distribution is not a worst-case guarantee across every plant, and randomization alone does not make the policy robust by construction. A recurrent policy may also infer context implicitly from history.

A parameter-conditioned predictor. Randomization supplies predictor-training data with context: $\hat{s}_{t+1}=f_\theta(s_t,u_t,\phi)$, where $\phi=(k,c,b)$. Deployment must supply that context, through measurement, known configuration or identification. Wrong context can degrade predictions, but its effect depends on the predictor and the error. Context quality is one requirement alongside model fit and coverage, not their replacement.

A parameter-blind mixture predictor. Pooled regression receives state and action but no explicit plant parameters. Under squared loss an unrestricted optimum estimates the conditional mean next state given those inputs; a finite feature class approximates it. That conditional mean need not equal a transition at the mean plant parameters, because states can contain information about the plant. A pooled fit is not itself a robustness guarantee for the controller that uses it.

Below, the blind and conditioned regressions use identical transition pairs. The conditioned model additionally receives parameter context and six feature terms, and deployment context comes from an adaptation episode. This compares those specified fitting routes; it does not isolate context from representation capacity or adaptation information.

Wider randomization can require more training data and capacity, but it does not necessarily make a policy slower or less precise. Shared structure can support several plants without a fixed capacity cost per plant. Compare broad training, measured context and adaptation under matched budgets; the right choice depends on uncertainty, coverage and the cost of gathering deployment data.

There is also a coverage question: what happens outside the declared randomization family? Training-family results alone do not establish performance there. Membership is not a guarantee either: finite sampling can leave sparse regions within the support. Extrapolation may succeed or fail depending on structure; the shifted plant below probes one outside-family case, not a universal discontinuity at a support boundary.

Randomization by itself does not provide a runtime detector. A fixed trained system may nevertheless tolerate drift covered by its training and learned behavior. Monitoring checks deployment evidence as conditions evolve, including changes outside anticipated variation. The two mechanisms therefore address different requirements.

We use a deterministic, CPU-sized deployment proxy: simulated dynamics stand in for a physical plant. The numerical results are synthetic, not hardware measurements. Relative to a nominal-plant check, the deployment baseline changes parameters; the delay and reduced-authority conditions each add a change to that deployment baseline. These matched comparisons distinguish specified changes without isolating every possible cause of the resulting controller behavior.

We reuse the damped oscillator from the dynamical-foundations material. The deployment plant evolves under semi-implicit Euler integration. The update advances the state by first updating velocity and then using the new velocity to update position:

$$\begin{aligned} v_{t+1} &= v_t + \Delta t\left(-k\,p_t - c\,v_t + b\,u_t^{\text{app}}\right) \\ p_{t+1} &= p_t + \Delta t\, v_{t+1} \end{aligned}$$

Velocity is updated first from the current state and applied command, then position uses the new velocity. This semi-implicit ordering differs from explicit Euler. For the undamped oscillator it admits bounded discrete oscillations when $0<\Delta t\sqrt{k}<2$, but it is not universally more stable for stiff damping. For this damped linear update, strict stability requires $k>0$, $c>0$ and $2c\Delta t+k\Delta t^2<4$. For example, $k=1$, $c=10$ and $\Delta t=0.2$ violate that condition even though explicit Euler is stable for the same case. An integrator mismatch is a discretization discrepancy; its magnitude need not be small. Here the applied command is $u_t^{\text{app}}=\mathrm{clip}(u_t,-u_{\max},u_{\max})$ with $u_{\max}=1.0\,\mathrm{m/s^2}$, where:

• $p_t, v_t$: position [m] and velocity [m/s] at timestep $t$, together forming the state $s_t = (p_t, v_t)$
• $u_t^{\text{app}}$: the applied (post-clip) command entering the plant
• $k, c, b$: the plant parameters for stiffness, damping, and control gain
• $\Delta t = 0.05\,$s: the discrete timestep

The control input $u_t$ is a scalar command in metres per second squared, and the state is $s_t = (p_t, v_t)$ in metres and metres per second. The units are declared so that the saturation limit and the reference amplitude are interpretable, but the system is abstract: you should read it as a stand-in for any second-order plant with a bounded actuator. The appeal of a second-order system as a proxy is that it is small enough that every effect can be traced by hand, nonlinear enough once saturation is introduced to exhibit structural behavior, and physically interpretable enough that the parameters carry meaning.

Let us set up the code.

The first block defines the plant step used for true trajectories. The batched nominal predictor later implements the same nominal update for candidate planning. Those two implementations must stay aligned in clipping, timestep and integration order; otherwise the experiment would introduce an unintended discrepancy.

Using the same trajectory to fit, select and report can bias a claimed generalization estimate. We keep four data roles separate so the reported test trajectory does not enter fitting or candidate selection:

• Training set: transitions from 24 randomized plants, used to fit the domain-randomized predictors.
• Adaptation set: a short episode with bounded commands from the deployment plant, used for system identification and residual fitting. Bounded commands alone do not establish a safe state trajectory.
• Validation set: a separate episode from the deployment plant, used only to gate residual candidates.
• Final test set: an untouched episode from the deployment plant, never used for any decision.

We also keep a plant outside the randomized support, to see what happens past the edge of the promise.

The generating family is $k\in[0.6,1.6]$, $c\in[0.2,0.8]$, $b\in[0.7,1.3]$. The printed ranges are extrema of only twenty-four draws, not those distribution endpoints. A plant with $c>0.8$ or $k>1.6$ is outside the declared family. Membership in the generating family and containment in the observed sample box are distinct checks; neither alone establishes predictive performance.

We compare four predictors. Each receives the same randomized training data where applicable, and each is used to drive the same controller, so differences in closed-loop cost come from the predictor and not from the planner. Holding the controller fixed is what makes the comparison a test of the predictor; if the controller changed with the model, we could not tell which change produced the difference.

• Nominal sim model: the hand-written simulator, with $k=1.0$, $c=0.4$, $b=1.0$, and clipping at $u_{\max}$. It is exactly the model you would ship if you trusted your simulator.
• Domain-randomized, parameter-blind: a single predictor fitted to the pooled transitions from all 24 plants, with no context input.
• Domain-randomized, parameter-conditioned: the same pooled data, but the predictor also receives the plant parameters. At deployment we feed it parameters identified from the adaptation episode.
• Nominal plus residual: the nominal simulator plus an explicitly shaped residual estimator fitted to the adaptation episode only.

The predictor class is deliberately modest: a quadratic feature map over $(s_t, u_t)$ appended with the ones vector, followed by a linear solve. Concretely, the feature map $\phi(s_t, u_t) \in \mathbb{R}^{10}$ is:

$$\phi(s_t, u_t) = \bigl[\,p_t,\; v_t,\; u_t,\; p_t v_t,\; p_t u_t,\; v_t u_t,\; p_t^2,\; v_t^2,\; u_t^2,\; 1\,\bigr]^\top$$

The feature map transforms the raw state and action into a fixed set of basis functions, allowing the linear predictor to capture quadratic nonlinearities. The feature map is a fixed, human-chosen transformation; the learning happens in the linear weights applied to it, so the expressive power of the predictor is exactly the span of these ten basis functions. Each element of the feature vector is defined below.

where:

• $p_t$: the position component of the state
• $v_t$: the velocity component of the state
• $u_t$: the issued scalar command
• the bilinear terms $p_t v_t, p_t u_t, v_t u_t$: couplings between state components ($p_t v_t$), and between state and action ($p_t u_t, v_t u_t$)
• the quadratic terms $p_t^2, v_t^2, u_t^2$: curvature in each variable
• $1$: the constant bias term

The base predictor is $\hat{s}_{t+1}=W^\top\phi(s_t,u_t)$ with $W\in\mathbb{R}^{10\times2}$: ten learned weights for each state component. The conditioned extension appends $k,c,b,kp,cv,bu$, giving sixteen features and a $16\times2$ weight matrix. These interactions represent the per-plant semi-implicit transition exactly within the tested command range. The small explicit class removes neural optimization complexity, but does not isolate training-distribution quality from context, feature capacity or adaptation information. A neural predictor could implement the same batched interface and evaluation workflow; its fit and numerical comparisons would need fresh validation.

The deployment plant sits inside the randomized support but is not the nominal plant: $k = 1.25$, $c = 0.62$, $b = 0.82$. We collect 140 steps of smooth excitation from it. That episode stands for a bounded, supervised data-collection run, not for autonomous exploration. The distinction matters: the experiment assumes we were permitted to excite the plant gently for a fixed, short interval under supervision, and it assumes the excitation was chosen to cover the directions we care about rather than to maximize information at any cost.

From that episode we do two things. First, system identification: invert the Euler step to get an acceleration target $\hat{a}_t = (v_{t+1} - v_t)/\Delta t$, then solve the following linear system. The system identification step recovers the plant parameters $(k, c, b)$ by fitting a linear model to acceleration data. The relationship is:

$$\hat{a}_t = -k\,p_t - c\,v_t + b\,u_t^{\text{app}}$$

where:

• $\hat{a}_t$: the acceleration target recovered by finite-differencing the observed velocity
• $p_t, v_t$: the state components at time $t$
• $u_t^{\text{app}}$: the applied (post-clip) command at time $t$
• $(k, c, b)$: the unknown plant parameters we want to estimate

The system is solved via least squares over the adaptation episode to recover the estimates $(\hat{k}, \hat{c}, \hat{b})$. The inversion works because the semi-implicit Euler step is affine in the parameters: given $(p_t, v_t, v_{t+1}, u_t^{\text{app}})$, the parameter vector $(k, c, b)$ appears linearly on the right-hand side. A linear least-squares problem in three unknowns is about as cheap as identification gets, which is the point: when the parametric part of the gap is identifiable and you have the excitation to see it, closing it costs almost nothing. Second, residual fitting: compute the residual target $r_t = s_{t+1} - f_{\text{nom}}(s_t, u_t)$ using the nominal model, and fit a shaped estimator to it.

The residual target must use the same nominal transition, timestep and input/target alignment as deployment. A different integrator or a one-step pairing error can introduce a learnable discrepancy that is numerical rather than physical. A residual estimator may fit that artifact with small training error, yet introduce error when applied to correctly aligned deployment inputs. Check the convention before interpreting a fitted correction as evidence of changed plant dynamics.

For closed-loop evaluation we use a short-horizon random-shooting model predictive controller. It samples candidate action sequences within the actuator bounds, rolls the predictor forward, and applies the first action of the lowest-cost sequence. The same planner and candidate-action draws are used for each predictor within a regime. Different predicted states can still produce different selected actions and subsequent trajectories; this is a comparison of each predictor coupled to that fixed planner.

At each control tick, the current reference position is held constant across all eight predicted steps. This is not lookahead tracking of the future reference sequence. The stage cost adds squared position error to an effort penalty, $0.02\,u^2$; with position in metres and command in metres per second squared, the coefficient has units of seconds to the fourth power so both terms have units of square metres. The chosen weight, horizon and 64 sampled sequences are experimental settings, not optimized controller parameters.

Notice what delay_steps does: the plant evolves from the true states[t], while the controller receives states[max(0, t - delay_steps)]. At zero delay, the controller has exact current-state observations. This is an idealized information condition, not a mathematical lower bound on tracking cost. The finite-horizon sampled planner is not globally optimal, and a change in delay can change its chosen actions in either direction. The zero-delay nominal-model run below is a reference for this specific controller, not an achievable-performance guarantee or a hardware baseline.

We compare a nominal-plant check, a parameter-shifted deployment baseline, and two changes to that deployment baseline. The last two retain the shifted parameters: their costs should be compared with the parameter-shift baseline to study the additional factor.

• Nominal plant: plant dynamics match the nominal simulator. Predictors adapted to the different deployment plant need not match this plant.
• Parameter shift: the deployment plant, no delay, full actuator authority. Pure parametric error.
• Sensor delay: the same deployment plant with a three-step observation delay. This changes the controller's information, not the physical transition.
• Tight saturation: the same deployment plant with $u_{\max}=0.5$ instead of $1.0$. Both planner and plant receive this bound, so the experiment measures reduced control authority, not an unmodeled clipping limit.

A note on process noise. These loops use zero process noise, fixed initial conditions and seeded planner sampling. Across-model differences therefore concern the predictors and the actions selected using them under these conditions. We do not run a noisy tracking ablation, so neither the tracking-cost change nor preservation of the model ranking under noise is established. The clean experiment isolates a comparison; it is not a performance estimate for noisy hardware.

Read the tables as separate measurements. Tracking MSE evaluates reference following; the envelope fraction evaluates breaches of the declared synthetic position and velocity limits. Every envelope fraction is zero in these runs, so this table does not distinguish the models and does not demonstrate model exploitation or certify safety. A controller could improve tracking while violating a limit in another experiment, but that event is not observed here.

Model-based planning can prefer actions whose predicted consequences are wrong, including actions with understated risk. Whether an optimizer selects them depends on its objective, constraints and search. This experiment does not demonstrate that mechanism: no declared envelope is breached. Keeping tracking and envelope metrics separate would let a broader test detect a tradeoff rather than bury it in one aggregate score.

The measured rankings differ from the simple expectation that a more accurate predictor must track better. In the parameter-shift regime, the nominal and parameter-blind models have similar tracking MSE, while the conditioned and residual models have higher tracking MSE despite fitting the deployment dynamics accurately. Two columns coincide because those fitted predictors are essentially identical. This is evidence about the combined predictor and fixed sampled MPC, not evidence that residual fitting failed to generalize. Finite search, a short prediction horizon, and the planner's effort-penalized objective are possible contributors, but this experiment does not isolate their causal contributions. Do not infer a universal ranking from this single reference, initial condition, and planner seed.

Closed-loop tracking error is not the same as prediction error, and the two can rank the models differently. To see this, we roll every predictor recursively from the same initial state under the same fixed action sequence from the held-out test trajectory. This is pure open-loop prediction with no feedback to hide mistakes. Recursion is the key word: each prediction is fed back in as the next state, so a small bias at one step becomes an input to the next, and the compounding described by the error bound becomes directly visible.

We sweep observation delay from zero to eight steps and rerun the same controller for the nominal, parameter-blind, and residual predictors. This changes the information supplied to the controller without refitting any predictor. The nominal curve decreases in this particular sweep; delay does not have a guaranteed monotone relationship with closed-loop tracking MSE. Neither a lower MSE at positive delay nor a worse MSE from an accurate model establishes why the controller behaves that way. A causal explanation would require further controller and task ablations.

The residual model has higher tracking MSE than the nominal model at zero delay, even though its fitted dynamics reproduce the deployment plant to numerical precision. Its delay curve decreases rather than rising in parallel with the other curves. There is no evidence here that its dynamics fit failed to generalize. The controller is also feeding an aged state into a one-step predictor trained on synchronized current states. An accurate transition from the aged state is not automatically an estimate of the current state; that timing mismatch is distinct from parameter error. The plotted MSE alone does not quantify how much each source contributes.

A possible remedy for known delay is to propagate the delayed observation through the known intervening action history, or to use a state estimator with time-stamped observations and actions. In a noiseless known model this can reconstruct current state; with uncertainty it needs an uncertainty-aware treatment. That estimator is not implemented or evaluated here. More synchronized adaptation data can improve a dynamics fit, but it does not by itself add the missing timing and history inputs to this controller. A plateau in improvement is not a diagnosis on its own: limited excitation, a misspecified model, optimization, and observation timing all need separate checks.

Monitoring compares predictions with observations to identify departures worth investigating. In filtering, an innovation is a new measurement minus its predicted measurement, conditioned on prior information. Our fully observed proxy uses a next-state residual instead. These are related prediction-error signals, but the innovation's noise law depends on the filtering model; it is not interchangeable with every learned dynamics residual.

The residual is defined as the difference between the observed next state and the model's prediction:

$$r_t = s_{t+1} - \hat{f}_\theta(s_t, u_t)$$

where:

• $s_{t+1}$: the observed next state delivered by the sensors
• $\hat{f}_\theta(s_t, u_t)$: the model's prediction of the next state from the current observation $s_t$ and the issued action $u_t$
• $r_t$: the residual, a vector of the same shape as the state
• the monitor's scalar magnitude: the norm after dividing the position and velocity residuals by the fixed 1 m and 1 m/s reporting scales, respectively

Assess residuals against the model's specified observation and disturbance law. Zero mean and whiteness are properties to justify under that law, not universal requirements. A mean shift, changing variance, spikes or temporal correlation can motivate tests of dynamics, sensors, timing or operating conditions. Such patterns can have multiple causes: zero-mean parameter errors and constant structural errors are possible, as the earlier taxonomy showed.

Sequential change detection accumulates evidence over time. A likelihood-based CUSUM uses log-likelihood increments for specified null and alternative models, resetting its running statistic when accumulated evidence becomes unfavorable. Centered tabular versions can be derived for particular distributions, such as a Gaussian mean change; an arbitrary centered residual sum need not be a likelihood ratio. A calibrated threshold can support a stated false-alarm property under its assumptions. Accumulation can improve detection of small persistent shifts, but does not make every shift distinguishable or remove the false-alarm/detection-delay tradeoff. The NIST CUSUM handbook describes the centered cumulative and tabular forms. The proxy below uses a simpler consecutive-exceedance rule, not CUSUM.

The practical decision is where to set the threshold. There are two honest options, and they are often conflated.

• A calibrated sequential guarantee. For example, set a threshold to meet a target expected stopping time under a specified null law, including temporal dependence. The claim is conditional on that law or on the validity of a stated robust null class; it must be checked for the deployment process.
• An empirical threshold from reference data. A sample quantile defines a cutoff. Estimate its same-condition exceedance rate on separate held-out data and account for sampling uncertainty and serial dependence. The quantile label alone supplies neither an exact population tail probability nor a sequential false-alarm guarantee.

We will use the second, and be explicit that it is not the first. The distinction matters because a quantile threshold carries no guarantee once the operating conditions drift away from the reference period: if the reference data was collected with a warm motor and the deployment runs cold, the "99th percentile" was the 99th percentile of a different distribution. Stating which kind of threshold you used is not bookkeeping; it determines what you are entitled to claim when the monitor stays quiet.

There is a hard limit on what a residual monitor can conclude, and it is a limit of identifiability, not of engineering. A residual is a scalar (or vector) summary of everything that differs between model and plant. A parametric drift, a sensor bias, an actuator degradation, and a state that has left the region where the model was fitted can all produce the same residual signature. From the residual alone you cannot tell them apart. This is not a defect of the residual; it is a property of any scalar summary that collapses many causes into one number.

Distinguishing causes requires discriminating information, which can come from existing excitation, additional sensors or a deliberately designed test. Active probing is one option, subject to an approved operating envelope, not always a necessity. Compare explicit hypotheses under the same observation and disturbance assumptions. If they induce identical full measurement laws under the chosen experiment, more of that same data cannot distinguish them. Equal predicted means alone are insufficient for this impossibility argument because their variances or other distributional properties may differ.

The shifted plant is outside the declared randomization family, and the monitor fires. That alarm alone does not establish whether parameters, observation processing or an omitted mechanism changed. The generator is known to us, but the detector does not receive that privileged diagnosis. The next section gates residual predictors on another deployment trajectory; it does not implement a fault-hypothesis comparison or diagnose the monitor's shifted plant.

Residual correction is the practice of keeping the nominal physics and learning only the correction. The corrected prediction adds a learned residual term to the nominal model:

$$\hat{s}_{t+1} = f_{\text{nom}}(s_t, u_t) + g_\theta(s_t, u_t)$$

The nominal model supplies prior structure while the learned term represents its discrepancy. Small residual magnitude does not imply an easy fitting problem: a small discontinuity can still require appropriate features. Conversely, a large constant offset can be simple to learn. Assess residual structure, coverage and uncertainty, not magnitude alone, when deciding whether the decomposition helps.

where:

• $f_{\text{nom}}(s_t, u_t)$: the hand-written nominal dynamics, exactly the simulator you would ship if you trusted it
• $g_\theta(s_t, u_t)$: the learned residual map, parameterized by $\theta$, intended to capture only what $f_{\text{nom}}$ gets wrong
• $\hat{s}_{t+1}$: the corrected next-state prediction used by the planner

We fit $g_\theta$ to the next-state difference $s_{t+1}-f_{\text{nom}}(s_t,u_t)$ from adaptation. Alignment is necessary but not sufficient: the residual class and data must represent the discrepancy too. Here both the fitted residual and the monitor derive from that vector difference, but the monitor reduces it to a scalar normalized magnitude. The correction is trained on vector targets; it is not trained on those magnitudes.

Related hybrid designs learn corrections around analytic structure. Ajay et al. (2018) augmented analytic simulation with stochastic recurrent residual models for planar pushing and bouncing. Zeng et al.'s TossingBot (2019) added learned corrections to throwing-release velocities proposed by a ballistic controller. The latter corrects control parameters, not a next-state map identical to ours. Their task-specific results motivate a pattern, not a guarantee for this residual estimator.

There are several design decisions that determine whether it works, and each one has a failure mode. Treating them as a checklist rather than an afterthought is what separates a correction that transfers from one that merely fits.

Shaping and regularization. An expressive residual can overfit noise or behave poorly away from its data, but neither outcome is inevitable. Choose features and penalties to reflect justified mechanisms, such as velocity dependence for a friction hypothesis or clipping for a known actuator limit. A weight penalty does not automatically bound function magnitude outside the data: even a linear function with a small nonzero slope grows without bound on an unbounded domain. State any operating-domain bound separately. Our polynomial basis and ridge solve constrain the fitted class without certifying extrapolation.

Alignment. Compute residual targets with the nominal model and state-action timing used at deployment. A different nominal model or timing convention defines a different correction target. Misalignment can be absorbed by a flexible estimator and hidden by an adaptation-set fit, but that outcome is not inevitable: residual patterns or independent timing checks may expose it. Evaluate aligned held-out transitions and controller behavior before accepting a correction; good adaptation error alone does not establish transfer.

Coverage and excitation. Data that never activate saturation cannot identify its limit without additional prior information. A structurally informed model may still generalize to unvisited instances; that inference must be justified and tested. For the three-parameter regression here, unique noiseless recovery requires rank three in the full matrix $[-p,-v,u_{\text{app}}]$. A constant nonzero input can suffice during an informative transient; zero input leaves the gain column zero. Designing informative excitation and ensuring it is permitted are separate requirements.

Identifiability limits. The delay sweep did not train a delay estimator and cannot establish a theorem about recoverability. A correct transition model does not supply the controller with the missing action history or timestamps; those inputs would be needed for the compensation procedure described above. Likewise, response outside the applied action range needs either additional data or justified prior structure. Identifiability depends on the observations, excitation, model class, and assumptions, not simply on the amount of data.

Now the discipline that makes updates testable: never update the model on the trajectory you are using to validate it, and never promote a candidate without a held-out comparison against a rule that was written down in advance. Writing the rule down in advance prevents changing it to fit the result. This gate measures prediction error on one trajectory; it does not establish closed-loop safety or tracking improvement.

The procedure has a standard shape.

1. Collect a bounded adaptation episode under supervision.
2. Fit one or more candidate corrections.
3. Evaluate each candidate on a held-out validation episode from the same plant.
4. Apply an acceptance rule that was fixed before evaluation, including a minimum improvement margin.
5. Promote only accepted candidates, and keep the previous model available for rollback.

A candidate can improve one-step error and still make things worse. Fitting noise in adaptation data can improve training one-step error while worsening held-out rollout error; it does not necessarily do so, especially if the same correction also removes a larger real bias. That is why this gate evaluates held-out rollout error rather than accepting a training-set improvement alone. The training-set number measures fit to the data used for estimation; the held-out number measures performance on that separate episode, not a guarantee for untested operating conditions.

Candidate A reuses the fit from the deployment adaptation episode. Candidate B is fitted on synthetic transitions from a different plant. Both are evaluated on the same held-out deployment trajectory with the fixed normalized state metric. Candidate B improves the rollout mean relative to the nominal model in this run, but its one-step error is worse, so the two-condition rule rejects it. This is a concrete rejection mechanism, not proof that every mismatched candidate must fail or that accepted candidates improve closed-loop tracking.

The gate rejects candidate B despite its lower mean rollout error because the one-step criterion also matters. That is the mechanism this experiment exercises. An accepted candidate still needs separate closed-loop, retained-condition, and safety validation. A prediction-error gate cannot substitute for those tests, particularly when better prediction did not improve the tracking MSE above.

Preserve an accepted prior artifact and a tested switchback path so rollback does not require retraining. For evaluation, the code keeps test_states out of fitting and gating. That dataflow separation does not prove historical preregistration or that a developer never inspected test results. Repeated adaptive use can bias an apparent held-out estimate, rather than making the data wholly uninformative. Use a fresh test after development decisions, and retain the criteria and evaluation record.

Before any of this touches a physical plant, it goes through a ladder of increasingly realistic test stages. The vocabulary is worth getting right, because the stages differ in what they can and cannot establish.

• Model-in-the-loop (MIL): the controller and the plant are both software models. You are testing the algorithm.
• Software-in-the-loop (SIL): controller code runs against a simulated plant, ordinarily on a development host. This tests executed code paths, software interfaces, and control logic. Host execution profiles can be measured, but ordinarily do not establish target real-time behavior.
• Processor-in-the-loop (PIL): target-compiled controller code runs on the target processor or an equivalent instruction-set simulator, with plant inputs supplied by a simulation harness. This tests the compiled component's numerical behavior and supported resource measurements. Target-code execution time can be measured; full-loop timing still depends on the scheduler, I/O, harness, and workload, and an instruction-set simulator needs its own timing justification.
• Hardware-in-the-loop (HIL): physical control-loop components exchange signals with simulated components in a closed loop. Controller HIL can connect target hardware to a real-time plant simulator to test timing, I/O and injected faults; engine- or power-HIL can include physical loads and power interfaces. Isermann et al. (1999) discuss engine-control HIL and sensor/actuator faults. Fathy et al. (2006) review automotive HIL and an engine-in-the-loop facility. Those configurations provide different evidence and hazards.
• Physical plant trials: the real system, in a controlled environment, under supervision, with a stated operating envelope and independent safeguards.

Each stage provides evidence within its tested conditions. MathWorks' SIL/PIL overview distinguishes host code from target-compiled execution, while its execution-time profiling documentation describes measurements available in both modes. These profiles must be tied to the measured configuration. HIL can measure deadline behavior and fault responses on target hardware, but finite trials alone do not prove that every possible workload meets a deadline. Simulator validation against physical evidence remains separate. Physical hazards can also exist in HIL setups, especially power-HIL systems; using a simulated plant is not a blanket safety guarantee. Test coverage and a formal guarantee, when available, must be stated separately.

We cannot run HIL here. What we can do is illustrate the class of logic HIL is designed to exercise: the relationship between control period, sensor period, observation age, and a staleness watchdog. This is a software fault-injection sketch on a CPU. It is not a HIL measurement, it does not characterize real-time performance, it says nothing about hardware fidelity, and it is not evidence of deployment readiness. Its only purpose is to make the relevant quantities concrete enough to reason about.

The control loop ticks every control_dt. The sensor publishes a new sample every sensor_dt, which is slower. The watchdog measures the age of the newest observation and forces the command to zero if that age exceeds a timeout. Then we inject a sensor dropout and watch the watchdog react. The design question is how the timeout should be chosen relative to the sensor period: too small and normal operation trips it, too large and a stall goes undetected for longer than the safety budget allows.