Sim-to-Real Deployment and Continual Correction › Continual Correction with a Nominal-Plus-Residual Model

Residual correction is the practice of keeping the nominal physics and learning only the correction. The corrected prediction adds a learned residual term to the nominal model:

$$\hat{s}_{t+1} = f_{\text{nom}}(s_t, u_t) + g_\theta(s_t, u_t)$$

The nominal model supplies prior structure while the learned term represents its discrepancy. Small residual magnitude does not imply an easy fitting problem: a small discontinuity can still require appropriate features. Conversely, a large constant offset can be simple to learn. Assess residual structure, coverage and uncertainty, not magnitude alone, when deciding whether the decomposition helps.

where:

• $f_{\text{nom}}(s_t, u_t)$: the hand-written nominal dynamics, exactly the simulator you would ship if you trusted it
• $g_\theta(s_t, u_t)$: the learned residual map, parameterized by $\theta$, intended to capture only what $f_{\text{nom}}$ gets wrong
• $\hat{s}_{t+1}$: the corrected next-state prediction used by the planner

We fit $g_\theta$ to the next-state difference $s_{t+1}-f_{\text{nom}}(s_t,u_t)$ from adaptation. Alignment is necessary but not sufficient: the residual class and data must represent the discrepancy too. Here both the fitted residual and the monitor derive from that vector difference, but the monitor reduces it to a scalar normalized magnitude. The correction is trained on vector targets; it is not trained on those magnitudes.

Related hybrid designs learn corrections around analytic structure. Ajay et al. (2018) augmented analytic simulation with stochastic recurrent residual models for planar pushing and bouncing. Zeng et al.'s TossingBot (2019) added learned corrections to throwing-release velocities proposed by a ballistic controller. The latter corrects control parameters, not a next-state map identical to ours. Their task-specific results motivate a pattern, not a guarantee for this residual estimator.

There are several design decisions that determine whether it works, and each one has a failure mode. Treating them as a checklist rather than an afterthought is what separates a correction that transfers from one that merely fits.

Shaping and regularization. An expressive residual can overfit noise or behave poorly away from its data, but neither outcome is inevitable. Choose features and penalties to reflect justified mechanisms, such as velocity dependence for a friction hypothesis or clipping for a known actuator limit. A weight penalty does not automatically bound function magnitude outside the data: even a linear function with a small nonzero slope grows without bound on an unbounded domain. State any operating-domain bound separately. Our polynomial basis and ridge solve constrain the fitted class without certifying extrapolation.

Alignment. Compute residual targets with the nominal model and state-action timing used at deployment. A different nominal model or timing convention defines a different correction target. Misalignment can be absorbed by a flexible estimator and hidden by an adaptation-set fit, but that outcome is not inevitable: residual patterns or independent timing checks may expose it. Evaluate aligned held-out transitions and controller behavior before accepting a correction; good adaptation error alone does not establish transfer.

Coverage and excitation. Data that never activate saturation cannot identify its limit without additional prior information. A structurally informed model may still generalize to unvisited instances; that inference must be justified and tested. For the three-parameter regression here, unique noiseless recovery requires rank three in the full matrix $[-p,-v,u_{\text{app}}]$. A constant nonzero input can suffice during an informative transient; zero input leaves the gain column zero. Designing informative excitation and ensuring it is permitted are separate requirements.

Identifiability limits. The delay sweep did not train a delay estimator and cannot establish a theorem about recoverability. A correct transition model does not supply the controller with the missing action history or timestamps; those inputs would be needed for the compensation procedure described above. Likewise, response outside the applied action range needs either additional data or justified prior structure. Identifiability depends on the observations, excitation, model class, and assumptions, not simply on the amount of data.

Now the discipline that makes updates testable: never update the model on the trajectory you are using to validate it, and never promote a candidate without a held-out comparison against a rule that was written down in advance. Writing the rule down in advance prevents changing it to fit the result. This gate measures prediction error on one trajectory; it does not establish closed-loop safety or tracking improvement.

The procedure has a standard shape.

1. Collect a bounded adaptation episode under supervision.
2. Fit one or more candidate corrections.
3. Evaluate each candidate on a held-out validation episode from the same plant.
4. Apply an acceptance rule that was fixed before evaluation, including a minimum improvement margin.
5. Promote only accepted candidates, and keep the previous model available for rollback.

A candidate can improve one-step error and still make things worse. Fitting noise in adaptation data can improve training one-step error while worsening held-out rollout error; it does not necessarily do so, especially if the same correction also removes a larger real bias. That is why this gate evaluates held-out rollout error rather than accepting a training-set improvement alone. The training-set number measures fit to the data used for estimation; the held-out number measures performance on that separate episode, not a guarantee for untested operating conditions.

Candidate A reuses the fit from the deployment adaptation episode. Candidate B is fitted on synthetic transitions from a different plant. Both are evaluated on the same held-out deployment trajectory with the fixed normalized state metric. Candidate B improves the rollout mean relative to the nominal model in this run, but its one-step error is worse, so the two-condition rule rejects it. This is a concrete rejection mechanism, not proof that every mismatched candidate must fail or that accepted candidates improve closed-loop tracking.