Sim-to-Real Deployment and Continual Correction › The Reality Gap, Stated Precisely

This section defines the reality gap precisely. We sort the sources of mismatch by how they behave under different remedies, and we distinguish two kinds of model error that organize the rest of the chapter.

Jakobi, Husbands, and Harvey's 1995 work used the term in its title, Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics. This is an attribution of that published use, not a claim of priority for inventing the term. For deployment, the practical question is task-specific: which distinctions must this simulator preserve for the controller and evaluation conditions? A simulator can support one task yet fail on another using the same physical system.

That reframing matters because it tells you where to look. A simulator can be extremely accurate about contact dynamics and completely silent about motor heating. It can model the arm perfectly and forget that the camera exports frames on a different clock than the controller. The gap is not a scalar error bar; it is a structured mismatch, and different parts of it have different fixes. A scalar error bar suggests a single knob to turn, whereas a structured mismatch tells you to go find the specific component of the system whose behavior is missing from the model and decide whether to model it, randomize over it, or monitor for it.

It helps to sort the sources by the part of the system they affect. Residual patterns can suggest tests, but do not uniquely identify these categories:

• Parameter mismatch. Wrong values in a chosen model family: masses, inertias, friction coefficients, damping, gains, gear ratios, battery-voltage curves or motor constants. Estimation can correct these when the experiment identifies them. They need not produce signed bias: a damping error gives a velocity-dependent residual whose mean can be zero over symmetric motion.
• Unmodeled structure. Missing or incorrectly represented mechanisms, such as deadband, backlash, stiction, hysteresis, cable stretch or contact stick-slip, relative to the chosen class. Correction can require a different functional form, state or input. Its residual need not depend on state: an omitted constant force is a structural discrepancy in a class that lacks a constant term.
• Timing and latency. Sensor delay, actuator delay, jitter, unsynchronized clocks, overruns or dropped frames. Sensor delay can leave the physical transition valid while making an observation-only prediction insufficient. Actuator delay changes which command reaches the plant and may require action history or additional state.
• Saturation and rate limits. Delivered commands differ from requested commands when limits bind. Limits may depend on speed or temperature for a particular actuator and operating range; that dependence is not universal. Gentle tests may never activate the limit. Once it does activate, the resulting state error can persist after the actuator leaves saturation.
• Observation channel differences. Noise statistics, quantization, offsets, occlusion or modality dropout. These affect what the controller knows even when the modeled plant dynamics are accurate.
• Distribution and task mismatch. Deployment reference trajectories, initial conditions or loads can differ from those used in model fitting and evaluation. Good fit on the simulated population does not establish performance under that new population. This can coexist with omitted physical mechanisms; the categories are a guide to designing checks, not mutually exclusive diagnoses.

One note on terminology before the formal distinction. Throughout this chapter the plant is the physical system being controlled, the model is the mathematical object used to predict its behavior, and the controller is the rule that maps observations into commands. These three words appear constantly, and keeping them separate is what makes the rest of the argument readable. A great deal of confusion in practice comes from sliding between them, for instance by saying "the model is wrong" when the model is right and the observation feeding the controller is stale, or by saying "the controller is unstable" when the controller is fine and the model it plans against is systematically biased.

Let us be precise, because this distinction organizes the whole chapter.

Suppose the true plant is a damped oscillator whose state consists of position $p$ and velocity $v$. The continuous-time dynamics are:

$$\begin{aligned} \dot{p} &= v \\ \dot{v} &= -k\,p - c\,v + b\,u_{\text{app}} \end{aligned}$$

where:

• $p$: position in metres, the first component of the state $s_t = (p_t, v_t)$
• $v$: velocity in metres per second, the second component of the state
• $\dot{p}$: the time derivative of position, which equals $v$ by definition
• $\dot{v}$: acceleration, or equivalently the time derivative of velocity
• $k$: mass-normalized stiffness, in $\mathrm{s^{-2}}$; for $k>0$, the term $-kp$ produces restoring acceleration
• $c$: mass-normalized viscous damping, in $\mathrm{s^{-1}}$; for $c\geq0$, its contribution dissipates mechanical energy
• $b$: dimensionless control gain, since the applied command already has acceleration units
• $u_{t}$: the scalar command issued by the controller at time $t$, in metres per second squared (the units of acceleration), which plays the role of the book's action $a_t$. We write the raw issued command as $u_t$ and the post-clip applied command as $u_t^{\text{app}} = \mathrm{clip}(u_t, -u_{\max}, u_{\max})$. The distinction between $u_t$ and $u_t^{\text{app}}$ matters because the model may predict a command the actuator cannot deliver. The physical acceleration is always written as $\dot{v}$ to avoid collision with the control symbol.

The oscillator has linear second-order dynamics in the applied input. Clipping from issued to applied command makes the overall commanded system nonlinear. A damped mechanical mode is a useful teaching example for inertia, restoring behavior and dissipation; it is not a claim that every controlled plant has such a mode. The parametric-versus-structural distinction does not depend on the simplicity of the plant; it depends on whether the discrepancy between the true plant and the model is confined to the constants or spills into the functional form.

A plant with $c^\star=0.62$ when the model uses $c=0.4$ has parametric mismatch if the remaining modeled structure is correct. Sensor delay is a different layer: $o_t=s_{t-3}$ leaves the physical transition $s_{t+1}=f(s_t,u_t)$ valid, but changes the controller's information. A predictor receiving only $(o_t,u_t)$ may lack enough history to determine the next state. Residual fitting can improve that conditional prediction, but exact current-state reconstruction needs additional assumptions and information, such as a known deterministic transition and the intervening applied actions. The delay sweep below tests a controller without that reconstruction, not the best possible time-aware estimator.

When a monitor fires, first test candidate explanations. If the discrepancy is identifiable parameter mismatch, an informative episode can support an update within the current class. If the class is misspecified, consider a different functional form, state or input. Neither category determines how difficult the correction is: an unexcited parameter can be unidentifiable, while an omitted constant term can be straightforward to estimate. The monitor alone does not make that distinction.

A rollout bound makes the distinction between local defects and propagated error explicit. Compare the true recursion $s_{t+1}=f^\star(s_t,u_t)$ with $\hat{s}_{t+1}=f(\hat{s}_t,u_t)$ under the same issued action sequence. Fix one norm and an operating region containing both trajectories. Suppose $\|f(x,u)-f^\star(x,u)\|\leq\epsilon$ throughout that region and $f^\star$ is $L$-Lipschitz in state with action held fixed. The triangle inequality gives

$$\begin{aligned} e_{t+1} &=\|\hat{s}_{t+1}-s_{t+1}\|\\ &\leq\|f(\hat{s}_t,u_t)-f^\star(\hat{s}_t,u_t)\| +\|f^\star(\hat{s}_t,u_t)-f^\star(s_t,u_t)\|\\ &\leq\epsilon+Le_t,\\ e_H&\leq L^He_0+\epsilon\sum_{j=0}^{H-1}L^j. \end{aligned}$$

where:

• $e_t=\|\hat{s}_t-s_t\|$ is endpoint error, not a sum of errors over time.
• $e_0$ is the initial-state discrepancy.
• $\epsilon\geq0$ is a uniform one-step defect bound in the same norm.
• $L\geq0$ bounds the true map's change with state while holding the action fixed.
• $H$ counts transitions; at $H=0$ the sum is empty.

For the example below, both rollouts start from the same state, so $e_0=0$. Define the resulting upper bound as $B_H=\epsilon\sum_{j=0}^{H-1}L^j$. It is a conditional guarantee for each trajectory satisfying the premises, not a prediction that a trajectory attains the bound. If either rollout leaves the operating region, this argument no longer supplies the stated guarantee.

The unamplified final term comes from the newest defect. A defect introduced one transition earlier can be multiplied once by $L$; an older one can be multiplied twice, and so on. These factors permit contraction when $L<1$ as well as expansion when $L>1$. The inequalities need not be jointly tight.

For $0\leq L<1$, $B_H$ approaches $\epsilon/(1-L)$. At $L=1$, it is $\epsilon H$. For fixed $L>1$ and $\epsilon>0$, $B_H=\epsilon(L^H-1)/(L-1)$ grows asymptotically as $\epsilon L^H/(L-1)$. Actual error may remain much smaller, including zero for identical maps. These open-loop bounds hold actions fixed; independently replanning controllers produce different action sequences and need a separate closed-loop analysis.

Use a dimensionless state norm for the oscillator: $\|( \delta p,\delta v)\|=\sqrt{(\delta p/(1\,\mathrm m))^2+(\delta v/(1\,\mathrm{m/s}))^2}$. The plot is a generic illustration with $\epsilon=0.01$ in this norm, not an estimated Lipschitz constant or defect bound for the fitted models below.

We plot the same bound for three values of $L$ at a fixed defect bound. Only the assumed amplification factor changes; the figure does not classify the fitted plant as contractive or expansive.