Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sim-to-Real Deployment and Continual Correction › A Controlled Deployment Proxy

Michael BrenndoerferAugust 10, 20267 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 71Sim-to-Real Deployment and Continual Correction › A Controlled Deployment Proxy

Reading section 4 of 13 · A Controlled Deployment Proxy · 7 minSection 4 of 13 · 7 min

Section 4 of 13 · 7 min read

Sim-to-Real Deployment and Continual Correction › A Controlled Deployment ProxyOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We use a deterministic, CPU-sized deployment proxy: simulated dynamics stand in for a physical plant. The numerical results are synthetic, not hardware measurements. Relative to a nominal-plant check, the deployment baseline changes parameters; the delay and reduced-authority conditions each add a change to that deployment baseline. These matched comparisons distinguish specified changes without isolating every possible cause of the resulting controller behavior.

The plant, the command, and the notationLink Copied

We reuse the damped oscillator from the dynamical-foundations material. The deployment plant evolves under semi-implicit Euler integration. The update advances the state by first updating velocity and then using the new velocity to update position:

\begin{aligned} v_{t+1} &= v_t + \Delta t\left(-k\,p_t - c\,v_t + b\,u_t^{\text{app}}\right) \\ p_{t+1} &= p_t + \Delta t\, v_{t+1} \end{aligned}

Velocity is updated first from the current state and applied command, then position uses the new velocity. This semi-implicit ordering differs from explicit Euler. For the undamped oscillator it admits bounded discrete oscillations when $0<\Delta t\sqrt{k}<2$ , but it is not universally more stable for stiff damping. For this damped linear update, strict stability requires $k>0$ , $c>0$ and $2c\Delta t+k\Delta t^2<4$ . For example, $k=1$ , $c=10$ and $\Delta t=0.2$ violate that condition even though explicit Euler is stable for the same case. An integrator mismatch is a discretization discrepancy; its magnitude need not be small. Here the applied command is $u_t^{\text{app}}=\mathrm{clip}(u_t,-u_{\max},u_{\max})$ with $u_{\max}=1.0\,\mathrm{m/s^2}$ , where:

$p_t, v_t$ : position [m] and velocity [m/s] at timestep $t$ , together forming the state $s_t = (p_t, v_t)$
$u_t^{\text{app}}$ : the applied (post-clip) command entering the plant
$k, c, b$ : the plant parameters for stiffness, damping, and control gain
$\Delta t = 0.05\,$ s: the discrete timestep

The control input $u_t$ is a scalar command in metres per second squared, and the state is $s_t = (p_t, v_t)$ in metres and metres per second. The units are declared so that the saturation limit and the reference amplitude are interpretable, but the system is abstract: you should read it as a stand-in for any second-order plant with a bounded actuator. The appeal of a second-order system as a proxy is that it is small enough that every effect can be traced by hand, nonlinear enough once saturation is introduced to exhibit structural behavior, and physically interpretable enough that the parameters carry meaning.

Let us set up the code.

The first block defines the plant step used for true trajectories. The batched nominal predictor later implements the same nominal update for candidate planning. Those two implementations must stay aligned in clipping, timestep and integration order; otherwise the experiment would introduce an unintended discrepancy.

In[5]:

Code

import numpy as np

# Abstract units, declared so the numbers are readable:
#   position p [m], velocity v [m/s], command u [m/s^2], timestep dt [s].
DT = 0.05
U_MAX = 1.0
NOMINAL = {"k": 1.0, "c": 0.4, "b": 1.0}


def step_plant(s, u_issued, params, u_max=U_MAX, rng=None, process_std=0.0):
    """Advance the plant by one timestep with semi-implicit Euler.

    s = (p, v). The issued command is saturated before it reaches the plant,
    so the actuator limit is part of the plant, not the controller.
    """
    p, v = s
    u_applied = float(np.clip(u_issued, -u_max, u_max))
    acc = -params["k"] * p - params["c"] * v + params["b"] * u_applied
    v_next = v + DT * acc
    p_next = p + DT * v_next
    if process_std > 0.0 and rng is not None:
        p_next += process_std * rng.normal()
        v_next += process_std * rng.normal()
    return np.array([p_next, v_next]), u_applied

Next, a rollout helper that records the true trajectory, and a delay helper that produces a causally delayed copy of any signal. The helper illustrates shape-preserving sensor buffering. The closed-loop experiment below uses the equivalent state-history indexing directly rather than calling this helper. Causal means the delayed signal at time $t$ depends only on samples up to time $t$ , so it is a faithful model of what a real sensor buffer delivers.

In[6]:

Code

def rollout(params, u_seq, s0=None, u_max=U_MAX, rng=None, process_std=0.0):
    """Roll the true plant forward under a fixed sequence of issued commands."""
    n = len(u_seq)
    if s0 is None:
        s0 = np.zeros(2)
    states = np.zeros((n + 1, 2))
    applied = np.zeros(n)
    states[0] = s0
    for t in range(n):
        states[t + 1], applied[t] = step_plant(
            states[t],
            u_seq[t],
            params,
            u_max=u_max,
            rng=rng,
            process_std=process_std,
        )
    return states, applied


def delay_signal(x, delay_steps):
    """Delay a time-major array; repeat the first sample during startup."""
    x = np.asarray(x)
    if x.ndim == 0:
        raise ValueError("x must have a time axis")
    if isinstance(delay_steps, (bool, np.bool_)) or not isinstance(
        delay_steps, (int, np.integer)
    ):
        raise TypeError("delay_steps must be an integer")
    delay = max(int(delay_steps), 0)  # Negative values mean no delay.
    if len(x) == 0:
        return x.copy()
    if delay >= len(x):
        # Avoid converting an arbitrarily large Python integer to an index dtype.
        return np.repeat(x[:1], len(x), axis=0).copy()
    indices = np.maximum(np.arange(len(x)) - delay, 0)
    return x[indices].copy()


# Shape and startup checks cover scalar signals, vector signals and long delays.
assert np.array_equal(delay_signal(np.arange(5), 2), [0, 0, 0, 1, 2])
assert delay_signal(np.arange(6).reshape(3, 2), 5).shape == (3, 2)
assert np.array_equal(
    delay_signal(np.arange(6).reshape(3, 2), 5),
    np.repeat([[0, 1]], 3, axis=0),
)
assert delay_signal(np.empty((0, 2)), 4).shape == (0, 2)
assert np.array_equal(delay_signal(np.arange(3), -1), np.arange(3))
vector_signal = np.arange(6).reshape(3, 2)
for long_delay in (2**63, 10**100):
    delayed = delay_signal(vector_signal, long_delay)
    assert np.array_equal(delayed, np.repeat([[0, 1]], 3, axis=0))
    assert not np.shares_memory(delayed, vector_signal)

Data splits that stay separateLink Copied

Using the same trajectory to fit, select and report can bias a claimed generalization estimate. We keep four data roles separate so the reported test trajectory does not enter fitting or candidate selection:

Training set: transitions from 24 randomized plants, used to fit the domain-randomized predictors.
Adaptation set: a short episode with bounded commands from the deployment plant, used for system identification and residual fitting. Bounded commands alone do not establish a safe state trajectory.
Validation set: a separate episode from the deployment plant, used only to gate residual candidates.
Final test set: an untouched episode from the deployment plant, never used for any decision.

We also keep a plant outside the randomized support, to see what happens past the edge of the promise.

In[7]:

Code

def reference_traj(n, freq=0.15, amp=0.55):
    """Smooth tracking reference for closed-loop evaluation."""
    t = np.arange(n) * DT
    return amp * np.sin(2.0 * np.pi * freq * t)


def exploration_actions(rng, n, u_max=U_MAX, smooth=0.85, scale=1.6):
    """Smooth, bounded, zero-mean excitation for identification data."""
    raw = rng.normal(size=n)
    u = np.zeros(n)
    prev = 0.0
    for t in range(n):
        prev = smooth * prev + (1.0 - smooth) * raw[t]
        u[t] = np.clip(prev * scale, -u_max, u_max)
    return u


def sample_params(rng):
    """Draw one plant from the randomization distribution."""
    return {
        "k": rng.uniform(0.6, 1.6),
        "c": rng.uniform(0.2, 0.8),
        "b": rng.uniform(0.7, 1.3),
    }

We generate randomized training data with smooth bounded commands and Gaussian initial states. Bounded commands do not impose a hard state envelope, and the Gaussian initialization is unbounded. In this fixed seeded experiment, the executed test and closed-loop state/action points fall inside the pooled training convex hull. That check does not establish dense coverage, cover every hypothetical planner rollout, or guarantee coverage for new references and initial states. The conditioned feature class can represent this noiseless transition exactly within the applied-command range; that structure also contributes to its good fit.

In[8]:

Code

rng = np.random.default_rng(20240517)
EP_LEN = 160

train_plants = [sample_params(rng) for _ in range(24)]
train_data = []
for params in train_plants:
    u = exploration_actions(rng, EP_LEN)
    s0 = rng.normal(scale=0.4, size=2)
    states, _ = rollout(params, u, s0=s0, rng=rng)
    train_data.append({"params": params, "s": states, "u": u})

Out[9]:

Console

Randomized training plants : 24
Pooled transition pairs    : 3840
k range                    : [0.69, 1.57]
c range                    : [0.21, 0.74]
b range                    : [0.73, 1.28]

The generating family is $k\in[0.6,1.6]$ , $c\in[0.2,0.8]$ , $b\in[0.7,1.3]$ . The printed ranges are extrema of only twenty-four draws, not those distribution endpoints. A plant with $c>0.8$ or $k>1.6$ is outside the declared family. Membership in the generating family and containment in the observed sample box are distinct checks; neither alone establishes predictive performance.