Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Video Generators as Candidate World Models › Worked Example: An Occluded Object on a 1D Corridor

Michael BrenndoerferJuly 21, 202616 min read

World Models HandbookAll chapters

Part 9 of 12Foundation and World-Action Models

Chapter 51Video Generators as Candidate World Models › Worked Example: An Occluded Object on a 1D Corridor

Reading section 10 of 13 · Worked Example: An Occluded Object on a 1D Corridor · 16 minSection 10 of 13 · 16 min

Section 10 of 13 · 16 min read

Video Generators as Candidate World Models › Worked Example: An Occluded Object on a 1D CorridorOne section · Sections in this chapter10 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We now build a small, CPU-sized scalar-position test bed for selected ladder diagnostics. The track is unbounded in the simulation; "corridor" names the one-dimensional coordinate and fixed occluder, not physical walls.

An object moves on a 1D corridor. It starts on the left and receives recorded actions from a right-biased behavioral policy; many, but not all, sampled trajectories pass through a fixed occluder band. A state $x_t \in \mathbb{R}$ is a scalar position; actions are in $\{-1, 0, +1\}$ . The dynamics are a damped velocity response:

v_{t+1} = d \, v_t + a_t, \qquad x_{t+1} = x_t + g \, v_{t+1} + \epsilon_t,

with $d = 0.5$ , $g = 0.4$ , and $\epsilon_t \sim \mathcal{N}(0, \sigma^2)$ with $\sigma = 0.02$ . Rewriting, the per-step displacement satisfies the ARX model

\Delta_t \;=\; x_{t+1} - x_t \;=\; d \, \Delta_{t-1} + g \, a_t + \epsilon_t - d\,\epsilon_{t-1},

which is a linear input-output relation we can fit from data, with one lagged displacement ( $\Delta_{t-1}$ ) and one control input ( $a_t$ ). The composite residual $\epsilon_t-d\epsilon_{t-1}$ is an MA(1) process, not independent noise; ordinary least squares below is a small-data illustration rather than an exact recovery guarantee. This is an input-output system-identification example related to Part II, Chapter 3.

The observation function is simple:

o_t = \begin{cases} x_t & \text{if } x_t \notin [4, 6] \\ \varnothing & \text{otherwise (occluded)}. \end{cases}

A predictor that only receives the visible position has no new position measurement during occlusion. It can still propagate a state estimate using past motion and actions. This scalar-position surrogate tests action-conditioned extrapolation; it is not a video generator or, by itself, a complete test of visual object permanence.

We compare three predictors on the same held-out trajectories:

Persistence. Predicts $\Delta_t = 0$ : the object stays where it was.
Action-blind ARX. Fits $\Delta_t = c_1 \Delta_{t-1}$ with no action input.
Action-conditioned ARX. Fits $\Delta_t = c_1 \Delta_{t-1} + c_2 a_t$ .

The three predictors are linear with at most two fitted parameters. This controlled comparison focuses on the action input rather than differences in network capacity. They are evaluated on the same held-out trajectories; the fitted ARX models use the same visible training transitions, while persistence has no training step.

Setting up the corridorLink Copied

We start with imports and constants.

In[3]:

Code

import numpy as np

OCCL_LO, OCCL_HI = 4.0, 6.0
DAMP, GAIN = 0.5, 0.4
PROCESS_NOISE = 0.02


def is_visible(x):
    return not (OCCL_LO <= x <= OCCL_HI)

The simulate function below produces a trajectory by sampling actions from a fixed behavioral policy and applying the ARX dynamics. Recorded actions are what the world model is allowed to see.

In[4]:

Code

def simulate(steps, seed):
    """Roll out the corridor dynamics and record timed actions and visibility."""
    r = np.random.default_rng(seed)
    x, v = float(r.uniform(1.0, 3.0)), 0.0
    xs, vs, acts, vis = [], [], [], []
    for _ in range(steps):
        a = int(r.choice([-1, 0, 1], p=[0.2, 0.25, 0.55]))
        xs.append(x)
        vs.append(v)
        acts.append(a)
        vis.append(is_visible(x))
        v = DAMP * v + a
        x = x + GAIN * v + r.normal(0.0, PROCESS_NOISE)
    return (
        np.array(xs),
        np.array(vs),
        np.array(acts, float),
        np.array(vis, bool),
    )

We now generate training and test sets with disjoint random seeds. This holds out full trajectories under the same simulator and policy; it does not test transfer to new scenes, mechanisms, or action policies.

In[5]:

Code

STEPS = 30
train = [simulate(STEPS, seed) for seed in range(24)]
test = [simulate(STEPS, seed) for seed in range(100, 110)]

occluded_fraction = float(np.mean([~t[3] for t in test]))

Out[6]:

Console

Held-out trajectories: 10
Fraction of held-out timesteps with the object occluded: 0.250

The aggregate fraction tells us whether the held-out set contains hidden timesteps; it does not establish that every trajectory crosses the band.

Fitting the three predictorsLink Copied

We fit the ARX coefficients by ordinary least squares, using only transitions whose three required positions are visible. This is an observation-availability restriction in the scalar surrogate, not a claim that a video system sees exact positions. Persistence has no fitted parameters.

In[7]:

Code

def arx_dataset(trajs):
    rows, targets = [], []
    for xs, _, acts, vis in trajs:
        for t in range(1, len(xs) - 1):
            if vis[t - 1] and vis[t] and vis[t + 1]:
                rows.append([xs[t] - xs[t - 1], acts[t]])
                targets.append(xs[t + 1] - xs[t])
    return np.array(rows), np.array(targets)


X_all, y_all = arx_dataset(train)
coefs_act = np.linalg.lstsq(X_all, y_all, rcond=None)[0]
coefs_blind = np.linalg.lstsq(X_all[:, :1], y_all, rcond=None)[0]
c1_act, c2_act = float(coefs_act[0]), float(coefs_act[1])
c1_blind = float(coefs_blind[0])

Out[8]:

Console

Fitted coefficients (training set only)
  action-blind ARX:          c1 = 0.610                 (true damping d = 0.5)
  action-conditioned ARX:    c1 = 0.499, c2 = 0.399   (true damping d = 0.5, true gain g = 0.4)

The action-conditioned fit recovers the true damping and gain closely on this training set. The action-blind fit has no action term and estimates a single lag coefficient of about 0.610, above the true damping of 0.5. That coefficient minimizes squared prediction error under a misspecified model and this data-collection policy; it is not an estimate of either physical parameter. The action-blind predictor cannot distinguish two different actions applied after the same observed history.

The coefficient estimates can be compared directly to the true dynamics. The cell below prepares the comparison, and the figure then shows the fitted and true coefficients side by side.

In[9]:

Code

coefficient_groups = ["action-blind ARX", "action-conditioned ARX"]
coefficient_bars = {
    "damping c1": np.array([c1_blind, c1_act]),
    "gain c2": np.array([0.0, c2_act]),
}
true_coefficients = {"damping d": DAMP, "gain g": GAIN}
coefficient_x = np.arange(len(coefficient_groups))
coefficient_width = 0.35

Out[10]:

Visualization

Grouped bar chart comparing fitted and true ARX damping and gain coefficients. — Fitted ARX coefficients compared with the true damping and action-gain values. The action-blind predictor omits the action input; its single fitted lag coefficient is about 0.610 and is not the true damping of 0.5. The action-conditioned fit is near the true damping of 0.5 and gain of 0.4 on this training set.

Recursive rollout and diagnosticsLink Copied

For an ARX predictor, the transition function is $\hat{\Delta} = c_1 \hat{\Delta}_{\text{prev}} + c_2 a$ . Multi-step rollouts feed the predictor's own estimate back in:

In[11]:

Code

MODELS = {
    "persistence": (0.0, 0.0),
    "action-blind ARX": (c1_blind, 0.0),
    "action-conditioned ARX": (c1_act, c2_act),
}


def rollout(xs, acts, coeffs, start=1):
    """Recursive prediction of x[start+1..end] from x[start], Delta[start-1], and actions."""
    x_hat = xs[start]
    d_hat = xs[start] - xs[start - 1]
    preds = []
    for k in range(start, len(xs) - 1):
        d_hat = coeffs[0] * d_hat + coeffs[1] * acts[k]
        x_hat = x_hat + d_hat
        preds.append(x_hat)
    return np.array(preds)

We compute one-step errors on held-out transitions with visible input and target positions, then recursive errors from a visible initial pair. The one-step metric is a local prediction diagnostic with reference state supplied at each eligible transition. The recursive metric propagates the predictor's own state without later position updates; its errors can accumulate.

In[12]:

Code

def one_step_errors(trajs):
    out = {}
    for name, coeffs in MODELS.items():
        errs = []
        for xs, _, acts, vis in trajs:
            for t in range(1, len(xs) - 1):
                if not (vis[t - 1] and vis[t] and vis[t + 1]):
                    continue
                d_prev = xs[t] - xs[t - 1]
                d_hat = coeffs[0] * d_prev + coeffs[1] * acts[t]
                errs.append(abs(d_hat - (xs[t + 1] - xs[t])))
        out[name] = float(np.mean(errs))
    return out


one_step = one_step_errors(test)

Recursive errors are averaged per horizon index and grouped by whether the true object was visible at that step. Because group membership also varies with trajectory and horizon, this descriptive split does not isolate an occlusion effect.

In[13]:

Code

HORIZON = 20
recursive = {name: np.zeros(HORIZON) for name in MODELS}
visible_err = {name: [] for name in MODELS}
occluded_err = {name: [] for name in MODELS}

for xs, _, acts, vis in test:
    for name, coeffs in MODELS.items():
        preds = rollout(xs, acts, coeffs, start=1)
        for k in range(min(HORIZON, len(preds))):
            t = k + 2
            e = abs(preds[k] - xs[t])
            recursive[name][k] += e**2
            (visible_err if vis[t] else occluded_err)[name].append(e)

for name in MODELS:
    recursive[name] = np.sqrt(recursive[name] / len(test))

visible_mean = {
    n: (float(np.mean(v)) if v else 0.0) for n, v in visible_err.items()
}
occluded_mean = {
    n: (float(np.mean(v)) if v else 0.0) for n, v in occluded_err.items()
}

Out[14]:

Console

One-step mean absolute error
  persistence                0.3695
  action-blind ARX           0.2994
  action-conditioned ARX     0.0205

Recursive RMSE at horizon 20
  persistence                5.6288
  action-blind ARX           5.1456
  action-conditioned ARX     0.0924

Mean error during occluded steps (recursive)
  persistence                2.3070
  action-blind ARX           1.7684
  action-conditioned ARX     0.0341

On these simulated held-out trajectories, the action-conditioned fit has lower one-step and recursive position error than the action-blind baselines. Since the models are comparably small, the action input is an important controlled difference here. The size and shape of this gap are properties of this simulator, policy, and fitted sample—not a forecast for real video generators.

Counterfactual action responseLink Copied

We now run the matched-initial-state counterfactual test from a specific state just left of the occluder. The object starts at $x_0 = 3.0$ with velocity $v_0 = 0.3$ , and we apply a constant action of $+1$ versus $-1$ for 12 steps. The reference true_branch sets the zero-mean process noise $\epsilon_t$ to zero, so its paths are conditional means for this linear simulator. The ARX model_branch initializes its prior displacement to $g v_0=0.4(0.3)=0.12$ , consistent with that noiseless reference. Under each model, we compare the final position gap.

In[15]:

Code

CF_STEPS = 12
X0, D0 = 3.0, 0.3


def true_branch(a_const, steps=CF_STEPS):
    x, d = X0, D0
    traj = [x]
    for _ in range(steps):
        d = DAMP * d + a_const
        x = x + GAIN * d
        traj.append(x)
    return np.array(traj)


def model_branch(coeffs, a_const, steps=CF_STEPS):
    x, d = X0, GAIN * D0
    traj = [x]
    for _ in range(steps):
        d = coeffs[0] * d + coeffs[1] * a_const
        x = x + d
        traj.append(x)
    return np.array(traj)


true_sep = float(true_branch(1.0)[-1] - true_branch(-1.0)[-1])
pred_sep = {
    name: float(model_branch(coeffs, 1.0)[-1] - model_branch(coeffs, -1.0)[-1])
    for name, coeffs in MODELS.items()
}

Out[16]:

Console

True final-position separation between right and left branches: 17.6004
  persistence                predicted separation: 0.0000
  action-blind ARX           predicted separation: 0.0000
  action-conditioned ARX     predicted separation: 17.5450

Persistence and action-blind ARX produce zero branch separation because neither uses the action input. The action-conditioned model reproduces the reference separation closely in this toy. The branches share an initial state and drift velocity and differ only in the constant applied action. Predicting no difference here fails to represent this intervention's known effect; it is not a general verdict about every action or environment.

The separation numbers can also be unpacked into the full branch trajectories. The cell below evaluates the true and model branches for the right-push and left-push counterfactuals, and the figure overlays them.

In[17]:

Code

cf_steps_axis = np.arange(CF_STEPS + 1)
cf_true_right = true_branch(1.0)
cf_true_left = true_branch(-1.0)
cf_act_right = model_branch(MODELS["action-conditioned ARX"], 1.0)
cf_act_left = model_branch(MODELS["action-conditioned ARX"], -1.0)
cf_blind_right = model_branch(MODELS["action-blind ARX"], 1.0)
cf_blind_left = model_branch(MODELS["action-blind ARX"], -1.0)

Out[18]:

Visualization

Line chart of true and predicted counterfactual branches for right and left actions. — Matched-initial-state counterfactual branch trajectories for constant right and left actions over twelve steps. The true branches diverge, the action-conditioned model follows both closely, and the action-blind model predicts one trajectory for both actions in this toy.

A rollout pictureLink Copied

Before plotting, we precompute the horizon curves and pick one held-out trajectory to display. All computation happens here, in plain Python; the figure cells below are presentation-only. This particular trajectory enters the occluder band only at its final two timesteps; the aggregate test below covers longer occluded intervals.

In[19]:

Code

example = test[3]
xs_ex, _, acts_ex, vis_ex = example
example_preds = {
    name: rollout(xs_ex, acts_ex, coeffs, start=1)
    for name, coeffs in MODELS.items()
}
example_t = np.arange(2, 2 + len(next(iter(example_preds.values()))))

horizons = np.arange(1, HORIZON + 1)

Out[20]:

Visualization

Line chart of true object position and three model predictions along a 1D corridor with a shaded occluder band. — Recursive predictions on one held-out corridor trajectory for persistence, action-blind ARX, and action-conditioned ARX predictors. The action-conditioned model tracks the true position, including the two final timesteps in the occluder band. Persistence holds its start-of-rollout position; the action-blind model drifts without responding to the recorded actions.

The action-conditioned curve stays close to the truth on this trajectory, including its two occluded timesteps. Persistence holds its start-of-rollout position and misses the subsequent motion. The action-blind predictor extrapolates from its prior displacement but does not respond to the recorded actions, so it too fails to keep up. The visibility-stratified comparison comes from the aggregate results below, not this one example.

Out[21]:

Visualization

Line chart of recursive RMSE vs rollout horizon for three linear predictors. — Recursive root-mean-square error as a function of rollout horizon, averaged over ten held-out trajectories, for persistence, action-blind ARX, and action-conditioned ARX predictors. The action-conditioned model stays far below the two action-blind baselines. The baseline errors rise overall across the displayed horizons, with local dips rather than strictly monotone growth.

The baseline curves rise overall as recursive predictions drift away from the true trajectory, although neither rises at every horizon. Persistence repeats an old position; action-blind ARX extrapolates motion without the recorded actions. The action-conditioned predictor stays much closer to the held-out trajectories. These curves compare predictors on this toy data; their shapes do not establish a universal error-growth law.

Occlusion memory and counterfactual responseLink Copied

We summarize the two most diagnostic scores in a single side-by-side figure.

Out[22]:

Visualization

Grouped bar chart of mean position error for three models on visible vs occluded timesteps. — Mean recursive position error for persistence, action-blind ARX, and action-conditioned ARX predictors, grouped by whether the true object was visible or occluded at each timestep. The action-conditioned predictor has much lower error in both groups. Since these are accumulated rollout errors rather than matched one-step errors, this split does not isolate the effect of occlusion.

Bar chart of counterfactual branch separation for three models with a dashed true-value line. — Final-position separation between the right-push and left-push counterfactuals from a matched initial state for each predictor. Persistence and action-blind ARX produce zero separation, while only the action-conditioned model matches the true value marked by the dashed line.

The left panel shows lower error for the action-conditioned predictor in both the visible and occluded groups. Its visible-step mean is about 0.06, compared with about 2.91 for action-blind ARX and 3.32 for persistence; the same ordering holds behind the occluder. Because errors accumulate over different trajectories and horizons, this grouping cannot by itself attribute the gap to occlusion. The right panel tests a different question: with the initial state held fixed, only the action-conditioned predictor separates the two commanded action branches by roughly the true amount. The toy permits this controlled comparison because its state and action mechanism are known.

This scalar surrogate does not benchmark a video system. It demonstrates horizon-resolved position error, a visibility-stratified descriptive check, and matched counterfactual action branches where reference state and dynamics are known. A proper occlusion-memory probe would additionally control horizon and motion regime, or score reappearance after a matched hidden interval. Real evaluations need task-specific baselines, held-out conditions, and reference outcomes appropriate to the claim.