Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Genie and Interactive Generated Environments › Interactive Video Rollouts

Michael BrenndoerferJuly 22, 202610 min read

World Models HandbookAll chapters

Part 9 of 12Foundation and World-Action Models

Chapter 52Genie and Interactive Generated Environments › Interactive Video Rollouts

Reading section 4 of 9 · Interactive Video Rollouts · 10 minSection 4 of 9 · 10 min

Section 4 of 9 · 10 min read

Genie and Interactive Generated Environments › Interactive Video RolloutsOne section · Sections in this chapter4 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

An interactive environment adds a feedback loop to prediction. At each step the model receives an action and produces an observation; the user or agent sees that observation before choosing the next action. An ordinary generated clip can also feed predicted frames forward, but its action sequence is not chosen in response to those frames. This two-way exchange is what the following rollout tests.

Interactive Rollout

An interactive rollout is a sequence of model-generated frames produced by feeding the model's own previous output back as the next input, with user or policy actions interleaved. It is a closed-loop system, not a single conditional prediction.

The loop has a recurring structure:

Initialize from an observed frame or an image prompt. In our toy, this is a rendered grid cell.
Receive an input from a user or agent. The original Genie accepts one of its learned latent-action codes. Google DeepMind reports keyboard and mouse inputs for Genie 2 and real-time navigation for Genie 3, but their interface and training mechanisms should not be inferred from Genie's latent-action architecture.
Interpret the input in the model's control representation. In the original Genie, a human must learn or calibrate the meaning of a latent code; the later releases do not publicly document an equivalent code-to-key lookup.
Generate the next observation within a latency budget. Genie 3's announcement reports 24 frames per second at 720p, with consistency for a few minutes. Genie 2's announcement also describes a lower-quality distilled variant playable in real time; the later release's headline figures should not be projected backward onto it.
Append the generated observation to the model's context and repeat.

Each of these steps can fail in a different way. Initialization can fail if the prompt is out of distribution. Input mapping can fail if the alignment from human commands to latent codes is wrong. Generation can fail if the model produces an incoherent frame. Context append can fail if the model's context window is too short or if the generated frame is treated as ground truth when it is not. Repetition can fail through drift, where small errors accumulate over many steps. The loop is simple, but it has more failure surfaces than a one-step predictor.

Both an autoregressive video clip and an interactive rollout feed generated outputs back into the model. Interaction adds action choices that can move the process into new states. An error at step $t$ can then affect step $t+1$ , but need not always be amplified. As in Video Generators as Candidate World Models, let $\hat{o}_t$ be the predicted observation and $o_t$ the reference under the same action sequence. Their difference is $\varepsilon_t = \hat{o}_t-o_t$ . A local linearization illustrates one possible error propagation pattern:

\varepsilon_{t+1} \approx J_t \, \varepsilon_t + \nu_t

where:

$\varepsilon_t$ : the error at step $t$ between the model's predicted observation and the true observation
$J_t$ : the Jacobian of the model's prediction map with respect to the previous observation; it scales how strongly errors are transferred forward
$\nu_t$ : new noise injected at step $t$ from stochastic sampling and approximation error

The Jacobian $J_t$ captures local sensitivity to the previous observation. A bound below one contracts an existing perturbation without new error, whereas fresh error $\nu_t$ can sustain a nonzero error even in a contractive system. A norm above one permits amplification but does not ensure monotonic growth along every direction or noisy trajectory. Any multi-step autoregressive clip can exhibit this feedback problem; persistent interaction raises the stakes because the model must continue responding consistently to new actions. The recursion is a local illustration, not a measured stability result for Genie.

We can compare three scalar gains under the same seeded noise sequence, so differences between the plotted paths come from the gain rather than different random shocks.

In[14]:

Code

error_rng = np.random.default_rng(3)
n_steps = 30
noise_std = 0.05
jacobians = [0.9, 1.0, 1.05]
shared_noise = error_rng.normal(0, noise_std, size=n_steps)
error_paths = {}
for J in jacobians:
    eps = np.zeros(n_steps)
    eps[0] = 0.1
    for t in range(1, n_steps):
        eps[t] = J * eps[t - 1] + shared_noise[t]
    error_paths[J] = eps

Out[15]:

Visualization

Line chart of signed error under three scalar feedback multipliers. — Signed scalar error in three recursions over 30 steps, with multipliers 0.9, 1.0, and 1.05 driven by the same seeded noise. The common shocks isolate the effect of feedback gain on these sample paths; even the subunit multiplier need not make total noisy error decay monotonically.

There is also a statistical version of the problem. During one-step training the toy predictor sees real frames; during rollout it feeds its own generated frames back. Those frames may be outside the training distribution, so the one-step training loss has not directly penalized errors on them. This train–rollout mismatch is often called exposure bias. It applies to autoregressive video generation as well as interactive use; sustained interaction makes it especially consequential.

Interactive use makes persistence and action consequences easier to probe. If you walk past a rock and then turn around, the rock should still be there; a wall should block movement, and a collected item should remain collected. Video can contain and teach these patterns, including revisits, but a plausible short clip does not establish that a model will preserve them under long, user-chosen action sequences. They need direct tests at the relevant horizon.

These are rules about transitions under actions, often discussed as affordances or invariants. A wall blocks movement; a rock remains in place when the camera turns; an inventory item persists. A model can render a plausible wall but still predict movement through it. Interaction lets us test that rule directly, whereas a good-looking frame alone cannot.

Let us build the rollout loop for our toy and see how it behaves over a fixed horizon.

In[16]:

Code

rng_test = np.random.default_rng(1)
n_test_traj = 30

test_obs_seq = []
test_act_seq = []
for _ in range(n_test_traj):
    pos = (int(rng_test.integers(GRID)), int(rng_test.integers(GRID)))
    while pos in WALLS:
        pos = (int(rng_test.integers(GRID)), int(rng_test.integers(GRID)))
    obs_seq = [render(pos)]
    act_seq = []
    for _ in range(horizon):
        a = ACTION_NAMES[int(rng_test.integers(4))]
        pos = step(pos, a)
        act_seq.append(a)
        obs_seq.append(render(pos))
    test_obs_seq.append(np.array(obs_seq))
    test_act_seq.append(act_seq)

In[17]:

Code

def rollout_model(model, start_obs, action_seq, use_perm):
    obs = start_obs.copy()
    seq = [obs.copy()]
    for a in action_seq:
        a_idx = action_index[a]
        if use_perm:
            a_idx = action_perm[a_idx]
        x = np.concatenate(
            [obs.reshape(1, -1), one_hot(np.array([a_idx]))], axis=1
        )
        obs = model.predict(x).reshape(start_obs.shape)
        seq.append(obs.copy())
    return np.array(seq)


horizon_errors_latent = np.zeros(horizon + 1)
horizon_errors_true = np.zeros(horizon + 1)
horizon_errors_persist = np.zeros(horizon + 1)

for obs_seq, act_seq in zip(test_obs_seq, test_act_seq):
    pred_true = rollout_model(model_true, obs_seq[0], act_seq, False)
    pred_latent = rollout_model(model_latent, obs_seq[0], act_seq, True)
    pred_persist = np.tile(obs_seq[0][None, :, :], (horizon + 1, 1, 1))

    for t in range(horizon + 1):
        horizon_errors_true[t] += np.mean((pred_true[t] - obs_seq[t]) ** 2)
        horizon_errors_latent[t] += np.mean((pred_latent[t] - obs_seq[t]) ** 2)
        horizon_errors_persist[t] += np.mean(
            (pred_persist[t] - obs_seq[t]) ** 2
        )

horizon_errors_true /= n_test_traj
horizon_errors_latent /= n_test_traj
horizon_errors_persist /= n_test_traj

The rollout curves expose the gap between one-step and recursive prediction. In this seeded run, both learned predictors start at zero error, rise after a few steps, and finish around $0.016$ – $0.017$ . Persistence repeats the first frame, yet its error is not flat: it changes as the true agent moves, remaining around $0.023$ – $0.029$ . The curves average 30 fresh trajectories; they do not imply that every trajectory's error grows monotonically.

Out[18]:

Visualization

Line chart of frame MSE over rollout horizon for three models. — Recursive rollout frame MSE against the toy simulator as a function of horizon. The two learned predictors' errors rise early and then vary slowly; persistence repeats its initial frame but its error varies as the true agent moves. These curves do not show guaranteed monotonic growth.

Now let us isolate the concept of an action branch: two rollouts from the same initial observation, using different action sequences. A world model that is truly responsive should produce visibly different trajectories for different actions. A world model that is action-insensitive will produce nearly the same output regardless of what you ask for, and the branches will stay close together.

Action branches test whether output depends on input actions. A model that ignores the action produces identical branches from the same start, but a responsive model's separation need not increase monotonically: branches can converge, hit walls, or collapse under an imperfect predictor. Separation is not a test of correctness, either; both branches can be wrong. We therefore compare it with simulator-grounded metrics rather than treating any nonzero gap as proof of playability.

In[19]:

Code

branch_start = (4, 4)
branch_a = ["left"] * 4
branch_b = ["right"] * 4

initial = render(branch_start)
rollout_a = rollout_model(model_true, initial, branch_a, False)
rollout_b = rollout_model(model_true, initial, branch_b, False)

branch_sep = np.array(
    [
        np.mean((rollout_a[t] - rollout_b[t]) ** 2)
        for t in range(len(branch_a) + 1)
    ]
)

Out[20]:

Visualization

Line chart showing divergence of two predicted branches over four steps. — Branch separation for left-versus-right inputs from the same initial frame. The predicted frames differ after one step, but their MSE does not grow monotonically over this four-step rollout; responsiveness alone does not establish correct control.

Here the branches separate on the first step, then partly converge. The model responds differently to these inputs, but the tiny, nonmonotone gap does not establish accurate movement. Near-zero separation across many states and action pairs whose true outcomes differ would be evidence that the model ignores control. At a blocked corner, by contrast, two different actions should legitimately have the same outcome. This is the distinction the matched-branch test in Video Generators as Candidate World Models also needs.

If a model ignores its action channel across states where actions truly differ, a planner cannot use it to predict those differences. Branch comparison can reveal that failure, but nonzero separation is only evidence of sensitivity, not controllability: noise or systematically wrong effects can also separate branches. Pair it with reference transitions or task outcomes whenever those are available.