Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Planning, Control, and Policy Evaluation › Planning Horizon and Compute Tradeoffs

Michael BrenndoerferAugust 3, 20267 min read

World Models HandbookAll chapters

Part 11 of 12Evaluation and Understanding

Chapter 64Planning, Control, and Policy Evaluation › Planning Horizon and Compute Tradeoffs

Reading section 4 of 9 · Planning Horizon and Compute Tradeoffs · 7 minSection 4 of 9 · 7 min

Section 4 of 9 · 7 min read

Planning, Control, and Policy Evaluation › Planning Horizon and Compute TradeoffsOne section · Sections in this chapter4 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A receding-horizon planner solves a local finite-horizon optimization at each decision step. Given the current state $s_t$ , it evaluates candidate action sequences of length $H$ , picks the one that maximizes predicted cumulative reward, executes only the first action, advances one step in the environment, and repeats. The policy emitted by this loop is not a static map; its behavior depends on where inside the model the search lands at each step.

Receding-horizon control executes the first action of a plan and refreshes the current state before planning again. This can limit the effect of trajectory prediction errors, but the chosen first action still depends on the candidate reward scores. Here each score uses $H$ pre-transition reward terms, evaluated at the current state and $H-1$ recursively predicted states; the final computed successor has no terminal reward and is not scored. Replanning is not a guarantee that those forecasts or actions are sound.

Random shooting draws $K$ candidate sequences, each containing $H$ actions, scores their model rollouts, and executes the first action of the highest-scoring candidate. This implementation performs $K\cdot H$ candidate transition evaluations per decision, excluding reward evaluation and ancillary overhead. It is a useful matched-search-count proxy when comparing these configurations, not a complete hardware-independent compute or latency metric.

Random shooting is a simple baseline. Cross-entropy methods iteratively refit a sampling distribution around elite candidates; gradient-based shooting uses differentiable dynamics. Here the random-shooting policy is determined by its state input, model, settings, and evolving random stream. Fresh policy factories reconstruct that stream per evaluation case so comparisons can share draws.

Proxies vs Latency

For a claim such as "our planner is faster," report measured latency and its hardware and implementation conditions. For a claim about this sweep under a matched candidate-transition budget, report $K\cdot H$ . The proxy and latency answer different questions.

Latency depends on implementation, hardware, batch size, memory layout, and model cost. Timing claims can generalize only with supporting measurements or a justified performance model. Matching $K\cdot H$ answers a specific depth-versus-breadth question; latency-matched comparisons and resource-performance frontiers answer other legitimate questions.

Under a fixed $K\cdot H$ budget, increasing $H$ exposes later rewards while reducing candidate count. Longer sequences explore a larger action space and use more recursively predicted states. Prediction errors can propagate, but their magnitude need not increase monotonically. Only the chosen first action is executed before replanning; all $H$ candidate reward stages contribute to the score.

These effects need not balance at the same horizon across tasks or search algorithms. Longer lookahead can help delayed rewards, while finite random shooting can struggle to cover longer action sequences. A horizon sweep measures the resulting behavior, not a unique decomposition of the causes.

Talvitie (2017) analyzes composed prediction errors and self-correcting models, rather than prescribing universally short MPC horizons. MBPO (Janner et al., 2019) uses short model rollouts branched from real data for policy learning; that is distinct from this MPC horizon sweep. Wang et al. (2019) explicitly studies a planning-horizon dilemma in model-based RL. These papers motivate measuring horizon sensitivity, not a theorem that shorter always wins.

The planner below owns its seeded random generator. Each evaluated episode creates a fresh policy with seed $110+i$ . Learned and true-dynamics planners with the same $(H,K)$ therefore use the same candidate arrays at corresponding decision indices. Different $(H,K)$ settings reshape draws into different candidate sequences, so the horizon sweep is not an identical-action-sequence intervention.

In[11]:

Code

def random_shooting_action(s, step_batch_fn, H, K, rng):
    a_candidates = rng.uniform(ACTION_MIN, ACTION_MAX, size=(K, H))
    S = np.tile(np.asarray(s, dtype=float), (K, 1))
    total = np.zeros(K)
    for h in range(H):
        a_h = a_candidates[:, h]
        total += reward_batch(S, a_h)
        S = step_batch_fn(S, a_h)
    best = int(np.argmax(total))
    return float(a_candidates[best, 0])


def make_mpc_policy(step_batch_fn, H, K, seed):
    rng = np.random.default_rng(seed)

    def policy(s):
        return random_shooting_action(s, step_batch_fn, H, K, rng)

    return policy


def make_mpc_factory(step_batch_fn, H, K):
    return lambda seed: make_mpc_policy(step_batch_fn, H, K, seed)


## Fresh factories reproduce draws; learned/true planners receive matched arrays.
check_s = np.array([0.2, -0.1])
check_factory = make_mpc_factory(learned_step_batch, H=3, K=8)
check_first, check_second = check_factory(909), check_factory(909)
assert np.array_equal(
    [check_first(check_s) for _ in range(4)],
    [check_second(check_s) for _ in range(4)],
)
check_rng_a, check_rng_b = (
    np.random.default_rng(110),
    np.random.default_rng(110),
)
assert np.array_equal(
    check_rng_a.uniform(ACTION_MIN, ACTION_MAX, (8, 3)),
    check_rng_b.uniform(ACTION_MIN, ACTION_MAX, (8, 3)),
)

We sweep $H$ while keeping $K\cdot H$ exactly 640 candidate transition evaluations per decision. This holds one search-count proxy fixed while changing depth and breadth; it does not hold wall-clock time or search-space coverage fixed.

In[12]:

Code

budget_sweep = [
    {"H": 2, "K": 320},
    {"H": 5, "K": 128},
    {"H": 10, "K": 64},
    {"H": 20, "K": 32},
    {"H": 40, "K": 16},
]

sweep_results = []
for cfg in budget_sweep:
    H, K = cfg["H"], cfg["K"]
    mpc_learned = make_mpc_factory(learned_step_batch, H, K)
    mpc_oracle = make_mpc_factory(true_step_batch, H, K)
    r_learned = evaluate_policy(true_step_batch, mpc_learned, S0_eval)
    r_oracle = evaluate_policy(true_step_batch, mpc_oracle, S0_eval)
    sweep_results.append(
        {
            "H": H,
            "K": K,
            "compute": K * H,
            "learned_mean": float(r_learned.mean()),
            "learned_std": float(r_learned.std()),
            "oracle_mean": float(r_oracle.mean()),
            "oracle_std": float(r_oracle.std()),
        }
    )

Out[13]:

Console

Matched-budget sweep on 20 held-out initial states; 30-step episodes
PD baseline (true-env) mean return: -3.588

  H     K   K*H   MPC-learned   MPC-oracle
  2   320   640        -4.635       -4.634
  5   128   640        -3.832       -3.834
 10    64   640        -4.612       -4.731
 20    32   640        -5.723       -5.985
 40    16   640        -7.693       -8.699

Both planners attain their highest observed mean at $H=5$ . Learned-model means for horizons $2,5,10,20,40$ are approximately $-4.635,-3.832,-4.612,-5.723,-7.693$ ; true-dynamics means are $-4.634,-3.834,-4.731,-5.985,-8.699$ . The fixed PD mean is about $-3.588$ , higher than every MPC mean in this sweep. These are descriptive estimates on the 20 fixed cases and prescribed streams, not replicated training-run averages.

Both curves improve from $H=2$ to $H=5$ and then decline. The true-dynamics controller also declines, so transition-model error is not necessary for that pattern. Paired candidate streams remove an avoidable draw mismatch within each configuration, but the sweep still changes both horizon and candidate count and allows feedback trajectories to diverge. It does not identify one remedy or establish which controller dominates beyond these cases.

Out[14]:

Visualization

Line chart of true-environment return versus planning horizon for learned and oracle MPC plus a PD baseline. — True-environment return of random-shooting MPC with $K \\cdot H=640$ candidate transition evaluations per decision. Both learned-model and true-dynamics planners improve from horizon 2 to 5, then decline. Transition-model error is not necessary for the decline, and search breadth changes with depth. Initial cases and candidate streams are paired within each configuration. The dashed line is the fixed PD baseline; this one-run comparison does not isolate a unique cause or remedy.

Do not generalize this sweep into a rule that shorter horizons are better. Delayed-reward tasks may require deeper lookahead or a terminal value estimate. Nor is the true-dynamics heuristic an upper bound: at several horizons the learned-model heuristic earns a higher observed return. Search, horizon, and model-quality sweeps under declared budgets can test possible remedies; this figure alone does not select one.

Equal candidate-transition counts need not produce equal latency. This NumPy implementation batches over $K$ and loops over $H$ , while a neural implementation has its own batching and sequence costs. Measure wall-clock latency on the relevant hardware if latency is the claim; do not infer that the two configurations must differ on every possible machine.