The World-Model Evaluation Ladder › Summary
Michael BrenndoerferJuly 31, 20262 min read
World Models HandbookAll chapters Part 11 of 12Evaluation and Understanding
The World-Model Evaluation Ladder › SummaryOne section · Sections in this chapter8 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
- The five protocols are one-step prediction, open-loop rollout, intervention response, closed-loop planning, and transfer. They are complementary questions, not strictly ordered guarantees.
- One-step squared risk under a common finite-moment evaluation law splits into conditional-mean error and average conditional noise. A held-out estimate needs a declared sampling protocol and uncertainty calculation.
- The deterministic rollout bound requires uniform reachable-input accuracy and Lipschitz conditions, common actions, a common initial state, and no corrections. Its geometric expression is an envelope, not a guaranteed observed growth rate.
- Deterministic branch comparisons and stochastic counterfactuals need different specifications. Nonparametric adjustment-based observational identification requires causal assumptions and action coverage. Extrapolating unsupported action effects requires additional structural assumptions; a simulator score alone does not establish identification.
- Closed-loop model errors affect real states through chosen actions. Re-observation avoids direct prediction substitution but does not guarantee smooth action response or stability.
- Transfer must specify what changed. Dynamics, observations, occupancy, objectives, and constraints can affect usefulness in different ways.
- In the stated toy, winners across the five metrics are Aggressive, Delayed, Delayed, Delayed, and Conservative. Those rankings are conditional on the horizon, input sequence, initial state, and controller.
- A scalar deployment score can be meaningful under a declared objective. Use the diagnostic vector to expose assumptions and untested behavior that the scalar would otherwise leave implicit.