Autonomous Driving › Closed-Loop Driving Simulation

An open-loop prediction benchmark takes a recorded history and scores a forecast against the recorded future. Real driving is closed-loop: ego actions change later states and observations. Open-loop prediction quality is useful evidence. It cannot by itself establish the safety or usefulness of a policy interacting with the environment.

The reason is compounding error and distribution shift. Open-loop evaluation conditions on recorded histories; a closed-loop policy generates its own future observations. Model errors can alter the ego's actions and later states, while other agents may react to those actions. This extends the compounding-error issue discussed in Sampling-Based Planning and Model Predictive Control to a coupled multi-agent system.

Closed-loop simulation lets the ego act and measures outcomes after the simulated world responds. It complements, rather than replaces, open-loop tests and on-road evidence. Three useful evaluation setups differ in what they replay or simulate:

• Log replay. The ego is re-driven through a logged scenario, with the other agents replayed from the log. Simple and reproducible, but the other agents ignore the ego. An alternative ego path can be checked against fixed replayed tracks, but replay cannot test how logged agents would react to that path or fully interactive counterfactual maneuvers.
• Reactive-agent simulation. Other agents are controlled by learned or rule-based policies that respond to the ego. CARLA and Waymo's Sim Agents work illustrate different tools and protocols in this broad family. Reaction improves some counterfactual tests, but fidelity and reproducibility depend on the agent models and seeds.
• Sensor-level simulation. The environment also renders observations for the ego's perception system. It tests a longer stack but introduces its own sensor and visual-domain gaps; more simulated components do not automatically make it more faithful to real roads.

The simulator's modeled components constrain which failure modes an evaluation can expose; scenario selection and measurements matter too. Log replay cannot evaluate how a logged agent would have responded to an alternative ego action. Reactive simulation can test modeled counterfactual responses, but they may differ from that agent's unobserved real response. Simulation can explore many rare or hazardous scenarios without exposing people to those tests, while real-world evaluation remains necessary for deployment claims.

For transfer testing, it helps to expose the policy to an interface close to the one it will use on the vehicle: compatible sensor data, maps, and actuator commands. Matching an interface does not guarantee matching real-world statistics or dynamics. Differences between simulated and real observations or responses are part of the simulation-to-reality gap and should be measured explicitly.

Simulation is also a diagnostic tool. Varying agent density, weather, or sensor noise can expose sensitivity, but a collapse under one perturbation does not identify a single faulty module without further ablation. False-positive occupancy may cause unnecessary braking; missed occupancy may create a collision risk. A useful evaluation records these distinct error types instead of reporting only one pass/fail outcome.

The simulator itself is a world model. This point is easy to miss: in a closed-loop driving simulation, the "environment" that the ego interacts with is another learned or hand-authored model. The quality of that environment model limits what the ego evaluation can establish. Overly predictable simulated agents may reward brittle planner behavior, while idealized physics may make simulated outcomes misleading about real-road dynamics. Neither mismatch guarantees a particular failure in a separately learned model, so simulator fidelity should be treated as an experimental variable rather than a fixed backdrop.

The worked example below is narrower than a reactive closed-loop simulator: it makes one decision from a prediction and rolls out the result. It still shows why an average trajectory error alone can obscure a collision-relevant mode, and why collision counts must be read alongside braking cost.

We'll return to closed-loop evaluation frameworks in The World-Model Evaluation Ladder and Datasets, Benchmarks, and Experimental Design, where we connect driving to broader evaluation methodology.