Autonomous Driving › Limitations and impact

The chapter's worked example is small enough to fit on a laptop, and that is precisely why it is useful. But the gap between this toy and a deployed driving world model is enormous, and it is worth stating clearly what that gap consists of.

The first limitation is scene complexity. Real scenes can contain many interacting agents, and factorized predictions or restricted interaction neighborhoods may omit important dependencies. A two-agent centerline example avoids occlusion, lane choices, traffic rules, realistic and varying vehicle dimensions, orientation, contact geometry, and perception error; it uses only fixed rectangular toy footprints. Success here says little about performance in a dense scene.

The second limitation is that real driving has continuous action and state spaces, and behavior need not split into two clean labels. A vehicle may accelerate slightly, drift within its lane, or yield while still moving. We supplied the two training labels in this toy problem; the model did not discover them unsupervised. The finite-mode equation is a simplified representation of trajectory uncertainty, and continuous-density or sample-based predictors are alternatives.

The third limitation is the simulation-to-reality gap. Simulated agent responses and sensor outputs may miss real behavior and environmental variation. Simulation outcomes are useful evidence but cannot by themselves establish real-road safety; deployment claims require broader validation under the applicable safety process.

The fourth limitation is the coupling between perception, prediction, and planning. The toy supplies the other agent's position and speed directly. On a vehicle, errors in object detection, depth, tracking, and maps can affect both predicted futures and the decision. Component-level ablations and end-to-end tests can complement each other when tracing how errors affect final decisions.

The value of these research directions is a richer way to represent and test driving futures. BEV encoders, multi-agent predictors, and reactive simulators let researchers study decisions under spatial uncertainty and interaction. Their practical benefits depend on the task, benchmark, and deployment setting; better open-loop prediction or more realistic-looking simulation does not automatically yield fewer real-world incidents.

The conceptual lesson extends beyond roads: when a learned model informs actions, evaluation must ask whether it preserves decision-relevant alternatives, provides uncertainty estimates that can be tested, and remains useful after actions change the state. The next chapter will apply that lens to digital environments.