Autonomous Driving › Bird's-Eye-View and Occupancy State

A spatial driving model often needs to encode where nearby hazards are, explicitly or implicitly. Raw sensors provide partial spatial measurements in their own frames: cameras provide perspective images, LiDAR samples 3D points, and automotive radar can estimate range, angle, and Doppler-derived radial velocity. A camera pixel corresponds to a ray through space rather than one 3D point. A LiDAR return has measurement error. A radar detection can be misinterpreted without its geometry and velocity convention. For planning, these observations are commonly aligned in a shared spatial frame and interpreted as evidence about occupancy, motion, and semantics.

We call the example here an ego-centric bird's-eye-view (BEV) state: a top-down, metrically scaled view of the world in a frame attached to the ego vehicle. BEV can also be expressed in a global map frame. A BEV state can contain several channels: drivable space, lane boundaries, dynamic agents with their bounding boxes and velocities, static obstacles, and free-space boundaries. One possible BEV representation is a probabilistic occupancy grid, which discretizes space into cells and assigns an occupancy probability to each cell.

Transforming sensor observations into BEV requires assumptions about calibration and spatial projection. A single monocular image does not directly specify metric depth for every pixel. To place image evidence on the ground plane, the model needs depth estimation, a ground-plane homography where applicable, multi-view geometry, or a learned lifting operation. Each approach has different failure modes:

• Flat-ground inverse perspective mapping (IPM) assumes a planar road surface and projects image pixels onto it. It is cheap and interpretable, but that projection is valid only for points on the assumed plane.
• Depth-based lifting estimates per-pixel depth, then unprojects. Depth errors can substantially distort the resulting BEV, alongside calibration and perception errors.
• Learned lifting can use cross-attention or a transform network to place image features into BEV, trained end-to-end against downstream tasks. Feature assignment is learned, while camera calibration and parts of the geometry may remain explicit.
• LiDAR-first lifting bins 3D points into voxels or vertical pillars, then aggregates their features in BEV, sometimes with a learned encoder.

The choice of lifting mechanism affects what the model can localize reliably. An image-space detector may identify a pedestrian, but applying flat-ground IPM to body pixels distorts their projected ground location. Transparent and reflective surfaces can challenge depth estimation, while learned lifting can be difficult to diagnose outside its training distribution. A planner consuming BEV features needs uncertainty information as well as an estimated location; a simple transform does not make its output trustworthy by itself.

Once the scene is in BEV, the state can be discretized into an occupancy grid. A cell can carry an estimated probability that it contains an obstacle or object under the grid's chosen occupancy definition. Semantic labels and velocity estimates are optional. Classical probabilistic occupancy grids in mobile robotics are commonly updated by combining prior cell beliefs with sensor evidence. The Bayesian framing clarifies the event an occupancy probability should describe, given the observations and the chosen definition. A neural probability output should be checked for calibration before treating it as a reliable probability of that occupancy event.

In a driving world model, occupancy can serve a different role than in a static mapping system. It can describe an estimated current scene and also be a prediction target for what may occupy cells later. A planning-oriented model may forecast occupancy several seconds ahead, conditioned on proposed ego actions and estimates of other-agent behavior. Predicted occupancy is one useful input for an occupancy-based planner. In a mapping system, occupancy is an uncertain estimate of the present or previously observed scene. A future-occupancy forecast can be checked against later observations; whether it helped the planner is a separate action-outcome question.

This brings us to a distinction that we must preserve throughout the chapter. For analysis, it helps to distinguish three layers of state, even when a system estimates them jointly:

1. Observed occupancy. The cells that sensors directly support. This is one layer a perception stack may report; it may also infer occupancy beyond current observations.
2. Inferred state. The cells that the model believes are occupied based on prior structure, occlusion reasoning, and dynamics, but that sensors cannot currently see. The van at the corner hides cells behind it; a good model infers that something may be there.
3. Predicted future occupancy. The cells that the model expects to be occupied at future times, conditioned on assumed actions of ego and other agents. This is one useful input to an occupancy-based planner.

Confusing these layers can make the planner overconfident. Treating an inferred hidden object as directly observed hides the uncertainty in its existence or position. Treating one predicted future as certain can trigger unnecessary braking or conceal an alternative hazardous future. Training only to reproduce observed occupancy does not by itself establish useful future prediction. Evidence about the present, belief about the hidden present, and predictions about future states have different meanings, and the planner should preserve those distinctions.

Occupancy grids are not the only spatial representation in use. Depending on the task, a driving system may combine an occupancy grid with:

• Voxel grids, which add height and enable 3D collision reasoning.
• Vectorized maps, which represent lane and intersection geometry as polylines and graphs, with road rules encoded in associated attributes or semantic annotations. These carry road structure that a pure occupancy grid loses.
• Instance-level agents, which represent other vehicles as trackable entities with their own state. These are useful for agent-centric interaction-aware prediction, which we cover next; grid-based models can represent interactions differently.

The map and the grid are complementary. The grid estimates where space is occupied; a semantic map describes road structure such as lanes and crosswalks. Perception can also classify occupied cells when semantics matter for prediction or planning, such as distinguishing a parked car from a pedestrian. A grid alone may not distinguish a lane from a shoulder, while a map alone cannot establish whether that lane is blocked now. Systems can combine map and occupancy features early in an encoder or later during planning. Early fusion makes map context available while forming features; late fusion can keep the two streams separately inspectable.

A second distinction worth keeping explicit is between ego frame and world frame. The BEV state can be expressed relative to the ego vehicle, reducing dependence on global translation for nearby planning, or in a global frame anchored to a map, which supports route-level reasoning and consistency across locations. Systems may maintain local state for near-term planning alongside a global map for localization and route-following. Predictions then need a specified transform between frames whenever the planner compares them with global map features.

The resolution of the grid is a design choice with consequences. A one-meter cell can blur a curb, narrow obstacle, or lane marking; a ten-centimeter cell preserves finer geometry but greatly increases the number of cells over the same area. Resolution trades compute against representational detail, and the useful value depends on the task and sensing range. Multi-resolution schemes can allocate finer cells near the ego or around important objects. Grid-based token and convolutional models need a defined spatial discretization, even when they support more than one scale; not every spatial model uses a grid.

Finally, occupancy is not the same as semantics. A cell can be occupied without the model knowing whether the object is a car, bicycle, or cone. Class information can help predict motion, but it does not replace collision geometry; geometry alone does not reveal intent. Some driving models jointly predict occupancy, semantic class, and motion, while others use separate heads or modules. When a planner consumes separate estimates, their meanings and uncertainties should be characterized. When internal features are latent, downstream tests should still probe decisions under plausible state-estimation and prediction errors.

As we discussed in Geometry, Mapping, and 3D/4D State, the choice of spatial representation determines what downstream algorithms can use. An occupancy grid is one compromise: it encodes geometry in a planner-friendly structure, but discretization and occlusion remain difficult. For semantic occupancy, scarce examples of rare classes add a further challenge. When a driving system misjudges a scene or its future, it helps to ask whether the error came from sensing, hidden-state inference, or future prediction.