Autonomous Driving › A worked example: a two-agent intersection

We now build a small, CPU-sized experiment that isolates the behavior-uncertainty part of the opening intersection. Unlike the fully occluded opening, the toy gives the ego the other agent's current position and speed but not its intent. The ego travels east along the $x$-axis toward a shared box centered at the origin. The other agent travels north along the $y$-axis. At $t=0$, a predictor proposes futures and a threshold rule decides whether the ego brakes. This is a one-decision action-outcome test, not a reactive multi-agent simulator or repeated closed-loop replanning.

We make the scenario minimal so that every step is auditable. The other agent has two possible policies. In continue mode it keeps a constant velocity. In yield mode it brakes to a stop before the intersection. Both policies share the same current-state input. This toy does not simulate an observation history. A classical constant-velocity baseline extrapolates that input without learning. The single-trajectory predictor approaches the mean of the two futures. The label-supervised predictor produces one trajectory per mode. We compare prediction metrics and, after the rollouts, report sampled toy-footprint overlaps and totals of the initial braking decisions.

For the overlap test, the ego is a 3 m long by 2 m wide axis-aligned rectangle traveling east. The other vehicle is a 3 m long by 2 m wide rectangle traveling north. With centers at $(x_{\text{ego}},0)$ and $(0,y_{\text{other}})$, these rectangles have positive-area overlap exactly when $|x_{\text{ego}}|<2.5$ m and $|y_{\text{other}}|<2.5$ m. Boundary-only touching is excluded. The shaded 5 m by 5 m box in the figures is this center-overlap region, not a claim about real intersection or vehicle dimensions. The rectangles are a toy geometry. The experiment does not model turning. It does not model shape uncertainty. It does not model contact dynamics. During rollout, overlap is checked at each 0.1-second sampled state, not continuously between steps.

We begin with imports and physical constants. The chapter uses fixed seeds and runs on CPU; runtime depends on the machine.

The training set is approximately balanced between the two modes. Each sample contains a 2D input. Its target is a 31-dimensional trajectory.

Before training anything, we see what the situation looks like from above. We plot the center-overlap box, the ego's nominal unbraked centerline, and a few other-agent trajectories in both modes.

The plot shows the two clusters of trajectories clearly. Continue-mode agents move north toward or through the box; yield-mode agents stop south of it. The ego's center crosses the box at $x \in [-2.5, 2.5]$. Under the stated rectangular toy footprints, simultaneous occupancy of the box's interior by the two centers is exactly the positive-area overlap condition.

The other agent's center position over time traces a curve in the $(t, y)$ plane. We discretize the lateral coordinate and estimate the probability that its center lies in each $y$ cell at each time step. Each time row sums to one: this is a per-time center-position distribution, not a distribution over all space-time cells. Physical footprint occupancy is different: one vehicle can cover several cells at once, so those cell-occupancy probabilities need not sum to one. This simplified display is small enough to inspect by eye.

We compute smoothed empirical center-position grids from ground-truth held-out trajectories grouped by their true modes, then combine the grids with equal weights. These dataset-averaged displays illustrate a possible output format; neither trained predictor produced them for the current scene. The Gaussian kernel only makes the sample positions visible as cell probabilities and is not a calibrated uncertainty model.

Formally, the mode-marginal position probability at time $t$ and lateral cell $y$ is

$$P_{\text{mix}}(y \mid t) = \sum_{k \in \{0,1\}} \pi_k \, P_k(y \mid t)$$

where:

• $P_k(y \mid t)$: the smoothed empirical probability mass for the agent's center in lateral cell $y$ at time $t$ among held-out examples with true mode $k$; its cells sum to one separately at each $t$
• $\pi_k$: the prior probability of mode $k$, here $\pi_0 = \pi_1 = 0.5$ since modes are drawn uniformly
• $\sum_{k} \pi_k \, P_k(y \mid t)$: the per-time position marginal under the two-mode mixture, averaged here across the test scenes for display

This uses the same equal weights as the data-generating mode prior, but the time-indexed marginals discard temporal dependence. The grid shows where empirical center-position mass lies at each time; it cannot by itself give the probability of an overlap at any time across an interval. The later planner checks complete scene-specific predicted trajectories and their weights instead of consuming this dataset-averaged display grid.

Now we train a small MLP that outputs a single trajectory. The training runs in the cell immediately below. It minimizes mean squared error, whose unrestricted population optimum is the conditional mean of the future given the input. A finite network trained for a finite number of steps only approximates that target. The architecture is deliberately simple, since the point of the experiment is the loss function and the output structure, not the network's capacity.

The training MSE is about $22\,\mathrm{m}^2$ per trajectory position for this seed. Under squared error, the population-optimal point prediction is the conditional mean $\mathbb{E}[\tau \mid o_{1:t},m] = \sum_k w_k\tau^k$. A finite-capacity network trained for a finite number of steps need not equal that mean exactly, but here it approaches a path between the two labeled behaviors. That path is a poor summary for a threshold collision check even when its average squared error is reasonable.

The multimodal predictor has a shared trunk and two heads. One head outputs mode logits, which softmax converts to probabilities; the other outputs one trajectory per mode. Because this synthetic dataset gives us the true behavior label for every training trajectory, we use that label to train the corresponding trajectory head and a cross-entropy objective for the mode probabilities. This makes the two output modes identifiable in this controlled example; it does not establish that an unlabeled mixture model would learn them as reliably.

The training objective is

$$\mathcal{L}(\theta) = \frac{1}{N} \sum_{n=1}^{N}\left[\frac{1}{T_{\rm pred}+1}\sum_{t=0}^{T_{\rm pred}}\frac{\bigl(\tau_t^{(n)}-\tau_t^{z_n,(n)}\bigr)^2}{(1\,\mathrm{m})^2} - \log w_{z_n}^{(n)}\right]$$

where:

• $N$: number of training samples
• $T_{\rm pred}$: number of prediction steps after the stored position at $t=0$; each trajectory has $T_{\rm pred}+1$ positions
• $z_n$: the observed continue-or-yield label for sample $n$
• $\tau^{(n)}$: the observed future for sample $n$
• $\tau^{z_n,(n)}$: the predicted trajectory assigned to the labeled mode
• $w_{z_n}^{(n)}$: the predicted probability of the labeled mode

The first term fits the trajectory head associated with the known label; the second trains the predicted mode probability. Dividing by a one-meter squared scale makes the position term dimensionless and leaves the numerical loss computed from meter-valued coordinates unchanged. It also fixes the relative weighting of position error and mode cross-entropy when interpreting the equation in physical units. Since the current input $(y_0,v_0)$ does not reveal intent, a calibrated classifier should assign approximately equal probabilities to the two modes. This labeled construction lets us examine the decisions of a two-mode predictor in this controlled example; learning modes without labels is a harder problem.

Now we evaluate the two learned predictors and a constant-velocity baseline on the test set, in a computation cell that runs once and prepares the metric values that figure cells consume. We compute seven metrics:

• Mean squared position error (MSE) of constant-velocity extrapolation overall and separately for the true continue and yield subsets.
• Mean squared position error (MSE) of the single predictor.
• Mean squared position error of the mixture mean, i.e., the probability-weighted average of the two modes.
• Min-of-K squared error for the two-mode predictor: the error of the closest candidate trajectory.
• Mode coverage: the fraction of test trajectories for which at least one mode is within a tolerance.

These metrics compare point-prediction error with candidate-set accuracy. The constant-velocity errors expose how much the extrapolation depends on the true behavior. The single and mixture-mean errors measure average squared error; the last two metrics describe best-candidate time-averaged error and coverage at a stated threshold. None directly measures action safety or unnecessary braking, and the candidate-set metrics ignore mode probabilities.

The plot shows the tension. The true future moves north toward the intersection. The single predictor, forced to choose one trajectory, produces a slower mean path. The two-mode predictor shows a continue path near the truth and a separate yield path.

Prediction metrics are useful diagnostics but are not sufficient to evaluate action safety. What matters for safety is whether the predictor enables a useful ego decision. In this one-decision scenario, the ego observes $(y_0, v_0)$, asks the predictor for a future, checks which predicted modes overlap its planned crossing window, sums their weights, and brakes if that risk score exceeds 0.1. The choice is Boolean: if it brakes, the ego follows a fixed $5\,\mathrm{m/s^2}$ deceleration profile. We then roll out both agents for up to six seconds, stopping at the first detected sampled footprint overlap. There is no repeated observation or replanning, so this is not a reactive closed-loop driving benchmark.

In the single-predictor panel, the ego crosses while the other agent is in the shared box. In the two-mode panel, the ego brakes and the other agent passes. The initial conditions and decision rule are the same. Only the predictions differ. This selected case does not show the cost of the additional braking in other cases.

The experiment is deliberately small, but it isolates a useful failure mode: an MSE-trained single trajectory can conceal a hazardous mode. Representing separate modes lets the simple planner avoid toy-footprint overlaps in this dataset, at the cost of many extra brakes compared with the oracle. Yet the constant-velocity baseline also avoids overlaps and brakes less often than the learned mixture in this particular toy. Richer prediction alone does not guarantee a better decision. The experiment therefore motivates measuring both modeled safety and progress or intervention cost, rather than prediction error alone.

Three caveats are important. The example uses two labeled behaviors even though real actions vary continuously. The other agent follows a fixed policy and never reacts to the ego; its predictions are action-independent. The ego makes one threshold decision at $t=0$ rather than repeatedly observing and replanning. The example is therefore an action-outcome check, not an action-conditioned response model or a realistic closed-loop driving simulation.