Autonomous Driving › Other-Agent Behavior Prediction

If the spatial state answers "what is where," behavior prediction asks "what might other road users do?" Other agents are decision-makers with goals and information the ego cannot observe directly. Vehicle dynamics constrain their motion, but do not determine their choices. A planning model therefore needs to represent multiple plausible behaviors, often with probabilities rather than only a kinematic extrapolation. A reachable or robust uncertainty set is another possible representation.

A standard formulation is trajectory prediction: given agent histories and map context, forecast positions over a specified horizon. Recurrent networks, graph neural networks, transformers, and diffusion-based methods have all been studied. Benchmarks such as Argoverse and nuScenes report displacement errors, but their values depend on the dataset, agent class, horizon, and protocol and do not directly measure decision safety.

A mean trajectory can be a poor planning summary when the future has distinct modes. In the toy intersection, the other vehicle either yields before the box or continues through it. Their mean is a slower path that need not resemble either behavior. A planner using only that mean may miss overlap risk; a planner that always protects against every plausible mode may instead brake unnecessarily. Both errors matter. The toy's risk threshold favors avoiding predicted overlap, and we separately count extra braking rather than optimizing an explicit cost function.

For this problem, a useful output is a set of trajectory modes with associated probabilities. The following expression is a discrete approximation: it assigns probability mass to each complete candidate trajectory rather than spreading density continuously over all trajectories.

$$p(\tau \mid o_{1:t}, m) = \sum_{k=1}^K w_k(o_{1:t},m) \, \delta\!\left(\tau - \tau^k(o_{1:t},m)\right)$$

where:

• $o_{1:t}$: the available observation history up to the current time $t$; hidden agents need not have observed tracks
• $m$: the map (lane geometry, rules) that conditions the prediction
• $\tau^k$: the $k$-th mode trajectory, a full future path over the prediction horizon
• $w_k$: the probability (mixture weight) assigned to mode $k$, with $w_k\geq 0$ and $\sum_{k=1}^K w_k = 1$
• $\delta(\tau - \tau^k)$: a point mass at the $k$-th mode trajectory, so the output is a discrete mixture over trajectories

In words, the predictor assigns weights to $K$ complete candidate trajectories. Both the weights and candidate paths depend on the available history and map. Sampling this predictive distribution selects one modeled candidate. The actual future need not equal any candidate. The planner can inspect all modeled candidates. Its expectation $\sum_k w_k\tau^k$ generally lies outside the finite set of candidates. When the second moment exists, that mean minimizes expected squared prediction error among unrestricted point trajectories. It need not be a dynamically plausible complete path. It is also not necessarily a sufficient summary of overlap risk.

Modes may correspond to interpretable intents such as turning, yielding, or continuing, although learned components do not always align cleanly with those labels. A network can produce candidate paths and logits from a shared representation, with $w_k=\mathrm{softmax}(\text{logits})_k$. The number $K$ trades candidate coverage against computation: too few candidates may miss relevant behavior, while more candidates raise inference and planning cost and call for evaluation. In fixed-output models, changing $K$ can also change the learned output head; sampling-based models can draw more candidates after training. If candidate weights are used as probabilities, their calibration should be checked.

Several approaches can produce multimodal predictions.

• Mixture density networks output mixture parameters directly. They can be trained with a mixture negative log-likelihood, but optimization may require scale constraints and the modes can collapse or overlap.
• Anchor-based methods start with reference endpoints or trajectories and learn to select or refine them. Anchors encourage coverage of specified behaviors but do not guarantee useful diversity after refinement.
• Latent-variable methods introduce a discrete or continuous latent $z$ and decode a trajectory conditional on it. Diffusion-based samplers are another route to diverse predictions, not necessarily the same parameterization as an anchor or fixed latent mixture.

The choice matters for coverage, probability calibration, and planning cost. An anchor can often be refined beyond its initial path, but its placement still affects what the model learns. Mixture and latent-variable approaches can also duplicate or collapse modes; diversity must be checked empirically rather than assumed from the architecture.

A second dimension is interaction awareness. Road users may respond to one another: a vehicle approaching an occupied intersection may slow, and a pedestrian may wait for a gap. Independent forecasts can miss this coupling. One approach encodes agents jointly, using a scene graph and attention so predictions depend on the shared context. Other interaction models are possible, and their computational cost depends on how many agent relationships they represent. In dense scenes, an independent predictor should be checked for mutually inconsistent futures rather than assumed adequate.

A third dimension is action conditioning. When a planner needs to evaluate how other agents would react to alternative ego actions, its response prediction needs to condition on each proposed action as well as the observed past. If the ego accelerates, the other agent may yield; if the ego brakes, the other may continue. A simpler planner can use action-independent forecasts when those responses are outside its model, as the toy below does, but it cannot then claim to predict interactive counterfactual responses. Learning such responses is hard: logged data records only one realized ego action and outcome per situation, while a planner asks what would happen under another action. That is an interventional question, not automatically answered by an observational forecast.

Action-conditioned prediction is closely related to a response model: a function that, given the current scene and a proposed ego action, returns a distribution of other-agent responses. Training on logged scenes with logged ego actions can estimate an associational conditional distribution, $P(Y\mid X,A)$, where $X$ is the represented scene, $A$ the ego action, and $Y$ the other-agent outcome. Using that estimate as an interventional response, $P(Y\mid X,\operatorname{do}(A))$, requires a well-defined ego action, consistency, adequate action support, and no unmeasured confounding after conditioning on a sufficient scene state. For action sequences, corresponding assumptions apply at each decision step. Hidden factors affecting both the logged action and the other agent's response can break the equivalence even for actions observed in the logs; unsupported actions add extrapolation risk. Restricting plans to well-supported actions can reduce extrapolation risk. It does not by itself identify the causal response. Interactive simulation, interventions where feasible, and sensitivity tests are ways to probe the remaining gap.

Finally, evaluation of behavior prediction is subtle. Low average displacement error does not guarantee that a predictor covers a rare hazardous future. Here, per-agent min-of-K error measures the time-averaged error of the best complete candidate. It does not assess candidate probabilities, joint interaction consistency, or the actions a planner takes. The worked example compares candidate coverage with action outcomes to show why both kinds of measurement are needed.

We'll return to evaluation in Planning, Control, and Policy Evaluation. For now, a trajectory predictor used for planning must represent futures that materially change the decision. The loss and architecture affect whether the predictor represents those futures. Whether candidate weights are calibrated affects how reliably the planner can interpret them as probabilities. The planner's risk rule affects whether represented futures influence its action.