Stochasticity and Uncertainty › Why Deterministic Models Fail in a Random World

Earlier chapters used transition models of the form $s_{t+1} = f(s_t, a_t)$, a function that maps state and action to next state. Deterministic environments are valid special cases, representable by a transition distribution concentrated at $f(s_t,a_t)$. When unresolved state, stochastic forcing, sensors, or other agents matter, the more general object is a conditional distribution from which $s_{t+1}$ is drawn:

$$s_{t+1} \sim p(s_{t+1} \mid s_t, a_t)$$

where:

• $s_t$: the state of the world at time $t$
• $a_t$: the action taken by the agent at time $t$
• $s_{t+1}$: the state at the next time step, now a random draw rather than a deterministic output
• $p(s_{t+1} \mid s_t, a_t)$: the conditional transition distribution over the chosen state representation; it is Markov-complete only when the conditioning state is sufficient

Notice what this notation changes conceptually. The same measured state and action can lead to different outcomes. Repeated differences falsify a deterministic transition only relative to the variables measured and controlled; hidden state, measurement error, uncontrolled inputs, and nonstationarity remain alternative explanations.

The randomness in this distribution has several distinct sources, and it is worth naming them because they demand different responses:

• Aleatoric uncertainty relative to the chosen representation: unresolved contact branches, sensor noise, other agents' decisions, or turbulent forcing can remain random after conditioning on the variables the model receives. More repetitions do not remove that conditional spread; the model must represent it.
• Hidden state: the transition looks random only because the state is not sufficient. As we discussed in the chapter on sufficient state, apparent noise can reflect a missing variable, and the causal framing of the previous chapters tells us when that is the case. In a constructed door model where hinge angle is the only omitted determinant, adding it to the state makes the open-versus-jam transition deterministic. The correct response is to find and include the missing variable.
• Model ignorance: the true dynamics may be effectively deterministic, but our model class cannot represent them, or our data does not identify the parameters. A better-specified model or additional informative data can reduce this uncertainty, although misspecification and non-identifiability can prevent it from vanishing.

These sources are blended in finite datasets. The distinction matters because informative data can constrain model ignorance, richer state can explain apparent noise, and an expressive distribution can represent spread that remains after conditioning. The categories are therefore relative to the model's information set, not immutable labels attached to a physical phenomenon.

Assuming a finite conditional second moment, population squared-error risk is minimized pointwise by the conditional mean $\mathbb{E}[s_{t+1} \mid s_t, a_t]$. If optimization reaches the global population-risk minimum within a restricted function class, the result is that class's best $L^2$ approximation. Finite samples, regularization, or imperfect optimization can move the fitted regressor away from that best-in-class target. The conditional mean can still lose branch identity when the predictive distribution has separated modes. Consider a ball that deflects left or right around a wedge. Its conditional mean can lie in a low-density gap between the two branches, even though neither branch passes through that location.

The figure below shows a one-dimensional construction: sampled next states concentrate at two separated deflections, while the empirical mean lies in a gap where this finite sample contains no outcome.

A planner that treats this mean as the realized next state can optimize for the low-density gap rather than either likely branch. Robustness, replanning, or explicit uncertainty can prevent that mistake.

A single predicted trajectory contains no internal decomposition of extrapolation, model uncertainty, and outcome noise. Two identical point outputs can therefore have different epistemic status. External residual and out-of-distribution diagnostics can add information, while uncertainty estimates are one important aid when decisions query poorly covered regions.