Bayesian Filtering and Belief States › The Bayes Filter

This section presents the general recursion, examines why the belief is the right object to maintain, and states the assumptions quietly baked into every filter we will build. Nothing here requires any particular functional form: the derivation holds for discrete states, continuous states, hybrid states, and arbitrary noise distributions. That generality is precisely what makes the abstract filter usually impractical as a direct implementation and invaluable as a reference point, since every practical method can be understood as a specific approximation strategy applied to these equations.

Assume we have three ingredients:

• Transition model $p(x_t \mid x_{t-1}, a_{t-1})$: how the state evolves given the previous state and the action taken.
• Observation model $p(o_t \mid x_t)$: how measurements are generated from the current state.
• Initial belief $b_0(x) = p(x_0)$: what the agent believes before seeing anything.

These three objects constitute the generative model of the partially observable system. Notice what they do and do not contain: they say nothing about rewards, policies, or plans. Filtering is purely an inference problem, and its outputs are beliefs, not decisions. The bridge from beliefs to decisions comes later, once we treat the belief as the state of a fully observable MDP.

The Bayes filter alternates two updates. The prediction step pushes the belief through the dynamics:

$$\bar{b}_t(x_t) \;=\; p(x_t \mid o_{1:t-1}, a_{0:t-1}) \;=\; \int p(x_t \mid x_{t-1}, a_{t-1})\, b_{t-1}(x_{t-1})\, dx_{t-1}.$$

The integral marginalizes over everywhere the state could have been, weighted by where the agent believed it was. For a discrete state space, replace the integral with a sum; hybrid spaces require the corresponding mixed sum-integral. Uncertainty often increases during prediction because process noise adds possible successors. It need not increase: deterministic or stochastic dynamics that merge or contract states can reduce entropy. In the familiar tracking case, uncertainty about which lane a car occupies becomes uncertainty about a wider set of forward positions, while added process noise spreads that predictive belief further.

The correction step folds in the new observation using Bayes' rule:

$$b_t(x_t) \;=\; p(x_t \mid o_{1:t}, a_{0:t-1}) \;=\; \eta \; p(o_t \mid x_t)\, \bar{b}_t(x_t),$$

where $\eta = p(o_t \mid o_{1:t-1}, a_{0:t-1})^{-1}$ is a normalizing constant. States that would have produced the observed data with high probability gain mass; implausible states lose it. Geometrically, the correction multiplies the predicted belief pointwise by the likelihood function, so the posterior concentrates wherever the two overlap. If the prediction says the drone is somewhere along a corridor and the camera sees a distinctive doorway, only the portion of the corridor near that doorway survives with appreciable probability. One over the normalizing constant, $1/\eta$, is the predictive evidence for the observation. In the Kalman setting, the innovation is Gaussian with covariance $S$; evaluating that innovation under its predicted distribution provides the analogous model-health signal. The covariance $S$ is not itself the evidence.

These two lines organize the chapter. The classical filters compute or approximate the two updates explicitly. A learned recurrent state may instead approximate their composition without representing either intermediate distribution. In either case, the useful questions are what information the update preserves, what assumptions make it valid, and what its approximation cannot represent.

The figure makes the rhythm concrete. Under the identity dynamics used here, the dotted prior keeps its mean while additive process noise produces the wider predicted belief, whose variance is the sum of prior and process variances. The likelihood encodes what the new measurement says about each candidate state. Multiplying the predicted belief and likelihood, then renormalizing, produces the posterior. The prior is shown for reference; every classical filter in this chapter computes some version of the prediction, likelihood weighting, and posterior. The learned filter later approximates their composition directly with a recurrent update.

It is worth pausing on why conditioning on the whole history collapses into a single distribution. For an open-loop or randomized action sequence chosen independently of the latent trajectory, the conditional joint over states and observations factors as

$$p(x_{0:t}, o_{1:t} \mid a_{0:t-1}) \;=\; p(x_0) \prod_{\tau=1}^{t} p(x_\tau \mid x_{\tau-1}, a_{\tau-1})\, p(o_\tau \mid x_\tau),$$

which uses two structural assumptions, together with the open-loop treatment of the action sequence:

• Markov dynamics: the next state depends on history only through the current state and action.
• Conditional observation independence: the observation depends on history only through the current state.

The notation here uses stationary transition and observation models; time-varying models follow by adding a time index. A feedback policy generally makes the full action sequence statistically informative about earlier states, so the displayed conditional-joint factorization is not the right derivation in that setting. The same Bayes-filter recursion still follows sequentially under the standard controlled Markov and observation assumptions: condition on the realized action at each update, predict from the previous belief, then correct with the new observation.

Under these assumptions, the exact belief $b_t$ satisfies a self-contained recursion: $b_t$ is computable from $b_{t-1}$, $a_{t-1}$, and $o_t$ alone. The claim is not that history is irrelevant or recoverable from the belief. Rather, the posterior contains the history's information needed to predict future states and observations. For control, the belief is also sufficient under the standard POMDP assumption that rewards depend on the current state and action. With those conditions, two agents holding the same exact belief face the same predictive and decision problem even if their past observations differ. This is what licenses treating the belief as the state of an equivalent fully observable MDP. Approximate beliefs, including finite particle sets and learned recurrent states, inherit this property only approximately. QMDP and belief-tree search are two common planning methods that operate on the belief state, although they exploit the belief-MDP construction differently: QMDP solves the underlying fully observable MDP and averages its value function over the belief, ignoring future observations, whereas belief-tree search plans forward in the belief MDP itself.

What happens when the assumptions fail matters, because they fail often in practice. If the true system has hidden variables that persist across time but are not part of $x_t$ (a changing friction coefficient, an unmodeled bias in a sensor), then the belief over $x_t$ alone is no longer sufficient, and the recursion quietly degrades. One standard remedy is to enlarge the state until the Markov assumption holds again, augmenting position with velocity, or adding slowly varying nuisance parameters to the state vector. Much of practical filter design is exactly this activity: finding the smallest state augmentation that makes the world Markovian enough.

The silent nature of these failures is what makes them dangerous. A blatant numerical bug may announce itself with an exception or a NaN, but numerical instability and violated modeling assumptions can both produce plausible-looking wrong answers. Collapsing covariance, drifting innovations, and disagreement between reported confidence and measured error are concrete signatures examined below.

Keep two roles distinct. The predictive role asks: what does $p(x_{t+1} \mid \text{history})$ look like, and how accurate is it as a forecast? The decision role asks: does the belief support good action selection? These come apart more often than newcomers expect. A filter can track the variables included in its state estimate yet still omit information a particular policy needs, and conversely a crude filter can suffice if the task only requires coarse information. A concrete example makes the point: a thermostat-quality controller may need only a binary belief about whether the room is warm or cold, in which case a full six-degree-of-freedom pose estimate is wasted computation, while a grasping policy may need millimeter-accurate object pose even if its long-horizon forecasts are poor. A careful evaluation therefore asks four related questions: are predictions accurate, can the chosen belief family represent features such as multimodality and correlations, is its uncertainty calibrated, and does it support useful decisions? Representation quality can be probed with held-out likelihood, posterior-shape diagnostics, or downstream probes rather than treated as an undefined score. Open-loop prediction quality and closed-loop decision usefulness must be measured separately. The compact GRU-versus-Kalman benchmark later in this chapter addresses only open-loop estimation error; it does not implement this broader evaluation protocol.