Bayesian Filtering and Belief States

A partially observable decision problem forces an agent to act on beliefs rather than directly observed states. This chapter develops the machinery that turns that insight into working algorithms: the Bayes filter, the recursive computation that converts a stream of noisy observations and past actions into a probability distribution over the current state. Every method in this chapter answers the same question, namely "given everything you have seen and done so far, what should you now believe about the world?", and they differ in how they represent and approximate that belief. That shared question is worth internalizing because it tells you what to inspect in a new filtering method: how the belief is stored, propagated, and corrected.

Filtering is the inference half of a world model. The transition and observation models describe how the world behaves; the filter describes how the agent should update its knowledge. Keeping these two roles distinct matters when diagnosing a deployed system. Otherwise it is easy to blame a filter for errors that originate in a bad transition model, or to tune noise covariances to compensate for a structural model defect. System identification is the complementary problem of learning the models that a filter consumes.

We will build the story in four passes over the same recursion. First, the abstract Bayes filter, which is exact but usually intractable. Second, the Kalman filter, which is exact for a linear transition and observation model when the process and observation noise are Gaussian, mutually independent, independent of the Gaussian initial belief, and independent across time. Third, approximations for nonlinear systems: extended and unscented Kalman filters, plus particle filters. Fourth, learned belief updates, where a neural network amortizes the filtering computation itself. Each pass trades some form of exactness or generality for tractability, and by the end you should be able to look at any filtering algorithm, classical or learned, and locate it on this spectrum: what does it assume about the dynamics, what family of distributions can its belief represent, and where does it spend its computational budget?

Consider a drone flying indoors. Its onboard camera reports visual features, its IMU reports accelerations and rotation rates, and none of these sensors directly tells the drone where it is. Position must be inferred: the IMU integrates to a rough pose estimate that drifts, while mapped visual landmarks anchor that estimate but remain noisy and occasionally ambiguous. Neither sensor alone suffices. Their complementary failure modes make a distributional belief essential: aliasing and missed detections can preserve several plausible poses even after a landmark observation.

This is the filtering problem in miniature. The agent holds a belief: a probability distribution over the hidden state of the world. Each time step, two things happen. The world evolves according to dynamics the agent does not fully control or observe, so the belief must be pushed forward through the transition model. Process noise often spreads that predictive belief, although deterministic dynamics can also compress it. Then a new observation arrives, and the belief is corrected by weighting states according to how likely they would have produced that observation. Averaging over the observations the model itself predicts, conditioning reduces uncertainty, but a surprising individual observation can leave a realized posterior more uncertain than the preceding belief. The classical filters below implement this predict-then-correct alternation explicitly. The learned recurrent update later approximates the composed recursion in one forward computation and need not expose a separate predictive belief.

The phrase "all the agent needs" is doing real work. For a specified POMDP, a correctly maintained belief is sufficient for predicting future states and observations and for computing decision-relevant quantities such as the optimal value function. It does not preserve arbitrary facts about the past, such as the exact observation sequence or its length. To appreciate why this is useful, consider what the raw history looks like: a list of observations and actions that grows without bound as time passes, with no obvious way to compare two different-length histories. The belief replaces the decision-relevant information in that history with a single object of fixed type, a distribution over states. This is why the belief state matters beyond estimation. A POMDP can be rewritten as a belief-space MDP, an ordinary Markov decision process whose states are distributions, because the exact belief update below depends on the past only through the current belief and the new action-observation pair. Filtering is therefore not a preprocessing step bolted onto control; it is what restores the Markov property that planning and reinforcement learning require. Without it, value functions would need to condition on entire histories, and the dynamic programming machinery that makes sequential decision problems solvable no longer applies directly.

This section presents the general recursion, examines why the belief is the right object to maintain, and states the assumptions quietly baked into every filter we will build. Nothing here requires any particular functional form: the derivation holds for discrete states, continuous states, hybrid states, and arbitrary noise distributions. That generality is precisely what makes the abstract filter usually impractical as a direct implementation and invaluable as a reference point, since every practical method can be understood as a specific approximation strategy applied to these equations.

Assume we have three ingredients:

• Transition model $p(x_t \mid x_{t-1}, a_{t-1})$: how the state evolves given the previous state and the action taken.
• Observation model $p(o_t \mid x_t)$: how measurements are generated from the current state.
• Initial belief $b_0(x) = p(x_0)$: what the agent believes before seeing anything.

These three objects constitute the generative model of the partially observable system. Notice what they do and do not contain: they say nothing about rewards, policies, or plans. Filtering is purely an inference problem, and its outputs are beliefs, not decisions. The bridge from beliefs to decisions comes later, once we treat the belief as the state of a fully observable MDP.

The Bayes filter alternates two updates. The prediction step pushes the belief through the dynamics:

The integral marginalizes over everywhere the state could have been, weighted by where the agent believed it was. For a discrete state space, replace the integral with a sum; hybrid spaces require the corresponding mixed sum-integral. Uncertainty often increases during prediction because process noise adds possible successors. It need not increase: deterministic or stochastic dynamics that merge or contract states can reduce entropy. In the familiar tracking case, uncertainty about which lane a car occupies becomes uncertainty about a wider set of forward positions, while added process noise spreads that predictive belief further.

The correction step folds in the new observation using Bayes' rule:

where $\eta = p(o_t \mid o_{1:t-1}, a_{0:t-1})^{-1}$ is a normalizing constant. States that would have produced the observed data with high probability gain mass; implausible states lose it. Geometrically, the correction multiplies the predicted belief pointwise by the likelihood function, so the posterior concentrates wherever the two overlap. If the prediction says the drone is somewhere along a corridor and the camera sees a distinctive doorway, only the portion of the corridor near that doorway survives with appreciable probability. One over the normalizing constant, $1/\eta$, is the predictive evidence for the observation. In the Kalman setting, the innovation is Gaussian with covariance $S$; evaluating that innovation under its predicted distribution provides the analogous model-health signal. The covariance $S$ is not itself the evidence.

These two lines organize the chapter. The classical filters compute or approximate the two updates explicitly. A learned recurrent state may instead approximate their composition without representing either intermediate distribution. In either case, the useful questions are what information the update preserves, what assumptions make it valid, and what its approximation cannot represent.

The figure makes the rhythm concrete. Under the identity dynamics used here, the dotted prior keeps its mean while additive process noise produces the wider predicted belief, whose variance is the sum of prior and process variances. The likelihood encodes what the new measurement says about each candidate state. Multiplying the predicted belief and likelihood, then renormalizing, produces the posterior. The prior is shown for reference; every classical filter in this chapter computes some version of the prediction, likelihood weighting, and posterior. The learned filter later approximates their composition directly with a recurrent update.

It is worth pausing on why conditioning on the whole history collapses into a single distribution. For an open-loop or randomized action sequence chosen independently of the latent trajectory, the conditional joint over states and observations factors as

which uses two structural assumptions, together with the open-loop treatment of the action sequence:

• Markov dynamics: the next state depends on history only through the current state and action.
• Conditional observation independence: the observation depends on history only through the current state.

The notation here uses stationary transition and observation models; time-varying models follow by adding a time index. A feedback policy generally makes the full action sequence statistically informative about earlier states, so the displayed conditional-joint factorization is not the right derivation in that setting. The same Bayes-filter recursion still follows sequentially under the standard controlled Markov and observation assumptions: condition on the realized action at each update, predict from the previous belief, then correct with the new observation.

Under these assumptions, the exact belief $b_t$ satisfies a self-contained recursion: $b_t$ is computable from $b_{t-1}$, $a_{t-1}$, and $o_t$ alone. The claim is not that history is irrelevant or recoverable from the belief. Rather, the posterior contains the history's information needed to predict future states and observations. For control, the belief is also sufficient under the standard POMDP assumption that rewards depend on the current state and action. With those conditions, two agents holding the same exact belief face the same predictive and decision problem even if their past observations differ. This is what licenses treating the belief as the state of an equivalent fully observable MDP. Approximate beliefs, including finite particle sets and learned recurrent states, inherit this property only approximately. QMDP and belief-tree search are two common planning methods that operate on the belief state, although they exploit the belief-MDP construction differently: QMDP solves the underlying fully observable MDP and averages its value function over the belief, ignoring future observations, whereas belief-tree search plans forward in the belief MDP itself.

What happens when the assumptions fail matters, because they fail often in practice. If the true system has hidden variables that persist across time but are not part of $x_t$ (a changing friction coefficient, an unmodeled bias in a sensor), then the belief over $x_t$ alone is no longer sufficient, and the recursion quietly degrades. One standard remedy is to enlarge the state until the Markov assumption holds again, augmenting position with velocity, or adding slowly varying nuisance parameters to the state vector. Much of practical filter design is exactly this activity: finding the smallest state augmentation that makes the world Markovian enough.

The silent nature of these failures is what makes them dangerous. A blatant numerical bug may announce itself with an exception or a NaN, but numerical instability and violated modeling assumptions can both produce plausible-looking wrong answers. Collapsing covariance, drifting innovations, and disagreement between reported confidence and measured error are concrete signatures examined below.

Keep two roles distinct. The predictive role asks: what does $p(x_{t+1} \mid \text{history})$ look like, and how accurate is it as a forecast? The decision role asks: does the belief support good action selection? These come apart more often than newcomers expect. A filter can track the variables included in its state estimate yet still omit information a particular policy needs, and conversely a crude filter can suffice if the task only requires coarse information. A concrete example makes the point: a thermostat-quality controller may need only a binary belief about whether the room is warm or cold, in which case a full six-degree-of-freedom pose estimate is wasted computation, while a grasping policy may need millimeter-accurate object pose even if its long-horizon forecasts are poor. A careful evaluation therefore asks four related questions: are predictions accurate, can the chosen belief family represent features such as multimodality and correlations, is its uncertainty calibrated, and does it support useful decisions? Representation quality can be probed with held-out likelihood, posterior-shape diagnostics, or downstream probes rather than treated as an undefined score. Open-loop prediction quality and closed-loop decision usefulness must be measured separately. The compact GRU-versus-Kalman benchmark later in this chapter addresses only open-loop estimation error; it does not implement this broader evaluation protocol.

The Bayes filter is exact but involves integrals over arbitrary distributions, which is why it is rarely computable as written. Rudolf Kálmán's 1960 insight was that an entire class of problems admits a closed-form solution: if the transition and observation models are linear, the process and observation noise are Gaussian, and the initial belief is Gaussian, the belief stays Gaussian forever. The recursion then reduces to updating a mean vector and a covariance matrix. This is a remarkable compression of the general problem. Instead of tracking a function over an infinite-dimensional space of distributions, the filter tracks finitely many numbers. Those numbers update through matrix operations that were feasible on 1960s hardware, although numerical stability required careful implementations. The filter was used in Apollo-era navigation within a decade of its publication.

Take a linear transition with additive Gaussian noise,

and a linear observation with Gaussian noise,

Assume that the initial belief, the full process-noise sequence, and the full observation-noise sequence are mutually independent, with fresh noise draws independent across time. If $b_{t-1}$ is Gaussian, then the prediction $\bar{b}_t$ is a linear transformation of a Gaussian plus independent Gaussian noise, hence Gaussian, with mean and covariance propagated by matrix algebra. The correction step multiplies the predicted Gaussian density by the likelihood $p(o_t \mid x_t)$ and normalizes the result. The product is proportional to a Gaussian. For a direct scalar observation its precision is the sum of the two scalar precisions; in the general linear case the likelihood contributes $H^\top R^{-1}H$ to state-space precision. This closure property does all the work. No integrals remain, only arithmetic on means and covariances.

It is worth spelling out the closure argument once, slowly, because the same pattern recurs whenever someone claims a filter is "exact." A distribution family is closed under an operation if applying the operation to members of the family yields another member. Linear maps take Gaussians to Gaussians, sums of independent Gaussians are Gaussian, and products of Gaussian densities are proportional to Gaussians, exactly Gaussian once normalized, which the correction step does. Since prediction uses linear maps plus sums, and correction uses normalized products, both steps stay inside the family. Nonlinearity breaks the first link, and non-Gaussianity breaks the second and third. The classical approximations below repair one of those broken links; the learned update later replaces the explicit recursion with a trained recurrent computation.

Start from an initial Gaussian belief $\mathcal{N}(\hat{x}_0, P_0)$. The mean $\hat{x}_0$ and covariance $P_0$ are part of the filter specification: they encode what is known before the first measurement.

Predict:

Correct:

Here $\hat{x}_t$ and $P_t$ are the posterior mean and covariance after absorbing the measurement; the $t\mid t-1$ subscript marks predicted values computed before it arrives. And $o_t - H\hat{x}_{t|t-1}$ is the innovation: the part of the measurement the model did not anticipate. Read the equations as a conversation between model and data. The predict step speaks first, asserting where the state should be and how confident the assertion is. The measurement replies, and the innovation records the disagreement. The gain decides how much the disagreement matters, and the final update moves the mean partway toward resolving it while shrinking the covariance to reflect the information gained. Every symbol plays a role in this dialogue, and none of them is decoration.

Everything interesting hides in the Kalman gain $K_t$. Look at its structure. The denominator contains $H P_{t|t-1} H^\top$, the predicted uncertainty of the measurement, plus $R$, the sensor noise. The numerator $P_{t|t-1}H^\top$ is the cross-covariance between the state and measurement. So the gain is essentially a ratio:

When the model is confident ($P$ small) or the sensor is poor ($R$ large), the gain shrinks and corrections barely move the estimate. When the model is uncertain or the sensor precise, the gain gives more weight to the measurement. The filter continuously re-weights trust between its own model and incoming evidence, and it does so optimally in the mean-squared-error sense for this problem class. This trust-balancing interpretation is the single most transferable idea in the chapter: every estimator, learned or analytic, faces the same question of how much to believe the internal model versus the latest evidence, and the Kalman gain is the optimal answer in the linear-Gaussian setting treated here.

For the scalar observation case plotted below, with $H=1$, the gain reduces to $K=p/(p+R)$, where $p$ is the predicted variance. In multiple dimensions $K=P H^\top(H P H^\top+R)^{-1}$ is a matrix, so neither its entries nor its eigenvalues generally follow the scalar curve.

Each curve traces the scalar trade-off at a different sensor quality. With a precise sensor ($R = 0.25$) the gain climbs quickly: even modest model uncertainty hands most of the correction weight to the measurement. The noisy-sensor curve ($R = 4$) remains the lowest of the three throughout the plotted range, but it is not always below the half-trust line: it crosses $K=0.5$ at $p=4$ and reaches $K=10/14\approx0.714$ at the right edge. In this $H=1$ scalar case, each crossing with the dotted half-trust line occurs exactly where $p=R$. For a general observation matrix, inspect the full matrix gain and innovation covariance instead of applying this scalar crossing rule componentwise.

Two further facts deepen the picture. First, the covariance update does not depend on the observed values, so with $A$, $H$, $Q$, $R$, and $P_0$ fixed, the full sequence of gains can be precomputed offline. Second, for a time-invariant system the gain converges to a fixed matrix solving a discrete algebraic Riccati equation under the usual stabilizability and detectability conditions. This is the object that connects filtering to optimal control. For linear-quadratic-Gaussian problems, the separation principle permits independent estimator and controller design with no loss of optimality. That independence of design generally disappears in nonlinear and learned settings, where the controller changes the distribution of states on which the estimator operates.

Real systems are nonlinear: angles wrap, contacts switch, aerodynamics saturate. Once either the dynamics or the observation model bends away from linearity, Gaussians no longer close, and we must approximate. The two dominant classical strategies push in opposite directions: the extended Kalman filter forces the belief to stay Gaussian and linearizes locally, while particle filters abandon parametric forms entirely and carry samples. Between these poles lies a spectrum of compromises, and choosing among them is fundamentally a choice about which failure mode you can afford: the bias of a wrong parametric family, or the variance and cost of Monte Carlo approximation.

The extended Kalman filter (EKF) keeps the Gaussian belief and uses first-order Taylor expansions of the nonlinear functions to build linearized covariance and correction maps. Write the general system as

with additive noise covariances $Q$ and $R$. The mean prediction uses the nonlinear transition $f$, and the predicted observation uses the nonlinear function $h$. The EKF linearizes only the covariance and correction maps, using the Jacobians

so $P_{t|t-1}=F_tP_{t-1|t-1}F_t^\top+Q$ and the correction uses $H_t$ in the Kalman gain. The intuition is that the Jacobian captures the local sensitivity of the dynamics at the operating point, so the covariance update approximates how neighboring trajectories spread apart or converge near the current estimate, even though the mean follows the full curved dynamics. A separate control-noise Jacobian is needed only when uncertainty enters through the control channel rather than as the additive state noise written above.

The EKF is cheap, ubiquitous, and fragile in equal measure. Its failure modes deserve explicit attention:

• Linearization error compounds over rollouts, because each corrected estimate becomes the expansion point for the next prediction. Small biases accumulate.
• Divergence: if the true belief is multimodal or strongly skewed, the Gaussian approximation can place its mean somewhere no plausible state exists, after which innovations may point the wrong way and the filter can walk away from the truth.
• Jacobian availability: deriving analytic derivatives is laborious; automatic differentiation helps, but discontinuous models (contact, mode switches) resist differentiation altogether.

Each failure mode suggests a partial remedy that practitioners have catalogued over decades: iterating the correction step to refine the expansion point, inflating $Q$ to hedge against unmodeled curvature, or switching to derivative-free variants when Jacobians are unavailable. None of these removes the fundamental constraint, which is the Gaussian bottleneck.

The unscented Kalman filter (UKF) attacks linearization error differently. Rather than differentiating, it propagates a small deterministic set of sigma points through the nonlinear functions and fits a Gaussian to the transformed cloud. A common construction uses $2n+1$ points for an $n$-dimensional state. It captures curvature that a single local Jacobian misses without requiring derivatives, which makes it useful for black-box simulation models. Whether it improves on an EKF depends on the system, tuning, and operating regime; both methods still force the belief into a Gaussian family. Neither can represent a multimodal belief. To see why that limitation bites, imagine localizing a robot in a symmetric corridor from ambiguous landmarks: the true posterior has two peaks, one for each consistent interpretation, and a single Gaussian summary places its mean between them.

The mixture represents two competing interpretations of the same observation. The dashed curve matches its mean and variance, but its mean lies in the low-density valley between the modes. The problem is not high confidence in this plotted Gaussian, which is broad; it is that a single mean and covariance erase the distinction between two hypotheses. A particle representation can retain both modes when its proposal and finite sample set cover them.

The particle filter represents the belief nonparametrically, as a weighted set of samples $\{(x^{(i)}, w^{(i)})\}_{i=1}^N$ approximating $b_t$. Prediction becomes simulation: draw each particle through the transition model. Correction becomes reweighting: each particle's weight scales by its likelihood $p(o_t \mid x_t^{(i)})$. Under standard sequential Monte Carlo regularity conditions, including a proposal whose support covers the target and controlled importance weights, the empirical distribution converges to the true belief as $N \to \infty$ without assuming a parametric belief shape. Multimodality, skewness, and heavy tails can therefore be represented by placing samples where they belong. This asymptotic result is not unconditional: finite particle budgets, poor proposals, unbounded weights, and high state dimension can still make the approximation fail badly.

The practical algorithm, often called the bootstrap or SIR filter (Gordon, Salmond, and Smith, 1993), adds one decisive step:

1. Propagate: sample $x_t^{(i)} \sim p(x_t \mid x_{t-1}^{(i)}, a_{t-1})$ for each particle.
2. Reweight: set $w_t^{(i)} \propto w_{t-1}^{(i)}\, p(o_t \mid x_t^{(i)})$, then normalize.
3. Resample when needed: draw $N$ particles from the current weighted set, resetting weights to uniform.

Resampling addresses weight degeneracy. Repeated likelihood updates can leave nearly all weight on a few particles; the rate depends on the proposal, observation informativeness, state dimension, and dynamics. The effective sample size is estimated as

and collapses toward one as weights concentrate. Resampling removes low-weight particles and duplicates high-weight ones, resetting the weights without restoring hypotheses that have disappeared. Repeated cloning can therefore erode state-space diversity. The particle-filter implementation built in the Code Implementation section ahead resamples when $N_{\text{eff}} < N/2$. Jitter, regularization, and observation-informed proposals can reduce this sample impoverishment.

Particle filters are useful when a finite sample set can represent structure that a Gaussian cannot, including multimodality, nonlinear sensing, and non-Gaussian noise. Their weakness is dimensional: maintaining useful coverage becomes rapidly more expensive as dimension grows, with the practical rate depending on posterior structure and proposal quality. Rao-Blackwellization exploits conditional linear structure by sampling only a smaller nonlinear or discrete component while filtering the remaining variables analytically. More generally, marginalizing a tractable subproblem instead of sampling it reduces Monte Carlo variance.

Nothing builds trust in the recursion like computing one cycle with numbers small enough to check mentally. Consider a scalar system with identity transition and observation matrices, process variance $Q = 0.25$, and measurement variance $R = 1$. Suppose our current belief is $\mathcal{N}(2.0,\, 1.0)$ and we receive the measurement $o = 3.0$.

Predict. With $A = 1$ and no input, the predicted belief is unchanged in mean and inflated in variance:

Correct. The gain blends the two sources of information:

The posterior mean moves from 2.0 toward the measurement 3.0, covering about 56% of the gap:

and the posterior variance contracts:

Notice the bookkeeping of information in this correction. Before the measurement the predicted belief carried $1/1.25$ units of precision; the sensor contributed one more unit; the posterior variance $0.556 = 1/(1/1.25 + 1/1.0)$ is the reciprocal of the summed precisions. Repeated conditionally independent readings add information, while each prediction step can add process uncertainty. Now let us scale this up to systems where hand computation fails.

The three vertical guides mark the predicted mean, the measurement, and the posterior mean. Because the predicted variance (1.25) slightly exceeds the measurement variance (1.0), the posterior lands just past the midpoint of the gap, at 56% of the way to the measurement, exactly as the hand computation predicts. The posterior curve is visibly taller and narrower than both inputs, the graphical signature of precision addition.

We will implement three filters on two test problems: a Kalman filter on a linear-Gaussian tracking task, then an EKF and a particle filter on a nonlinear bearing-only tracking task, and finally a small neural network trained to perform the belief update itself. The progression is deliberate. We start where the analytic solution is optimal under the known model, so we can compare implementation output with simulated state truth. We then move to a problem where the analytic filter is only approximate, and finally remove the equations from the learning procedure to preview amortized inference in modern world models.

Our first testbed is a noisy double integrator: a cart whose state is position and velocity, driven by random accelerations, observed through a noisy position sensor. The Kalman filter is the model-specified analytic baseline for this known linear-Gaussian system. The state components are coupled through time, and that coupling lets the filter infer unobserved velocity from position observations.

The simulation produces paired transitions and noisy position readings. Every trajectory starts from the known state $x_0=0$; the default Kalman initialization uses that same mean but a deliberately broad covariance $P_0=10I$. The sensor sees position with a two-meter standard deviation, while velocity must be inferred from correlations across time. Comparing the result with raw measurement differences checks only that the implementation beats a noisy one-step baseline. It does not isolate model-based fusion from generic position smoothing.

The filter itself is compact. We write it generically so the same function serves as a baseline later. Note how directly the code mirrors the equations: each line corresponds to one formula from the Filter Equations section, in the same order, which is a good habit when implementing anything mathematical, since it makes review and debugging a matter of matching lines to symbols.

Both diagnostics matter in practice. Converging variance confirms the filter reaches an information steady state, where each step sheds as much uncertainty as the process noise injects. The second panel shows the signed innovation divided by its predicted standard deviation, not the raw residual. Under a correctly specified linear-Gaussian model, these signed normalized innovations are standard normal and uncorrelated over time. In the run above, the filter deliberately starts from a broadened prior ($P_0 = 10I$ against a deterministic initial state), so the first few normalized innovations are underscaled until that initial transient passes; thereafter they follow the standard-normal reference, and unusually large magnitudes or persistent temporal structure provide a principled model-health alarm. Learned systems can also compare predictions with outcomes, but the point-estimate GRU below does not report a predictive variance, so its benchmark checks estimation error rather than uncertainty calibration.

Now we move to a nonlinear problem. A sensor at the origin measures only the bearing to a moving target, not its range. The observation function is $h(x) = \operatorname{atan2}(y, x)$, so one measurement constrains the target to a ray. Our simulated target follows an offset circle, which gives a more varied bearing trace than a circle centered on the sensor. It does not make absolute scale observable to the filters used here: scaling every position and velocity by the same positive factor leaves all bearings unchanged, and neither filter encodes the circle center or radius. Their Cartesian trajectories are therefore anchored by the initial belief alone; the constant-velocity transition is scale-equivariant, so it carries whatever scale the prior sets without contributing independent scale information. The truth uses circular kinematics, while both filters deliberately predict with a constant-velocity transition; their approximation error therefore includes dynamics mismatch as well as, for the EKF, local linearization.

Two implementation details deserve comment. First, the innovation is wrapped into the principal interval $[-\pi, \pi]$ using the sine/cosine trick, because angles live on a circle. Under the code's ordering, measurement minus prediction, a prediction of $179°$ and a measurement of $-179°$ give a raw difference of $-358°$, which wraps to $+2°$, not $-2°$ or $358°$. At an exact antipodal difference of $\pi$, floating-point signs can choose either equivalent endpoint. Ignoring wrapping is a classic EKF bug. The deeper lesson generalizes: whenever a state or measurement lives on a manifold rather than flat Euclidean space, angles on circles, orientations on spheres, poses on SE(3), the arithmetic of averaging and differencing must respect the geometry, and naive subtraction silently corrupts the filter. Second, we start the filter with a deliberately wrong initial guess and broad covariance. Early measurements can pull the estimate toward a bearing-consistent trajectory, while the prior continues to anchor absolute scale, which the scale-equivariant constant-velocity model merely carries forward. A broad covariance correctly represents that the initial Cartesian state is uncertain; it does not manufacture range information absent from the observations.

The particle filter handles the same problem without a single derivative. Each particle is a full four-dimensional state hypothesis; the bearing likelihood reweights them each step. Where the EKF maintains one Gaussian hypothesis and bends it locally, the particle filter maintains thousands of discrete hypotheses and lets the data decide which survive. This flexibility does not guarantee a better estimate. The bootstrap proposal below samples only from the constant-velocity transition model, while each of the $T_{nl}=60$ bearing observations has noise standard deviation $0.05$ radians and constrains angle but not absolute scale. The initialization $P_0=\operatorname{diag}(100,100,10,10)$ corresponds to ten-meter marginal position standard deviations and $\sqrt{10}\approx3.16$ m/s marginal velocity standard deviations, broad enough to include very different ranges and speeds. With that broad prior and circular truth that violates the proposal dynamics, resampling can eliminate range hypotheses that the bearing stream cannot restore on its own. Their survival depends on prior support and the assumed transition model.

The trajectory plot makes the representative run's RMSE concrete. Bearing residuals can remain small along a plausible ray even when Cartesian position error is large, so evaluate the state quantities a downstream controller needs rather than only the likelihood used for filtering. Observation-informed proposals, motion constraints, or conditional analytic updates can be more effective than increasing particle count alone when the proposal is structurally weak.

In this reproducible sequence, the EKF and the representative particle run settle on different Cartesian scales even though both receive the same bearings. That contrast is not evidence that the angles identify absolute range. It shows how the prior, transition approximation, and finite-sample proposal select among scale-equivalent bearing explanations.

Classical filters require known models and hand-derived update rules. A learned alternative trains a recurrent network to consume observations and controls and output a state estimate, amortizing an approximation to the composed filtering recursion into its weights. Here “amortize” means paying an upfront training cost so that each later update uses one recurrent forward step.

We return to the linear-Gaussian cart, generate training trajectories, and train a gated recurrent unit (GRU) to map $(o_{1:t}, u_{0:t-1})$ to an estimate of $x_t$. In the zero-based arrays below, row (t) pairs control (u[:,t]) for the transition from (x_t) to (x_{t+1}) with measurement (z[:,t]) of (x_{t+1}); the target is therefore (x[:,t+1]). This is the same convention with array indices shifted by one. The controls are random exogenous inputs, not actions selected by a policy. The network never sees the Kalman equations. Because position and velocity have different scales, we standardize inputs and residual targets using training-set statistics. Position uses a residual target, $x_{t+1,\mathrm{pos}}-z_t$, and evaluation reconstructs the position by adding $z_t$ back. The linear head itself is ordinary; the residual structure lies in this target parameterization. The learned update remains tied to its training distribution, which here contains only trajectories generated by the specified linear-Gaussian simulator and random exogenous controls. Every training and test episode starts from the deterministic state $x_0=0$. The held-out Kalman baseline uses that correct zero initial mean but retains a broad covariance $P_0=10I$, so this benchmark does not test generalization to unknown or nonzero initial states.

The seeded observations above are sanity checks for these executed trajectories, not guarantees that the same orderings or numerical comparisons hold after changing the data, seed, or training budget. The structural assertions check array alignment and implementation contracts, not benchmark superiority.

The belief-space MDP argument is exact only for an exact belief. The EKF, UKF, finite particle set, and learned recurrent state all approximate that belief, so they restore the Markov property only to the extent that their representation preserves decision-relevant information.

No filter is better than its models. The Bayes recursion is exact only under its assumptions, and a practical filter can fail when they are violated. Understating process noise $Q$ tends to make a filter overconfident; in extreme cases its covariance collapses around a wrong estimate. Overstating $Q$ tends toward overly diffuse, underconfident beliefs. Nonstationarity invalidates fixed parameters, while heavy-tailed or outlier-corrupted measurements invalidate a Gaussian likelihood and can pull an estimate far from the truth. Drifting innovations, persistent bias, and serial structure in normalized innovations are direct warnings that the prediction and observation model no longer match the data. Robust and adaptive filters address these cases in different ways. The same residual computations carry over to learned world models, but the classical Gaussian reference distribution is a guarantee only under the corresponding probabilistic assumptions. Across these systems, the most useful warning is a gap between self-reported confidence and measured accuracy.

Compounding rollout error deserves special emphasis because it anticipates model-based reinforcement learning. During prediction with no corrective measurements, disturbances and transition-model errors accumulate. A common conservative choice is to increase $Q$ to cover some unmodeled variation, accepting wider beliefs in exchange for less overconfidence. The transferable lesson is narrower than a literal equivalence: recursively applying a learned transition model also compounds error, so its uncertainty estimates must be checked against realized rollout errors.

This controlled example uses independent zero-mean disturbances, so both empirical RMS error and the correctly predicted standard deviation grow with the square root of horizon. The understated model predicts the same shape at the wrong scale and becomes increasingly overconfident in absolute terms. Biased model errors can grow differently, including approximately linearly, so the square-root curve is not a universal rollout law. The calibration test is general: compare predicted uncertainty with realized error under the deployment distribution.

Computational scaling draws the practical map. A dense Kalman covariance update is cubic in state dimension, so large systems require sparsity or low-rank structure. Particle methods can represent more general distributions, but useful coverage becomes expensive in high dimensions unless structure is exploited. Square-root filters propagate a covariance factor for better numerical stability; ensemble filters approximate moments with a finite sample set; and auxiliary-variable methods augment the representation to make sampling or conditioning easier. Along with the unscented method introduced earlier and Rao-Blackwellized methods beyond this chapter's scope, these approaches occupy different points between exact dense algebra and unconstrained sampling. The practical lesson is to exploit the conditional structure of the problem rather than choose a method by name alone.

The GRU training cell uses all 128 simulated training trajectories for exactly 1,200 random minibatch updates of size 32. It constructs no validation split and implements neither early stopping nor hyperparameter search, so this result should not be read as a tuned architecture comparison. These recursions appear in navigation, localization, tracking, attitude estimation, data assimilation, and robot state estimation. Modern learned world models often use an analogous template: maintain a latent state, predict it forward, and correct it with new evidence. A learned recurrent state is not automatically an exact Bayesian belief, but the classical filter remains a useful reference for checking prediction residuals, calibration, and collapse under repeated rollout.

• Under exact Bayesian updating and the stated POMDP assumptions, the Bayes filter alternates prediction and correction, and its belief is sufficient for future inference and decision-making.
• The Kalman filter solves the recursion exactly when the transition and observation models are linear, the initial belief is Gaussian, and the Gaussian process and observation noises are independent of one another, the initial state, and across time. The recursion reduces inference to mean and covariance updates governed by a gain that balances model trust against sensor trust.
• The EKF and UKF extend filtering to nonlinear systems by local linearization and sigma-point propagation respectively, remaining Gaussian approximations with corresponding blind spots.
• Particle filters represent beliefs with weighted samples and can preserve multimodal or non-Gaussian structure, but coverage becomes rapidly more expensive in high dimensions. Effective sample size diagnoses weight concentration and can trigger resampling.
• Learned belief updates can amortize an approximation to the composed recursion into a recurrent network. Their accuracy and calibration depend on the training distribution, architecture, and objective.
• Filters can fail smoothly and misleadingly under misspecification. Monitor normalized innovations and compare reported uncertainty with realized error.

With estimation in hand, the natural complementary question is where the transition and observation models themselves come from. System identification addresses it by fitting model parameters from observed data.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Bayesian filtering and belief states.