Observations, Sensors, and Multimodal Data › Multimodal Fusion and Alignment

Given calibrated, synchronized, degradation-aware streams, the task of fusion is to combine them into a single belief or representation that is better than any individual stream. The word "better" here admits the same four-way decomposition we introduced earlier: fusion can improve predictive fidelity, improve representation quality, improve uncertainty quality, or improve decision usefulness, and these do not automatically come together. A fusion scheme that minimizes state-estimation error does not automatically produce a representation a planner can act on, and one that produces confident predictions does not automatically produce calibrated ones. Keeping the objective of fusion explicit prevents the common failure of optimizing a fusion module for a benchmark correlation while the downstream task silently degrades. Fusion strategies occupy a spectrum based on where the combination happens.

• Early fusion combines raw or lightly processed signals before substantial encoding, for example by joining pixels and joint readings near the input. It exposes low-level cross-modal interactions but requires explicit handling of heterogeneous scales, rates, and missingness. Modality-specific normalization or shallow frontends can still precede the join.
• Late fusion processes each modality independently, then combines the results, such as averaging predictors or multiplying calibrated Gaussian posteriors. It is modular and can tolerate a missing modality. It may discard interactions available only in raw or intermediate features, though a late fuser can still compare final predictions, residuals, uncertainty, and disagreement.
• Intermediate (hybrid) fusion gives each modality its own encoder, then combines the embeddings into a shared latent state with mechanisms such as attention or gating. This pattern is common in multimodal robotics because each encoder can match its input statistics while the fusion layer models cross-modal interactions.

For Gaussian linear models, fusion has an exact and instructive answer. Suppose $z_i=x+\epsilon_i$, where the two errors are independent and $\epsilon_i\sim\mathcal{N}(0,\sigma_i^2)$. Combining these two likelihoods without an additional informative prior (equivalently, using a flat prior), the fused estimate is the precision-weighted average:

$$\hat{x} = \frac{\sigma_2^2\, z_1 + \sigma_1^2\, z_2}{\sigma_1^2 + \sigma_2^2}, \qquad \sigma_{\text{fused}}^2 = \left(\sigma_1^{-2} + \sigma_2^{-2}\right)^{-1}$$

where:

• $z_1$, $z_2$: the measurements reported by the two independent sensors of the same scalar quantity
• $\sigma_1^2$, $\sigma_2^2$: the noise variances of the two measurements; a noisier sensor contributes less
• $\hat{x}$: the fused estimate, which weights each measurement by the other sensor's variance (its inverse precision)
• $\sigma_{\text{fused}}^2$: the variance of the fused estimate, the harmonic-style combination of the inverse precisions

The intuition is precision weighting: the weight on $z_1$ is $\sigma_2^2 / (\sigma_1^2 + \sigma_2^2)$, so a low-variance measurement dominates. If sensor 1 is infinitely precise ($\sigma_1^2 \to 0$), the fused estimate converges to $z_1$. If the variances are equal, the estimate is the plain average and its variance is half of either individual variance. For two finite positive variances with independent errors, the fused variance is strictly smaller than either individual variance. Every Kalman update you implemented in Part II uses the same precision logic in matrix form. The independence assumption matters: two cameras looking through the same dirty protective window share part of their error, so treating them as independent double-counts information. Correlated errors often enter through shared vibration, clock error, power, or overlapping encoder training data.

The independence assumption can be made exact. For two sensors with equal variance $\sigma^2$ and error correlation $\rho$, the fused variance is $\sigma_{\text{fused}}^2 = \sigma^2 (1 + \rho) / 2$: the fusion advantage shrinks linearly to zero as $\rho$ goes to one:

At zero correlation the fused variance is half the single-sensor value. At perfect correlation the two curves meet: the second measurement adds no independent information. When a fusion system underperforms, measuring cross-modal error correlation is a useful diagnostic.

Alignment can involve three distinct registers: temporal, spatial, and semantic. A fusion design needs the subset relevant to its modalities and representation. The scalar Gaussian example needs compatible quantities and timing, while image-language fusion also needs a shared semantic representation.

• Temporal alignment: ensuring paired observations refer to the intended instants, using the synchronization machinery above. A fixed offset can be absorbed as apparent dynamics or bias when the model is not told about it. More similarly misaligned data reinforces that target convention rather than repairing the timestamps.
• Spatial alignment: expressing measurements in a common frame via calibration. Fusing a camera detection with a lidar return requires knowing where each sensor is, or the "same object" is never the same object. Spatial misalignment tends to scale with distance: a small angular calibration error produces a small offset for nearby objects and a large offset for far ones, which is one reason perception errors that look random in a table-top evaluation become systematic in large environments.
• Semantic alignment: mapping heterogeneous representations so their relationship has a learned meaning. Contrastively trained encoders, which place an image and its caption near each other in a shared embedding space, are the flagship example. In shared-embedding world models, this alignment lets a language instruction constrain a latent plan because proximity represents agreement in meaning. Other fusion interfaces, including cross-attention, gating, and learned adapters, can instead learn a cross-modal mapping without requiring raw embedding proximity.

A final framing will serve us for the rest of the book: fusion can be done as inference or as learning. The filtering view combines likelihoods using explicit noise models. A learned fuser estimates combination rules from data and can absorb biases or misalignments present in that data. The trade is partly about where assumptions live: the filter puts them in inspectable parameters such as $R$ and calibration transforms, while a learned module encodes them in weights. A common practical pattern uses classical filters for high-rate low-level state estimation and learned fusion for semantic content. A trained attention layer can learn reliability-sensitive weighting, but its coefficients are not automatically inverse variances or calibrated trust scores. The closed-form Gaussian case is a reference point, not an identity that generic attention must satisfy.