Observations, Sensors, and Multimodal Data › A Tour of the Modalities
World Models HandbookAll chapters Observations, Sensors, and Multimodal Data › A Tour of the ModalitiesOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Different sensors occupy different points in the tradeoff space of bandwidth, latency, noise, coverage, and physical risk. No single sensor wins on all axes. Before we can fuse modalities, we need a concrete sense of what each one offers. Table modality-properties gives illustrative device-level ranges for five modalities used in robotics and embodied AI; actual values depend on the device, configuration, preprocessing pipeline, and definition of an observation update.
| Modality | Typical rate | Typical latency | Strength | Weakness |
|---|---|---|---|---|
| Vision (camera, depth) | 10–120 Hz | 10–100 ms | Rich semantics, geometry, wide coverage | Heavy compute, lighting-sensitive, occlusion |
| Audio waveform | 16–48 kHz carrier sampling | 5–50 ms | Detects audible events out of view | Low spatial precision, reverberation |
| Language / text | Event-driven | Variable | Abstract, task-level information, non-visual knowledge | Not a direct physical measurement |
| Touch (tactile arrays) | 100–1000 Hz | 1–20 ms | Contact physics at the interface | Tiny spatial coverage, wear and damage |
| Proprioception (encoders, IMU) | 100–4000 Hz | < 1–5 ms | Fast, locally available without scene visibility | Drift, bias, no external scene content |
Read the rate column against the latency column and a useful engineering tendency appears. Many stacks pair high-rate, low-dimensional inertial or joint signals with slower, higher-dimensional perceptual streams. A kilohertz IMU sample is a handful of numbers; a 30 Hz image may contain millions of pixels whose task meaning requires substantial processing to extract. Audio is a reminder not to turn this tendency into a law: its waveform carrier is sampled quickly even when the downstream event or feature updates are much slower. These differences are one practical reason layered estimation stacks use fast low-level loops and slower high-level perception.
The contrast can be sketched on a scatter plot, with each modality's characteristic raw sampling rate against a rough description of one sample. We encode richness as an illustrative ordinal score from 1 (a few scalars) to 5 (high-dimensional or symbolic content). The score is not a physical quantity, and the axes mix carrier sampling with event-driven updates, so the figure is a design sketch rather than a measured law:
import numpy as np
modalities = ["Proprioception", "Touch", "Audio", "Vision", "Language"]
rates_hz = np.array([1000.0, 500.0, 32000.0, 30.0, 1.0])
richness = np.array([1.0, 2.0, 1.2, 4.5, 5.0])
The illustrative points show a broad tendency, not a per-decade rule. Proprioceptive and tactile streams offer frequent compact updates; vision and language offer richer but slower observations; audio sits apart because raw waveform rate and semantic event rate are different quantities. A layered stack is one effective response to those mismatched rates and representations, but the figure does not establish a universal sensor law.
Vision
RGB and depth cameras often produce high-data-rate observations with rich spatial and semantic cues. A single RGB frame can contain evidence about object identity, layout, texture, and lighting; a calibrated depth camera or stereo pair adds metric geometry. This richness is why much of the world-model literature, which we will see in Part V, predicts future frames. Generalizing that prediction across varied scenes benefits from information about object persistence, motion, illumination, and geometry. The target encourages representations of predictive dynamics, but it does not prevent memorization or shortcut correlations by itself.
The costs are equally important. Conventional passive cameras depend on ambient light, suffer from motion blur when either the camera or the scene moves, and provide no direct measurement of surfaces they cannot see. Occlusion means the effective observation rate for any particular object can be far lower than the frame rate. A camera producing 30 frames per second is, for an object hidden behind a shelf, producing no new visual information about that object. Vision pipelines also tend to pass through a learned encoder before downstream models see them. That encoder is part of the effective observation function. Training it on a different distribution or fine-tuning it mid-project changes that function, so models trained before the change no longer consume the representation on which they were calibrated. This can be a subtle source of irreproducibility.
Audition
Microphones sample fast, consume little power, and can detect sound around obstacles. For a robot, audio often acts as an event cue: a dropped tool, a change in motor pitch, footsteps in the next room. Among the local physical modalities in this tour, it is especially useful for audible events outside the camera's field of view because sound diffracts and reflects. Its spatial precision is usually poorer than vision, and reflections complicate timing-based localization indoors. With a clear line of sight, the direct wave ordinarily arrives before reflections. When the direct path is blocked or severely attenuated, however, the earliest detectable wavefront may be diffracted or reflected, so a naive time-of-arrival method can infer the wrong direction. Audio can therefore provide an early cue that vision then examines in detail.
Language and Text
Language differs from direct physical sensing. It is an asynchronous symbolic observation or conditioning channel carrying task specifications, corrections, and attributed knowledge such as "the red cup is fragile." Appearance may suggest a material property, but language can provide attributes that vision alone cannot establish reliably. In a world model, language may shift priors over dynamics and goals rather than measure an instantaneous physical state. Told that the cup is fragile, the model receives a constraint on acceptable future trajectories and relevant physical parameters such as contact tolerance. A message can still be represented as a POMDP observation, but treating it as a synchronized measurement of the current scene would discard its loose time indexing and underspecification.
Touch
Tactile sensors directly measure pressure, shear, vibration, and temperature at the contact interface. In many gripper configurations, the robot body partially or completely occludes that interface from ordinary external vision, so touch supplies information the vision system may miss. Tactile arrays also wear, and their calibration can drift with temperature and use. High-rate tactile signals can capture the early stages of contact formation and slip. Slip may unfold over milliseconds; on platforms whose vision loop is slower than the contact dynamics, touch can provide the local feedback needed for reactive control. High-speed or event-based vision can also contribute when the platform supports it.
Proprioception and Interoception
Proprioception is the agent's sense of its own body: joint angles from encoders and body-motion signals from IMUs. Motor currents and internal load signals are often grouped as interoception. These measurements are fast and usually available, which is why high-bandwidth inner control loops rely heavily on them. Outer-loop estimation and control often add exteroceptive aids such as vision, range sensing, or GPS. Proprioceptive weaknesses include drift and bias: wheel odometry accumulates unbounded error, IMU orientation estimates wander without an external reference, and gear backlash introduces direction-dependent error that no constant offset can correct. An unaided odometry estimate does not merely get noisier over time; its absolute error can grow without bound, and that drift cannot be identified from the dead-reckoning stream alone. A landmark, vision measurement, or GPS fix can re-anchor it.
It is worth keeping a hard conceptual line between signals that measure the agent's own body (proprioception) and signals that measure the external scene (exteroception, such as vision and touch). They enter the state differently: proprioceptive signals are often close to components of the agent's state itself, while exteroceptive signals are projections of the environment's state through the agent's sensors. We will return to this distinction when we build self-models and body schemas in a later chapter of this part.
The distinction has practical consequences beyond bookkeeping. For direct joint encoders, the observation map can be close to the identity. Other body sensors are less direct: IMU readings depend on frame orientation, gravity, dynamics, bias, and calibration. Exteroceptive signals depend on the relative pose between sensor and environment, so a camera reading depends jointly on where an object is and where the sensor points. Some calibration problems use known agent kinematics, as hand-eye calibration does; others estimate camera or target motion directly from calibration observations. When an estimator does rely on agent kinematics, errors in the self-model can masquerade as errors in the world model.
One more observation ties this section together: proprioception is often fast but local to the body, while exteroception can be richer but slower or intermittent. Bayesian filtering is designed for that complementary structure. Its prediction step propagates belief through the dynamics, while updates incorporate measurements at their actual rates. Each modality retains different failure modes. The worked example is configured so a frequent noisy encoder and a slower, more accurate camera improve one another when their uncertainties and timestamps are modeled correctly.