Observations, Sensors, and Multimodal Data › Sensor Calibration and Synchronization

Before any fusion or learning can happen, the spatial frames and measurement times of different sensors must be related through known or estimated transforms and clock offsets. An implementation may transform data into one common frame and clock, or retain native coordinates and apply those relationships inside the estimator. Calibration handles the spatial relationship; synchronization handles the temporal one. Errors in either propagate into downstream estimates and can be mistaken for model error.

Each sensor has its own coordinate frame: the camera's optical axis, the IMU's case, the tactile array's surface. The rigid-body transforms between these frames and the robot body are extrinsics. A sensor's intrinsics describe its internal measurement geometry; for a camera, they include focal length, principal point, and lens distortion. Camera intrinsics alone do not map the world into pixels: that projection combines intrinsics with the camera pose supplied by extrinsics. Associating raw camera and lidar measurements therefore requires the relevant intrinsics plus the transform between their frames. If upstream processing has already produced metric quantities in calibrated frames, downstream fusion may consume those quantities without applying every raw calibration parameter again.

For cameras, intrinsics come from standard calibration procedures using a known pattern viewed from many poses. The known geometry of the pattern provides correspondences between world points and image points, and the calibration solves for the parameters that best explain all the correspondences at once. Extrinsics between a camera and the robot's hand or base come from hand-eye calibration: move the robot to many poses, observe a fixed target from each, and solve for the unknown transform from the resulting loop-closure equations. The name comes from the classic robot-arm setting, where the "eye" is a camera in the hand, but the technique applies to any pair of one moving sensor and one moving body. The reason many poses are needed is that each pose contributes one loop-closure equation, and a single equation cannot disentangle the unknown camera-to-hand transform from the unknown target-to-world transform; only the accumulated constraints across many motions pin both down. IMU-to-camera extrinsics are estimated similarly, often jointly with time offset in unified calibration toolkits, because spatial and temporal calibration turn out to be coupled: an unknown time offset between a camera and an IMU produces errors that mimic a spatial misalignment, and vice versa.

Two practical warnings. First, calibration is not a one-time event: thermal expansion, impacts, and mounting creep can change extrinsics. Production systems should recalibrate or monitor residuals, and some perform that monitoring online. A robot calibrated in the morning and dropped at noon may no longer satisfy its calibration tolerance. Second, a wrong extrinsic can look like a dynamics error. A camera mounted one centimeter from its assumed pose can make a forward model predict systematically displaced objects. Before retraining the model, audit the transforms that produced its data.

Timing is the more insidious problem, because there is no visible artifact when it goes wrong. A spatial misalignment at least produces geometrically impossible overlays that a human can spot in a visualization; a temporal misalignment produces data that looks entirely plausible and is simply wrong. Three distinct time intervals get confused:

• Sampling time: the instant the physical measurement was taken (the midpoint of the camera's exposure, for instance).
• Timestamp: the time attached to the observation by software, which may be assigned at readout, at driver receipt, or at consumer arrival.
• Arrival time: when the downstream component receives the data.

Ideally the timestamp represents sampling time on a common clock, while arrival time and end-to-end latency are recorded separately. In practice, timestamps are assigned late, clocks on different devices can drift apart by milliseconds per minute, and queues add jitter. The gap between sampling time and timestamp can exceed the latency budget of a fast control loop and vary with bus contention or driver behavior. The timestamp is itself an estimate of when the measurement occurred, produced by an imperfect clock.

The mitigations form a hierarchy:

• Hardware triggering uses a physical sync line or common strobe to lock acquisition instants across devices. Clock synchronization, including Precision Time Protocol (PTP, standardized by IEEE 1588), aligns device clocks; devices that support scheduled acquisition can then sample against that shared timebase. These mechanisms address different parts of the timing problem and can be used together.
• Software synchronization estimates and corrects clock offsets in software, for example by cross-correlating signals that both sensors should observe, such as the IMU and camera both seeing the same acceleration spikes. The cross-correlation approach finds the lag that best aligns the two signals and treats that lag as the clock offset; it works well when the shared signal has sharp features and degrades when both streams are smooth.
• Online time-offset estimation treats the sensor offset as an unknown parameter and estimates it jointly with state, as some visual-inertial systems do. It can track offset drift continuously, at the cost of extra estimator parameters and computation.

Latency belongs to the whole pipeline. A camera's effective latency includes exposure, readout, compression, transfer, and decoding; pipelining can overlap some stages, but every stage on the critical path contributes to end-to-end delay. Control depends on the latency from photon to belief rather than the frame interval printed on the datasheet. On a platform moving at one meter per second, 80 milliseconds of unmodeled latency means the platform has traveled eight centimeters since the measurement time. Whether that is acceptable depends on the task geometry and control margin. We will handle delay explicitly in the worked example.