Geometry, Mapping, and 3D/4D State › Coordinate Frames and Camera Geometry

This section covers the vocabulary that everything else depends on: frames of reference, rigid transforms, and the pinhole camera model that connects 3D points to pixel observations. The material is classical, but it deserves careful treatment, because many practical failures in 3D perception and learned world modeling trace back to frame bookkeeping done carelessly: a rotation applied in the wrong direction, a translation expressed in the wrong frame, a depth confused with a range.

A frame is a coordinate system attached to something: the world (allocentric), the robot's body (egocentric), a camera, an object. Every geometric statement is implicitly relative to a frame, and much of practical 3D robotics is bookkeeping about which frame each quantity lives in. The phrase "the chair is at position $(1.2, 0.5, 0)$" is incomplete on its own; it only becomes a claim about the world once we say whose coordinates those are. In a world model, this incompleteness is dangerous, because learned networks can silently internalize whatever frame dominated their training data, and then fail unpredictably when the deployment frame convention differs.

Rigid transforms move points between frames. A rigid transform is a rotation followed by a translation; it preserves distances and angles. Physically, this is exactly the transformation relating any two frames that are bolted to rigid bodies, which is why it suffices for cameras mounted on robots, hands holding objects, and sensors on vehicles. We write the transform from world frame $W$ to camera frame $C$ as:

$$\mathbf{x}_C = R_{CW} \mathbf{x}_W + \mathbf{t}_{CW}$$

where $R_{CW} \in SO(3)$ is a $3 \times 3$ rotation matrix ($R^\top R = I$, $\det R = 1$) and $\mathbf{t}_{CW}$ is a translation vector. The two constraints defining $SO(3)$ have physical meaning: orthogonality says the camera's axes stay perpendicular, and unit determinant says the axes keep a right-handed orientation, which excludes reflections. The subscript convention $R_{CW}$ reads naturally as "the rotation taking points expressed in $W$ into points expressed in $C$." The set of all such transforms forms the group $SE(3)$, and composing them is just matrix-style chaining: walking from the camera frame to the body frame to the world frame composes two transforms into one. This composition property is what makes transform graphs workable in practice. A robot typically maintains a tree of frames, camera-to-gripper, gripper-to-wrist, wrist-to-base, base-to-world, and any point can be moved between any two frames by composing the transforms along the connecting path.

The awkward part of this notation is that rotations and translations do not combine into a single matrix multiplication on 3-vectors. Composition becomes messy: $(R_2, \mathbf{t}_2) \circ (R_1, \mathbf{t}_1) = (R_2 R_1, R_2 \mathbf{t}_1 + \mathbf{t}_2)$, and inverting means remembering that the translation must be rotated back before being negated. Homogeneous coordinates fix all of this. We append a 1 to every point, making it a 4-vector, and write the transform as one $4 \times 4$ matrix:

$$\tilde{\mathbf{x}}_C = T_{CW} \tilde{\mathbf{x}}_W, \qquad T_{CW} = \begin{bmatrix} R_{CW} & \mathbf{t}_{CW} \\ \mathbf{0}^\top & 1 \end{bmatrix}$$

Now composition is ordinary matrix multiplication, $T_{AC} = T_{AB} T_{BC}$, and inversion is (mostly) a structured transpose-like operation: for a rigid transform, $T^{-1} = \begin{bmatrix} R^\top & -R^\top \mathbf{t} \\ \mathbf{0}^\top & 1 \end{bmatrix}$, which needs only the already-available $R^\top$ rather than a general matrix inverse. When the transform is a pure frame-to-frame map like $T_{CW}$, the inverse is the reverse map $T_{WC}$; software can obtain it with the structured rigid-transform inverse or cache it as an implementation choice. Homogeneous coordinates also let us treat points at infinity uniformly, which matters because projection is about to send finite 3D points onto a 2D plane where direction, not distance, survives. A direction is represented by a homogeneous vector whose final coordinate is zero. Under projection, a family of parallel 3D lines meets at a vanishing point, while the directions contained in a plane form a horizon line. The projective framework handles these ideal elements with the same algebra as ordinary points rather than as special cases.

The pinhole model is the workhorse idealization of a camera. Light passes through a single point (the optical center). The physical sensor lies behind that center and forms an inverted image. The conventional computer-vision model used below instead intersects rays with a virtual image plane in front of the center; the image-coordinate convention absorbs the physical inversion and gives the positive-sign projection equations. Lens distortion is a common geometric departure that calibration can estimate, but real image formation also includes finite-aperture blur, aberrations, motion blur, rolling shutter, sampling, exposure, and other effects that may need explicit models for the task at hand. An ideal visible 3D point in camera coordinates $\mathbf{x}_C = (X, Y, Z)^\top$, with positive optical-axis depth $Z > 0$, projects to pixel coordinates:

$$u = f_x \frac{X}{Z} + c_x, \qquad v = f_y \frac{Y}{Z} + c_y$$

The equations have a direct physical reading. Dividing by $Z$ is the perspective effect: for fixed lateral coordinates $X$ and $Y$, increasing positive depth moves the projection toward the principal point. The focal lengths $f_x, f_y$ scale normalized ray coordinates $X/Z$ and $Y/Z$, the tangents of the off-axis ray angles, into pixels. Treating pixel offset as directly proportional to angle is a small-angle approximation. A longer focal length means a given lateral offset $X$ at depth $Z$ produces a larger pixel displacement. The principal point $(c_x, c_y)$ accounts for the fact that the optical axis does not usually pierce the sensor exactly at its center. In matrix form, stacking the whole pipeline from world to pixel:

$$\tilde{\mathbf{u}} \sim K \begin{bmatrix} R & \mathbf{t} \end{bmatrix} \tilde{\mathbf{x}}_W, \qquad K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$

This compact expression is worth internalizing because it separates cleanly: the $3 \times 4$ matrix $[R \mid \mathbf{t}]$ does the world-to-camera rigid motion, the calibration matrix $K$ does the camera-to-pixel mapping, and the result is a single linear operation in homogeneous coordinates. Homogeneous coordinates isolate this linear projective transform from the nonlinear perspective division that follows; real cameras add lens distortion and other nonidealities outside the displayed model.

The tilde and the $\sim$ carry real meaning: $\tilde{\mathbf{u}}$ is a homogeneous 3-vector, and $\sim$ means equality up to multiplication by any nonzero scale. Converting that homogeneous image vector to ordinary pixel coordinates then requires dividing by its third coordinate. That division is the nonlinear heart of perspective, and it is where information dies. Up to that division, the mapping from world point to image point is linear, which is why much of multi-view geometry can be developed with linear algebra, and why the division step is exactly where reconstruction difficulty concentrates.

Look at what the projection discards. The pixel $(u, v)$ depends only on the direction of $\mathbf{x}_C$, not its length. The camera-frame points $(X,Y,Z)=(1,0,2)$ and $(2,0,4)$ land on exactly the same pixel because both $X/Z$ and $Y/Z$ match. The perspective division by $Z$ has erased the overall scale of the camera-frame point. Every observation is a ray from the optical center, and the observation alone cannot tell you where along the ray the world is. Formally, the two pixel coordinates provide two scalar constraints on a three-dimensional point, leaving a one-parameter family of locations along the ray. This single fact drives most of the rest of the chapter: recovering depth requires either extra information (a second viewpoint, a depth sensor, a learned prior) or extra assumptions. Whenever a system appears to get depth from a single image for free, one of these three is hiding somewhere, and it is worth finding out which.

To see the collapse concretely, the next cell places several points along a single viewing ray at different depths, projects each one, and also compares axial depth against Euclidean range for off-axis points.

Also note that $Z$ is depth along the optical axis, not Euclidean distance to the point. Conflating the two is a common source of subtle bugs. For a point well off-axis, the Euclidean range can be substantially larger than the axial depth, and a stereo algorithm, a lidar, and a learned depth network may each report a slightly different one of these quantities by convention. When fusing sensors or comparing predictions against ground truth, always check which convention each source uses.

Given the ray ambiguity above, how do agents recover depth? There are three broad strategies, and they differ in what they assume and what they guarantee. It is worth being explicit about the source of depth information in each case, because that source determines the failure modes.

• Stereo and multi-view geometry: two cameras with a known baseline see the same point from two positions. The point's depth is the only value consistent with both observations. Geometrically, each pixel defines a ray. For a correct correspondence from compatible calibrated pinhole cameras, noise-free rays meet at the scene point in a nondegenerate configuration. For a rectified pair with parallel optical axes and a horizontal baseline $B$, horizontal disparity $d$ and focal length $f$ in pixels obey $Z = f B / d$. Arbitrary calibrated viewpoints instead require triangulation from their projection matrices. The rectified-pair inverse relationship has important operational consequences. A larger baseline improves first-order depth precision when focal length in pixels and disparity uncertainty are controlled. More pixels across the same field of view can also help when optical quality, calibration, correspondence quality, and disparity uncertainty in pixel units do not degrade. Resolution alone cannot recover information lost to blur or matching error. Distant points have tiny disparity and are hard to localize, which is why stereo depth maps are accurate nearby and noisy in the distance.

The inverse relationship is easiest to trust when plotted: the next cells evaluate the stereo equation over a depth range for two baselines.