Object-Centric and Relational Worlds › Identity, Permanence, and Tracking

Discovery gives you objects now. A world model needs objects over time: the mug you plan to grab next is the mug you saw two seconds ago, even though it passed behind the kettle. This section covers the three intertwined problems of temporal binding. They are worth separating because they fail in different ways and are solved by different machinery: permanence is about memory and prediction through gaps in observation, tracking is about assigning identities across frames, and the sufficiency of the tracked state determines whether any of it is useful for decisions.

Object permanence is continued existence while an object is unobserved; an expectation of a continuous hidden trajectory is an additional assumption. In controlled violation-of-expectation studies, infants are shown matched possible and impossible events, and longer looking at the impossible event is commonly interpreted as evidence of an expectation under the experiment's controls. IntPhys (Riochet and colleagues, 2018) and ADEPT (Smith and colleagues, 2019) adapt surprise-based physical probes to models. CLEVRER (Yi and colleagues, 2020) is different: it evaluates descriptive, explanatory, predictive, and counterfactual question answering over collision videos. Matched occlusion probes are designed to reduce surface shortcuts and test whether surprise depends on representing a hidden object; forgetting alone does not logically imply zero surprise under every classifier or statistical model.

Permanence requires memory and prediction without direct observations of the object. A transition prior can advance hidden state while indirect evidence from occluder boundaries, contacts, camera motion, or other objects still corrects it. Transition error can make a reappearing observation harder to associate, especially when appearance cues are weak, but it need not force failure. Flat and slotted latents can both preserve hidden-object beliefs. Slots make a per-object trajectory easier to inspect; they do not certify permanence. Training on documented synthetic occlusions or explicit object-permanence objectives can provide relevant gradients; the unsupported generic "slot dropout reconstructs masked slots" recipe is not assumed here.

Tracking is the problem of assigning object identities across frames. In tracking-by-detection, filtering and data association are two central subproblems, alongside detection, gating, and track management:

• Per-object filtering. A tracker maintains a state estimate or belief for each track and predicts it forward before correction. A Kalman tracker carries a Gaussian estimate; other trackers need not represent a full posterior.
• Data association. Given predicted and detected states, the Hungarian algorithm finds a minimum-cost one-to-one assignment for the supplied cost matrix. Dummy assignments permit unmatched tracks or detections; lifecycle rules then interpret them as misses, candidate births, or deaths.

SORT (Bewley and colleagues, 2016) combines a constant-velocity Kalman model with Hungarian association over detections. DeepSORT (Wojke and colleagues, 2017) adds an appearance metric to help association when geometry is ambiguous. Their explicit assignments are inspectable, although overlap, missed detections, and long disappearances can still cause swaps or failed re-identification.

Slot-based video models instead encourage continuity by recurrent conditioning. In SAVi, prior slots are predicted forward and corrected using the current frame, removing a separately exposed matching decision but not guaranteeing identity. A swap may be silent unless the system adds identity-consistency diagnostics. Explicit association is easier to inspect; recurrent conditioning is end-to-end. Either design should be validated under the task's occlusion, contact, and re-entry conditions rather than chosen by a binary rule.

It is worth connecting this to sufficient state. The necessary contents depend on the task: velocity when motion prediction needs it, grasp-relevant geometry for manipulation, and appearance when re-identification is required. A slot is only as useful as the sufficiency of what it encodes. A slot that binds an object cleanly but lacks pose may support video prediction while failing a grasp planner. One design is to keep the slot free-form while supervising or probing particular variables. In the code section, the synthetic world gives us ground-truth positions and velocities directly.