Object-Centric and Relational Worlds › Object Discovery and Slot Representations

The first problem is discovery. Without segmentation supervision, the model must learn a useful scene decomposition. Reconstruction alone does not require human-aligned objects: parts, textures, and background regions can also minimize the loss. A widely studied family uses slot representations: the scene is represented by a fixed set of $K$ exchangeable latent vectors, with architectural competition intended to encourage different slots to specialize.

Competition is one mechanism for making slots specialize; it is not a definition of every slot model. Without competition, redundancy is possible, although other priors, routing rules, or objectives can also break symmetry. The next two subsections show how soft masks and attention normalization create competition for explanatory weight over pixels or tokens.

Several early unsupervised object-discovery methods use component-wise scene decomposition with exclusive or competitive masks: MONet (Burgess and colleagues, 2019), IODINE (Greff and colleagues, 2019), GENESIS (Engelcke and colleagues, 2020), and GENESIS-V2 (Engelcke and colleagues, 2021). Their inference mechanisms differ: MONet uses recurrent attention, IODINE iteratively refines variational posterior parameters, GENESIS uses an autoregressive scene prior, and GENESIS-V2 uses embedding clustering with stochastic stick breaking. In the following common mixture view, slot $k$ decodes to a mask logit $m_k(u)$ and appearance prediction $a_k(u)$:

where:

• $\hat{x}(u)$: the reconstructed color at pixel location $u$
• $a_k(u)$: slot $k$'s appearance prediction at pixel $u$
• $m_k(u)$: slot $k$'s mask logit at pixel $u$
• $\gamma_k(u)$: the normalized mask weight at pixel $u$, a softmax over slots that says what fraction of pixel $u$ slot $k$ explains
• $k'$: an index running over all $K$ slots

The softmax gives each pixel one unit of mixture weight divided fractionally among components. This creates competition and can encourage near-exclusive masks, but it does not force a one-hot or semantically correct object assignment. Decoding a mask from one latent supplies a spatial-coherence bias rather than enforcing connected contours. IODINE performs iterative amortized variational inference using likelihood, gradients, masks, and other refinement inputs; it is inspired by iterative refinement rather than a literal alternating mask/appearance algorithm. On simple multi-object datasets, these models can learn object-aligned components without segmentation labels, while split, merged, texture-based, and background components remain possible.

Slot Attention (Locatello and colleagues, 2020) is an influential object-centric encoder. It resembles transformer attention, but combines competition across slots with a second normalization that forms a weighted mean for each slot.

We have a set of input feature vectors $x_1, \dots, x_N$ (for an image, one per spatial location, typically from a CNN) and $K$ slots $s_1, \dots, s_K$. Compute dot-product affinities between slots and inputs:

where:

• $s_k$: the current vector of slot $k$
• $x_n$: the $n$-th input feature vector
• $W_q$, $W_k$: learned projection matrices applied to the slots (queries) and the inputs (keys)
• $d$: the projected dimension; when query and key coordinates have roughly fixed, weakly correlated variance, dividing by $\sqrt{d}$ stabilizes the typical dot-product scale
• $d_{kn}$: the raw affinity between slot $k$ and input $n$

The first normalization is the competitive one: normalize each input's affinities over slots. A second normalization over inputs will form each slot's weighted mean below.

where:

• $a_{kn}$: the weight that input $n$ assigns to slot $k$
• $k'$: an index running over all $K$ slots, so the weights on slots sum to one for each input

In standard attention, each query usually normalizes over keys. Here, each input distributes one unit of weight across slots. Comparable affinities split that weight; with two equal logits, the split is exactly one half each. This soft allocation creates competition and encourages a partition, but does not force a discrete or object-correct assignment.

The competition is easiest to see in a toy matrix of affinities.

Each column sums to one, so increasing one slot's share reduces the others' shares. The middle columns show the greatest ambiguity between the two slots.

Canonical Slot Attention next renormalizes each slot's competitive weights across inputs to form a weighted mean, then updates the slot. With a small $\epsilon>0$ for stability,

where:

• $a_{kn}$: the competitive weight that input $n$ assigns to slot $k$
• $w_{kn}$: slot $k$'s weights renormalized across inputs, which sum to one over $n$
• $W_v$: a learned value projection applied to the inputs
• $u_k$: the weighted-mean value update for slot $k$
• GRU: a gated recurrent unit that blends the new evidence $u_k$ into the slot's previous content
• MLP and LayerNorm: a small multilayer perceptron applied after layer normalization, added as a residual update

The GRU in the middle of this update is easy to overlook but does real work: because the update is applied iteratively, the slot needs a way to keep what it has already learned about its object while incorporating new evidence, rather than overwriting itself at every pass. The GRU's gating provides exactly that selective memory within the iteration loop.

The canonical setup samples initial slots from learned Gaussians and uses three refinement iterations, although the count is a tunable hyperparameter. The process resembles soft clustering: attention weights act like assignments and slots like iteratively updated representatives. After the configured iterations, many variants decode each slot independently with a spatial-broadcast decoder and combine masks with a softmax mixture. Independent decoding is a strong component-separation bias, not a requirement; SLATE instead uses a joint transformer decoder conditioned on all slots.

Two properties make slot attention attractive for world modeling:

• Permutation symmetry. When initialization and downstream processing preserve the symmetry, permuting initial slots permutes the outputs. No discovery-time index is privileged, which discourages fixed conventions such as "slot three is always leftmost" but does not prevent swaps or every index-based degeneracy.
• Variable input sets, fixed object capacity. Attention can process a variable number of input features, but a fixed $K$ is a direct ceiling on simultaneously represented objects. Slot width, iteration count, positional encodings, and encoder resolution are additional capacity and resolution choices.

For world modeling we need object structure in time, not just per frame. A per-frame encoder returns an unordered set, so correspondence across frames is unresolved. SAVi, or conditional Slot Attention (Kipf and colleagues, 2022), propagates a fixed set of prior slots through a predictor and corrects them with the current frame. This learned temporal conditioning replaces an explicit matching stage and encourages continuity, but it does not guarantee a stable track ID or solve births and deaths. SAVi used optical flow as a training target, treating coherent motion as a useful grouping cue rather than a definition of objecthood. SAVi++ (Elsayed and colleagues, 2022) replaces flow supervision with depth prediction and evaluates synthetic depth and sparse-LiDAR supervision.

SLATE (Singh, Deng, and Ahn, 2022) uses a dVAE to discretize image patches, maps the token embeddings into continuous slots with Slot Attention, and autoregressively reconstructs the token grid with a transformer conditioned jointly on all slots. The discrete vocabulary belongs to image patches, not to the slots, and SLATE itself is an image model rather than a temporal world model. SlotFormer (Wu and colleagues, 2023) supplies the separate temporal step by predicting sequences of continuous slots and evaluating them for video prediction and model-based control.

Reconstruction quality alone is an incomplete proxy for object-centric segmentation or downstream usefulness. A model can reconstruct well while splitting one object or merging touching objects, because the pixel mixture can be correct even when the decomposition is not the one the task needs. Evaluation can therefore add adjusted Rand index (ARI) against ground-truth masks, foreground ARI (FG-ARI), and downstream tests of dynamics prediction or control. Each metric answers a different question.