Interaction Data and Passive Observation › Agent-Collected Trajectories

Action-conditioned control models are commonly trained or grounded on trajectories produced by an agent, operator, or simulator. We represent an interactive environment as a Markov decision process $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s' \mid s, a)$ is the transition kernel, $R(s, a)$ is the reward function, and $\gamma$ is the discount factor, as formalized in Part II: Inference, Decisions, and Control. An agent trajectory is a sequence

$$\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots, s_T),$$

where:

• $\tau$: a trajectory, i.e., a sequence of states, actions, and rewards from time $0$ to terminal time $T$
• $s_t$: the state observed at time $t$
• $a_t$: the action selected at time $t$
• $r_t$: the reward received after taking action $a_t$ in state $s_t$
• $T$: the terminal time index of the trajectory

sampled by an iterative loop: observe state $s_t$, choose action $a_t \sim \beta(\cdot \mid s_t)$, receive reward $r_t$ and next state $s_{t+1} \sim P(\cdot \mid s_t, a_t)$. A dataset $\mathcal{D}$ of trajectories is a collection of such sequences. The loop is the same primitive that appears throughout the book, from filtering and control in Part II to planning in Part VII. What changes between chapters is not the loop itself but the assumptions we make about the data it produces and the questions we ask of the resulting model.

Three distinct stochastic processes produce a trajectory dataset, and conflating them is a common source of confusion:

• The environment transition kernel $P(s' \mid s, a)$: the physical or logical dynamics we want the world model to learn.
• The behavior policy $\beta(a \mid s)$: the rule used to choose actions during data collection. This may be a heuristic, a random policy, a human operator, or a previous learned agent.
• The initial-state distribution $\rho(s_0)$: where episodes begin.

A world model trained on $\mathcal{D}$ is learning to approximate $P$ under the marginal state-action distribution $d^{\beta}(s, a)$ induced by $\beta$ and $\rho$. Outside that support, internally consistent predictions can still be wrong. The environment kernel describes the dynamics; the behavior policy and initial-state distribution describe how we chose to look at those dynamics. All three matter when we assess what the dataset can identify.

Interaction data records which command was applied before each transition $(s_t, a_t, s_{t+1})$. Under an important condition, this can supply interventional evidence: the logged state or history must include the information used to choose the action, and the action-selection randomization must be independent of unobserved transition disturbances after conditioning on that information. The behavior policy is still state-dependent, since $a_t \sim \beta(\cdot\mid s_t)$; "controlled" does not mean "independent of state." If an operator uses an unlogged sensor or private intent, interaction logs can be confounded in the same way as observational data. With the condition satisfied, varying $a_t$ lets us estimate how the agent's available controls change the next-state distribution rather than relying only on observed correlations.

Three concrete consequences:

• Counterfactual reasoning. Given an action-grounded model of $P(s' \mid s,a)$ learned under the stated no-unobserved-confounding condition, the agent can compare candidate commands. Observational actions can support the same conditional effect only when an identification assumption such as conditional ignorability is justified.
• Targeted exploration. The agent controls the data it will collect. As discussed in Part VII, Ch 5: Active Perception, Dual Control, and Exploration, it can deliberately choose trajectories that reduce model uncertainty in regions the planner will visit.
• Reward and value signal. A collection system can record or construct rewards alongside interaction trajectories, enabling control-oriented objectives. The action log alone does not guarantee that reward labels are present.

Interaction can be expensive: real robots wear out, human operators require time, and simulators require engineering. The behavior policy shapes the resulting distribution. An uncertainty-seeking explorer may emphasize novel regions, while a deployed policy may spend more time in routine states. That mismatch is not inevitable, but it must be measured rather than inferred from the label "exploration data."

Offline model-based RL uses a fixed trajectory dataset, while off-policy online methods may use a growing replay buffer. Both store transitions of the form

$$\mathcal{D} = \{ (s_t^{(i)}, a_t^{(i)}, r_t^{(i)}, s_{t+1}^{(i)}, \mathrm{done}_t^{(i)}) \}_{i=1}^{N},$$

where:

• $\mathcal{D}$: the trajectory dataset containing $N$ transitions
• $i$: the transition index
• $t$: the time index associated with transition $i$
• $N$: the number of transitions in the dataset
• $s_t^{(i)}$: the state at time $t$ for transition $i$
• $a_t^{(i)}$: the action applied in that state
• $r_t^{(i)}$: the reward received
• $s_{t+1}^{(i)}$: the resulting next state
• $\mathrm{done}_t^{(i)} \in \{0, 1\}$: a terminal indicator equal to 1 when the transition ends an episode and 0 otherwise

For a one-step supervised objective, implementations often flatten trajectories into transition records and shuffle them into minibatches. This changes how the optimizer samples the replay buffer, but it does not make transitions independent: adjacent records were produced by the same Markov chain. Stationarity keeps the marginal distribution fixed, while independence is a separate assumption. Temporal mixing and episode structure therefore affect the effective sample size even when the loss is written as an average over individual transitions.

These assumptions are worth stating explicitly because they are frequently implicit in the training code.

• Markov. $s_{t+1}$ depends on the full history only through $(s_t, a_t)$. If the observed state is only a proxy for the true state, the correct model is over a belief state, as discussed in Part II, Ch 2: Bayesian Filtering and Belief States.
• Stationary. The transition kernel does not change over time or episodes.
• Deployment coverage. The behavior-induced occupancy places enough probability near the state-action pairs the planner will query. Ergodicity of the behavior-induced Markov chain alone does not guarantee this target-relative coverage.

These are separate checks, and each can fail on its own. A process can be stationary but poorly covered, or well covered for a target while its observation is non-Markov. Naming the checks separately makes deployment failures easier to diagnose.

Interaction datasets come in two practically distinct forms, and they impose different training protocols:

• On-policy data is generated by the policy being evaluated or updated. Many policy-gradient and actor-critic procedures collect such batches. The data can support multiple optimization epochs, but after the policy changes it is no longer exactly on-policy; excessive reuse can increase the mismatch between the sampling and updated policies.
• Off-policy data is generated by a different behavior policy, often an older version of the same agent, a hand-designed explorer, or a heterogeneous pool of policies. It can be retained and reused, although stale or low-quality transitions may still be filtered. Its behavior mixture determines a coverage distribution that must be measured rather than assumed.

The distinction between the two flavors concerns whether the data-generating policy matches the policy currently being evaluated. On-policy collection reduces that immediate mismatch but requires new collection as the policy changes. Off-policy collection permits reuse, but training and evaluation must account for where the behavior distribution places mass.

The Data Curation and Synthetic Data chapter (Part VI, Ch 4) will discuss how to filter, weight, and curate these datasets at scale. For now, the key point is that "I have interaction data" is not a specificity claim: what actions, in what states, under what behavior?