Digital and Software Environments › GUI and API Transition Models

A full-state learned model approximates the environment's transition kernel:

$$\hat T(s'\mid s,a)\approx T(s'\mid s,a).$$

Here $s'$ denotes the next state. In deployment, the agent generally cannot supply the true hidden $s$ directly.

Under the observation convention in our definition, the ideal observation distribution conditional on a known state is

$$p(o'\mid s,a)=\sum_{s'}\Omega(o'\mid s')T(s'\mid s,a).$$

The sum weights each possible next state's observation distribution by the probability of reaching that state. For continuous state spaces, the sum becomes an integral. A deployable prediction instead conditions on history and marginalizes the uncertain current state:

$$p(o'\mid h_t,a)=\sum_s b_t(s)\sum_{s'}\Omega(o'\mid s')T(s'\mid s,a).$$

Here $h_t$ is the observation-action history and $b_t(s)$ is the belief weight assigned to current state $s$. The outer sum averages over that uncertainty; it does not require access to the true hidden state. The learned observation model $\hat p(o'\mid h_t,a)$ approximates this marginal, possibly through an estimated latent representation. Conditioning only on $o_t$ is an additional approximation, appropriate only when it retains the relevant predictive information.

The following GUI and API categories organize interfaces, not an exhaustive taxonomy or a ranking of model families. Both can be combined with learned dynamics, explicit simulators, contracts, and verifiers.

A GUI world model predicts changes in interface observations. Here we distinguish pixel-level and structural inputs; hybrid representations can use both.

A pixel-level model predicts the next screenshot from observations or history, an action, and possibly an estimated latent state. Video-prediction architectures from Part V and Part IX offer relevant tools, but an architecture's ability to accept screenshots does not establish transfer to any application. Exact text can be important: "carol@example.com" and "carol@example.org" differ in three character positions. Downsampling or compression may make text harder to distinguish, but this chapter does not demonstrate a raster collision between those strings.

A structural model uses a DOM, accessibility tree, or serialized widgets. These channels can expose exact text and control identities without asking an image decoder to reproduce them. They may omit visual details such as canvas content or an unlabeled icon, and their coverage depends on the interface. Model size and comparative accuracy depend on the architecture, representation, training data, and evaluation; neither follows from "structural" alone.

Neither channel automatically reveals server-side persistence. The same current screenshot may appear with and without a saved row in the authoritative store. That screenshot alone cannot identify which latent state holds. Earlier responses or later reads may distinguish them, and a model can represent uncertainty rather than being forced to predict "saved". Additional scale cannot create information absent from every available input, but improved history use or a new observation tool can change what is identifiable.

An API observation model predicts structured responses or later observations from a request and the available history. A response can expose identifiers and validation outcomes more directly than a screenshot, but it need not reveal all server state. A successful POST can return different statuses depending on endpoint semantics: creation may return 201, acceptance for later processing 202, or a completed request with no response content 204. The protocol status and application contract must be interpreted together; RFC 9110 defines these success statuses.

A payload validator can reject a prediction that violates a schema. It cannot show that the server executed the predicted side effect. Some APIs supply structured error details; others return sparse codes, free text, or transport failures without a response. A useful model separates response format, documented semantics, hidden permissions, and the evidence of eventual persistence. This chapter makes no comparative claim about how quickly API and GUI agents learn.

Use only interfaces and observations the agent is authorized to access. An undocumented endpoint observed in a client's traffic may change without a stable contract, and observing it does not grant permission to invoke it. If no authorized API exposes the relevant evidence, GUI observations, local instrumentation, or a human confirmation may be necessary.

Contracts describe constraints on behavior. An OpenAPI document can describe operations and response formats; a JSON Schema can constrain payload structure. Types constrain representable values, migrations describe database changes, and tests assert behavior in covered cases. None automatically specifies the complete transition kernel, including session revocation, concurrent writes, queues, and external effects. The OpenAPI specification describes an interface, while JSON Schema validation defines structural validation keywords.

• Learned dynamics estimate outcomes from data and can be used where rules are incomplete, but generalization to a new field or redesigned UI must be measured.
• Contracts and executable checks provide explicit constraints for their stated scope, but can be incomplete, stale, or inconsistent with the implementation.

An executable specification that truly defines all relevant transitions can serve as a simulator $T_{\mathrm{spec}}$. A schema alone is not such a simulator. When available, use contracts to reject malformed predictions and tests to challenge predicted effects; use learned or hand-built dynamics for effects not specified by those constraints. These roles complement one another without establishing that one family generally outperforms another.

A hybrid design can use the DOM to locate a control, an authorized API to obtain an identified record, and a learned model to predict how navigation changes the screen. This is one possible design, not a claim about the architecture of a typical deployed agent. It connects to the hybrid models discussed in Part V.

Track provenance: which constraint is documented, which behavior was tested, which outcome was observed, and which prediction is an extrapolation. Predictive entropy can reflect both transition randomness and uncertainty about the model, depending on the formulation; a single entropy value does not reveal that provenance or guarantee calibration.