Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Vision-Language-Action and World-Action Models › Worked Example: The Arithmetic of Action Tokenization

Michael BrenndoerferJuly 24, 20267 min read

World Models HandbookAll chapters

Part 9 of 12Foundation and World-Action Models

Chapter 54Vision-Language-Action and World-Action Models › Worked Example: The Arithmetic of Action Tokenization

Reading section 6 of 10 · Worked Example: The Arithmetic of Action Tokenization · 7 minSection 6 of 10 · 7 min

Section 6 of 10 · 7 min read

Vision-Language-Action and World-Action Models › Worked Example: The Arithmetic of Action TokenizationOne section · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before writing any code, it is worth doing the arithmetic that drives real architecture choices. Take a 7-DoF arm with a 1-DoF gripper, controlled at 50 Hz, using 1-second action chunks.

Latency budget from the model. The illustrative throughput range of 50 to 150 tokens per second describes a sequential autoregressive decoder. A non-autoregressive head can emit an action chunk in one forward pass. FAST reduces the sequence length of action chunks, but when its tokens are generated by an autoregressive VLA, they still use that model's next-token decoder; fewer tokens, not a different decoding mechanism, produce the latency benefit.

Resolution. Normalize each dimension to $[-1,1]$ and use $B=256$ endpoint-grid reconstruction values as in the equation above. Adjacent values are $2/255 \approx 0.00784$ apart, so the worst-case error for an in-range action is $1/255 \approx 0.00392$ normalized units. A separate midpoint-bin encoder gives spacing $2/256$ and worst-case error $1/256$ . If one normalized unit represents one radian, the endpoint-grid bound is about 3.92 milliradians. Whether discretization is a bottleneck depends on the controller, action semantics, and task tolerance.

Token count with language tokens. One second at 50 Hz contains 50 action vectors. With eight scalar dimensions per vector and one token per scalar, that is $50\times8=400$ tokens, ignoring delimiters or auxiliary tokens. At an illustrative decoding throughput of 50 to 150 tokens per second, emitting the chunk takes about 2.7 to 8 seconds.

The comparison is easier to read than to recite, so let us put the required token rates and the decoding capacity on the same axes. The figure below plots required tokens per second of behavior for three representations, with the illustrative decoding band shaded to show which schemes sit inside the feasible region.

In[3]:

Code

## Deterministic arithmetic only; this preparation cell has no imports and no random state.
## Control rate, chunk length, and action dimensionality are taken from the worked
## example above. The decoding throughput is the illustrative 50 to 150 tokens per
## second used throughout this section, not a measured hardware number.
CONTROL_HZ = 50
ACTION_DIMS = 8
DECODE_TOKENS_PER_SECOND_LOW = 50
DECODE_TOKENS_PER_SECOND_HIGH = 150

tokens_per_scalar_second = (
    CONTROL_HZ * ACTION_DIMS
)  # 400 tokens per second of behavior
tokens_fast_per_second = 30  # illustrative FAST-compressed token rate

decode_low = DECODE_TOKENS_PER_SECOND_LOW
decode_high = DECODE_TOKENS_PER_SECOND_HIGH

token_scheme_labels = [
    "one token per scalar",
    "chunked, uncompressed",
    "frequency-domain (FAST)",
]
token_scheme_rates = [
    tokens_per_scalar_second,
    tokens_per_scalar_second,
    tokens_fast_per_second,
]

Out[4]:

Visualization

Bar chart of required token rates for three representations plus an illustrative decoding-throughput band. — Required token rate per second of behavior for three action representations in the worked example. One token per scalar and chunked-but-uncompressed representations both demand 400 tokens per second of behavior, because chunking changes how often the policy is queried rather than how many tokens it emits per second. Frequency-domain compression reduces the stream to about 30 tokens per second, well inside the shaded band that marks the illustrative 50 to 150 token per second decoding throughput. Only the compressed stream sits in the feasible region, while the uncompressed streams are several times above the decoding capacity.

Chunking alone. Replanning once per chunk rather than once per physical control step reduces policy-query frequency from 50 Hz to 1 Hz. It does not reduce the autoregressive decoding steps needed to emit 400 tokens. Under the illustrative throughput above, generation still takes about 2.7 to 8 seconds, longer than the one-second chunk. Chunking alone is insufficient in that serving setup.

Frequency-domain tokenization. FAST applies a discrete cosine transform along the time axis, quantizes coefficients, and compresses the result into tokens. Smooth trajectories often concentrate energy in low-frequency coefficients, reducing sequence length.

Token counts become intuition only once they are converted into time, so the second figure restates the same arithmetic as latency against the control period.

In[5]:

Code

## Pure-Python arithmetic, matching the worked example above. Bars span the low and
## high ends of the illustrative 50 to 150 token per second decoding throughput; the
## chunk period is one second because chunks are decoded once per second of behavior.
CONTROL_HZ = 50
ACTION_DIMS = 8
DECODE_LOW = 50
DECODE_HIGH = 150
chunk_period_seconds = 1.0

tokens_uncompressed = (
    CONTROL_HZ * ACTION_DIMS
)  # 400 tokens per second of behavior
tokens_fast = 30  # illustrative compressed rate for this worked example

latency_labels = ["one token per scalar", "frequency-domain (FAST)"]
latency_low = [tokens_uncompressed / DECODE_HIGH, tokens_fast / DECODE_HIGH]
latency_high = [tokens_uncompressed / DECODE_LOW, tokens_fast / DECODE_LOW]
latency_mid = [
    0.5 * (low + high) for low, high in zip(latency_low, latency_high)
]
latency_err = [
    0.5 * (high - low) for low, high in zip(latency_low, latency_high)
]

## The uncompressed and compressed streams differ by more than an order of magnitude
## in this latency view, which makes the gap hard to read on the same axes. Switch to a
## nonlinear vertical scale so both the slow and fast generations remain visible.
LATENCY_SCALE = "log"

Out[6]:

Visualization

Bar chart with error bars showing seconds to generate a one-second action chunk. — Time to generate a one-second action chunk at the illustrative 50 to 150 token per second decoding throughput. Bar heights are midpoints of the resulting latency intervals, and error bars span each interval's endpoints; they are not latencies at the midpoint decoding throughput. The dashed line marks the one-second control period. Under these assumptions the uncompressed stream is too slow, while the compressed stream finishes within the period.

Read against the dashed line, the uncompressed stream cannot produce a chunk as fast as the chunk is consumed, whereas the compressed stream finishes well before its one-second budget expires. If a particular 400-scalar chunk compressed to 30 tokens, its required average output rate would be 30 tokens per second of behavior; at 50 to 150 decoded tokens per second, generation would take about 0.2 to 0.6 seconds, excluding encoding, prefill, communication, and execution overhead. Actual compression and latency must be measured.

Continuous heads. Alternatively, train a flow-matching head that produces the whole chunk in about ten network evaluations on a small transformer, with no autoregression at all. The cost is a separate generative module and a training objective that no longer shares the language model's next-token machinery.

The $\pi_0$ flow-matching policy and its FAST-tokenized counterpart illustrate the two approaches. The arithmetic showing that chunking alone leaves autoregressive per-token decoding work unchanged explains why these architectures seek either parallel action generation or fewer action tokens.