Transformers and Neural State-Space Architectures › Limitations and Impact

Attention and structured recurrence offer different ways to represent history in learned predictors. They did not invent long-horizon partially observable control: classical filtering and belief-space planning, as well as pre-transformer external-memory networks such as Neural Turing Machines, already addressed aspects of it. Modern transformer deployments demonstrate large context capacities (for example, Llama 3.1's reported 128K context), but capacity is not a guarantee of useful world-model memory. Structured SSMs connect learned sequence layers with linear-system analysis; hybrids combine mechanisms without removing their task and resource dependencies.

Several limitations matter when using these architectures.

• Recursive prediction. Our two models train with recorded observations and are evaluated with self-fed predictions. No architecture alone guarantees accurate forecasts under model mismatch, but exact or contractive dynamics need not exhibit growing model error. Attention can read retained real context during imagination while newer tokens are synthetic; whether that helps depends on learned representations and queries. Scheduled sampling and suitable rollout-aware objectives can reduce some conditioning mismatch. Their statistical interpretation must be assessed separately: not every multistep objective sacrifices a valid likelihood.
• Retained information. Attention cannot directly read absent KV entries, although summaries or other representations may preserve related information. An append-only cache grows with tokens; bounded windows need not grow without bound. Fixed-width finite-precision recurrence may merge histories requiring different predictions or actions. A sufficient task state avoids that problem, whereas more training data alone cannot reverse a distinction already absent from the representation.
• Hybrid configuration. Layer ratios, state widths, sharing, and cache policies add design choices. Complexity and stability analysis constrain some choices but do not identify the best task-specific configuration. Kernel and debugging support must be checked for the chosen implementation rather than assumed uniformly mature or immature.

Architecture choice must be evaluated against intended use. Memory probes and closed-loop environment tests complement one-step and recursive forecasts. Point MSE, as used here, scores a point prediction. Negative log-likelihood instead scores a predictive distribution: it is sensitive to assigned probabilities, though one aggregate value does not separately diagnose calibration, representation sufficiency, and decision utility. A planner can exploit poorly modeled regions outside the logged distribution. Likelihood directly supervises observed contexts and targets; extrapolation elsewhere depends on learned parameters and inductive biases, without a general off-support calibration guarantee. These issues motivate Part VI and Part XII.