The reviewed record of science sign in
Pith

arxiv: 2606.28127 · v1 · pith:KICZQ2UC · submitted 2026-06-26 · cs.CL · cs.AI· cs.LG

From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

Paul Dubois This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 03:59 UTCgrok-4.3pith:KICZQ2UCrecord.jsonopen to challenge →

classification cs.CL cs.AIcs.LG
keywords large language modelsworld modelsnext-token predictionJEPAstate spaceautoregressive modelslatent predictioncontinuous spectrum
0
0 comments X

The pith

LLMs are a degenerate special case of world models where states are token sequences and the only action is appending one token.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that framing LLMs and world models as opposites is a false dichotomy. LLMs fit inside the world-model definition once the state space is restricted to sequences of tokens and the sole permitted action becomes appending the next token. This makes world models a strict generalization rather than a rival approach. A continuous spectrum then connects next-token prediction to latent-space methods such as JEPA, with several intermediate prediction schemes already in use. Moving along the spectrum relaxes token-level constraints but removes the two properties that currently allow training at internet scale.

Core claim

LLMs are a degenerate special case of world models: the state space is the set of all token sequences, the only action is appending one token, and world models are therefore a strict generalisation of LLMs, not a replacement. There is a natural continuous spectrum from NTP to JEPA, with multi-token prediction, future-summary prediction, and next-latent prediction as intermediate stations already populated by current research. Moving along this spectrum relaxes the LLM constraints one by one and progressively surrenders the two practical advantages that make LLMs trainable at scale: internet-scale self-supervised data, and a transformer architecture co-designed for discrete token prediction.

What carries the argument

Redefinition of a world-model state space as exactly the set of all token sequences, with token-append as the sole action.

If this is right

  • Relaxing constraints along the spectrum replaces single-token prediction with multi-token, summary, or latent targets.
  • The shift removes access to internet-scale self-supervised text data.
  • The shift removes the transformer architecture that was co-designed for discrete tokens.
  • Two open questions arise: sourcing instrumented action-labelled data at scale and determining whether transformers suffice for continuous-state prediction or a new primitive is required.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The spectrum view suggests that hybrid models occupying intermediate stations could retain some scaling advantages while gaining simulation capacity.
  • Research could test whether the loss of self-supervised text data can be offset by synthetic environment rollouts generated from existing LLMs.
  • The architecture question implies that any new primitive for continuous states would need to support the same level of parallelism that made transformers trainable on GPUs.

Load-bearing premise

Treating the set of token sequences as a valid state space for world models produces a meaningful and non-vacuous generalization rather than an empty relabeling.

What would settle it

An explicit construction showing that the token-sequence state space yields inconsistent transition predictions or cannot be embedded inside any standard world-model formalism.

Figures

Figures reproduced from arXiv: 2606.28127 by Paul Dubois.

Figure 1
Figure 1. Figure 1: A world model iteratively applies the transition function to simulate multi-step [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: illustrates this mapping concretely. State st = token sequence "The cat sat on the" Action at = next token "mat" State st+1 "The cat sat on the mat" LLM policy π(·|st) deterministic append [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Containment hierarchy as nested sets. LLMs (innermost box) are the most constrained [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: OthelloGPT is a standard transformer trained only on move tokens. The token interface [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The spectrum from LLMs to JEPA. Each node shows the prediction objective, training [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Architectures in the (state type × planning horizon) design space. The dashed diagonal traces the spectrum of Section 3. CoT sits directly above Standard LLMs; it increases planning horizon without changing the state type. The spectrum reveals a densely populated intermediate region that LeCun’s binary framing overlooks. 4 Discussion and Conclusion The preceding sections have established that LLMs are worl… view at source ↗
read the original abstract

The AI community has framed the relationship between large language models (LLMs) and world models as a dichotomy: LLMs predict tokens; world models simulate reality. Yann LeCun argues in 2022 that reaching general intelligence requires abandoning autoregressive token prediction in favour of latent-space architectures. This framing is unnecessarily binary. Two claims will be defended. First, LLMs are a degenerate special case of world models: the state space is the set of all token sequences, the only action is appending one token, and world models are therefore a strict generalisation of LLMs, not a replacement. Second, there is a natural continuous spectrum from NTP to JEPA, with multi-token prediction, future-summary prediction, and next-latent prediction as intermediate stations already populated by current research. Moving along this spectrum relaxes the LLM constraints one by one. It also progressively surrenders the two practical advantages that make LLMs trainable at scale: internet-scale self-supervised data, and a transformer architecture co-designed for discrete token prediction. Both are examined as open research questions: the data question (the cliff from self-supervised text to instrumented action-labelled environments) and the architecture question (whether the transformer generalises to continuous-state prediction, or whether a new primitive is needed).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper claims that LLMs are a degenerate special case of world models, with the state space defined as the set of all token sequences and the only action being appending one token, making world models a strict generalization of LLMs rather than a replacement. It further claims there is a continuous spectrum from next-token prediction (NTP) to JEPA, with intermediate stations such as multi-token prediction, future-summary prediction, and next-latent prediction already populated by existing research. Moving along the spectrum relaxes LLM constraints but surrenders the advantages of internet-scale self-supervised data and a co-designed transformer architecture, which are examined as open research questions.

Significance. If the reframing holds, it unifies LLMs and world models as points on a spectrum, potentially guiding incremental architectural research rather than requiring an abrupt shift. A strength is the explicit acknowledgment that the spectrum involves surrendering practical scaling advantages, with the data question (cliff from self-supervised text to instrumented environments) and architecture question (transformer generalization to continuous states) framed as concrete open problems. No new derivations, proofs, or empirical tests are supplied, so the significance rests on whether the definitional perspective provides actionable guidance beyond existing work.

major comments (1)
  1. Abstract, first claim: The assertion that LLMs are a 'degenerate special case' of world models is established by stipulating that the state space consists exactly of token sequences and the sole action is token append; this renders the generalization true by construction, but the manuscript provides no independent grounding, external benchmark, or falsifiable prediction against which the claim's utility for guiding AI architecture research could be evaluated.
minor comments (2)
  1. Acronyms NTP and JEPA should be expanded on first use to improve accessibility.
  2. The reference to Yann LeCun's 2022 argument would benefit from a specific citation or paper title.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and recommendation of minor revision. The manuscript is a perspective paper offering a definitional reframing of LLMs within world models and identifying a spectrum of architectures along with associated open questions. We respond to the major comment below.

read point-by-point responses

  1. Referee: Abstract, first claim: The assertion that LLMs are a 'degenerate special case' of world models is established by stipulating that the state space consists exactly of token sequences and the sole action is token append; this renders the generalization true by construction, but the manuscript provides no independent grounding, external benchmark, or falsifiable prediction against which the claim's utility for guiding AI architecture research could be evaluated.

    Authors: We agree that the claim follows directly from applying the standard MDP formulation of world models (state space, actions, transitions) to autoregressive token prediction. This is intentional, as the paper's purpose is to demonstrate that LLMs constitute a limiting case rather than a competing paradigm. The grounding lies in consistency with the world-model definitions used throughout model-based RL and robotics. As a perspective piece with no new empirical results, we do not supply benchmarks or falsifiable predictions; the claimed utility instead rests on the second claim, which organizes existing intermediate architectures (multi-token prediction, latent prediction) and explicitly frames the data-scaling and architecture-generalization questions as open problems that must be solved to advance along the spectrum. revision: no

Circularity Check

1 steps flagged

Central claim reduces to definitional stipulation of state space

specific steps
  1. self definitional [Abstract, first claim]
    "First, LLMs are a degenerate special case of world models: the state space is the set of all token sequences, the only action is appending one token, and world models are therefore a strict generalisation of LLMs, not a replacement."

    The generalization is obtained by defining the world-model state space to be precisely the token-sequence set; once that definition is adopted, LLMs satisfy the definition of a special case by construction. No independent properties of world models or external data are invoked to establish the relationship.

full rationale

The paper's lead claim is advanced by stipulating that world-model state spaces are exactly the set of token sequences (with token-append as sole action). This makes LLMs a 'degenerate special case' true by construction. The manuscript presents the move as a reframing to be defended rather than a theorem derived from independent properties of world models or external benchmarks. No equations, falsifiable predictions, or non-definitional grounding appear in the supplied text. The second claim (continuous spectrum) inherits the same framing but does not add independent derivation. This matches self-definitional circularity at the load-bearing step; the rest of the paper is descriptive.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The argument rests entirely on definitional choices about world models and the spectrum; no free parameters, new entities, or external axioms beyond the initial framing are introduced.

axioms (1)
  • domain assumption A world model is any system that maintains a state space, takes actions, and predicts future states.
    This definition is invoked to position LLMs inside the category.

pith-pipeline@v0.9.1-grok · 5754 in / 1225 out tokens · 41932 ms · 2026-06-29T03:59:16.837818+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages

  1. [1]

    2018 , eprint =

    World Models , author =. 2018 , eprint =

    2018

  2. [2]

    2022 , month =

    A Path Towards Autonomous Machine Intelligence , author =. 2022 , month =

    2022

  3. [3]

    2024 , eprint =

    Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task , author =. 2024 , eprint =

    2024

  4. [4]

    2023 , eprint =

    Emergent Linear Representations in World Models of Self-Supervised Sequence Models , author =. 2023 , eprint =

    2023

  5. [5]

    2024 , eprint =

    Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models , author =. 2024 , eprint =

    2024

  6. [6]

    2024 , eprint =

    Language Models Represent Space and Time , author =. 2024 , eprint =

    2024

  7. [7]

    2024 , eprint =

    Better & Faster Large Language Models via Multi-Token Prediction , author =. 2024 , eprint =

    2024

  8. [8]

    2026 , eprint =

    Next-Latent Prediction Transformers Learn Compact World Models , author =. 2026 , eprint =

    2026

  9. [9]

    Y., Pezeshki, M., Mitliagkas, I., Lopez-Paz, D., and Ahuja, K

    Divyat Mahajan and Sachin Goyal and Badr Youbi Idrissi and Mohammad Pezeshki and Ioannis Mitliagkas and David Lopez-Paz and Kartik Ahuja , year =. Beyond Multi-Token Prediction: Pretraining. 2510.14751 , archivePrefix =

    work page arXiv

  10. [10]

    Emergent Response Planning in

    Zhichen Dong and Zhanhui Zhou and Zhixuan Liu and Chao Yang and Chaochao Lu , year =. Emergent Response Planning in. 2502.06258 , archivePrefix =

    work page arXiv

  11. [11]

    From Next Token Prediction to (

    Carlos N. From Next Token Prediction to (. 2026 , eprint =

    2026

  12. [12]

    Understanding World or Predicting Future?

    Jingtao Ding and Yunke Zhang and Yu Shang and Jie Feng and Yuheng Zhang and Zefang Zong and Yuan Yuan and Hongyuan Su and Nian Li and Jinghua Piao and Yucheng Deng and Nicholas Sukiennik and Chen Gao and Fengli Xu and Yong Li , year =. Understanding World or Predicting Future?. 2411.14499 , archivePrefix =

    work page arXiv

  13. [13]

    Beyond World Models: Rethinking Understanding in

    Tarun Gupta and Danish Pruthi , year =. Beyond World Models: Rethinking Understanding in. 2511.12239 , archivePrefix =

    work page arXiv

  14. [14]

    2026 , eprint =

    From Word to World: Can Large Language Models be Implicit Text-Based World Models? , author =. 2026 , eprint =

    2026

  15. [15]

    2026 , eprint =

    Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement , author =. 2026 , eprint =

    2026

  16. [16]

    2021 , eprint =

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author =. 2021 , eprint =

    2021

  17. [17]

    Adrien Bardes and others , year =