Linear Spatial World Models Emerge in Large Language Models

Christian Bolivar Moya; Guang Lin; Matthieu Tehenan; Tenghai Long

REVIEW 4 major objections 6 minor 56 references

Linear Spatial World Models Emerge in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs keep a vector map of space inside their activations.

desk verdict The composition claim is not supported by the evidence as presented because the PCA subspace is fit to the same atomic directions being summed; the paper deserves a rigorous referee but the headline result needs a held-out spatial test. read the letter →

arxiv 2506.02996 v1 pith:OZBJXRSP submitted 2025-06-03 cs.AI

Matthieu Tehenan , Christian Bolivar Moya , Tenghai Long , Guang Lin This is my paper · ORCID

classification cs.AI

keywords worldmodelsspatialreasoninglinearrepresentationprobingactivationsteeringinterpretabilitycompositionalityresidualstream

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that a large language model builds a linear spatial world model as a by-product of next-token prediction: inside the residual stream there is a low-dimensional subspace, approximately a copy of $\mathbb{R}^3$, in which "above", "left", and "in front" act as orthogonal basis directions, their opposites are negative directions, and phrases such as "above and to the left" are encoded as vector sums. If this is right, spatial language is not surface mimicry: the model has an internally consistent geometry that can be read and manipulated. The authors test this on a synthetic dataset of object-position sentences, using linear probes, PCA projections, and activation steering, and report that linear probes match nonlinear ones, that composed relations align with vector sums in the PCA subspace, and that steering these directions changes generated answers in 74.3% of trials. A sympathetic reading is that the paper supplies the strongest current evidence for a compositional spatial code that is not merely present but causally used.

What carries the argument

The central object is the PCA subspace recovered from six atomic probe directions. Linear probes are trained on residual-stream activations to classify spatial relations; each probe contributes a direction vector $\mathbf{w}_i$. Principal component analysis projects these six directions onto a low-dimensional plane or 3D volume, and that projected subspace is treated as the model's internal $\mathbb{R}^3$. It does the work of turning the claim "linear spatial world model" into testable geometry: inverse pairs are scored by cosine, independent pairs by orthogonality, composed relations by angle to the vector sum, and steering vectors are reconstructed by lifting the PCA directions back into the residual stream.

What would settle it

Fit the subspace using only a subset of atomic directions, say above, below, left, and right, and then measure whether the composed vectors for "above and behind" or "left and in front" align with vector sums in that fixed subspace; if their cosine falls back to roughly the original-space value of 0.39 rather than the reported 0.99, the composition claim does not survive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLMs encode linear spatial world models. The authors define a spatial world model as a tuple $\mathcal{W}=\langle \mathbb{R}^3, O, S\rangle$ and propose that in the model's contextual embeddings there exists a basis $\{\vec{r}_{\text{left}}, \vec{r}_{\text{above}}, \vec{r}_{\text{in front}}\}$ satisfying three geometric properties: inverse relations are antipodal, independent atomic relations are orthogonal, and composed relations equal vector sums of their parts. The evidence is a probing and intervention pipeline on a 3-billion-parameter instruction-tuned transformer. Linear probes recover all six spatial relations essentially perfectly, PCA projections of the six probe directions show the predicted antipodal and orthogonal configuration, object embeddings cluster at consistent locations in the same subspace, and adding the projected directions back into the residual stream steers outputs toward the intended relation, with near-perfect success for "above", "below", and "left". The authors also report that composition holds in the PCA-reduced space with a mean cosine of 0.9931 and a mean angular deviation of 6.02 degrees in 2D, while the same comparison in the original activation space is far weaker, with a mean cosine of 0.3952.

Load-bearing premise

The load-bearing assumption is that the PCA subspace fitted to the six atomic probe directions faithfully reveals the model's internal spatial geometry; if the projection merely rearranges those six chosen vectors, the claimed Euclidean basis and vector-sum composition could be artifacts of the fitting procedure.

Editorial extensions

If this is right

Spatial relations are linearly decodable from the residual stream at every tested layer (8, 16, 24), so a downstream system can read object positions with a simple affine map rather than a deep decoder.
Because composed expressions are vector sums in the PCA subspace, unseen combinations such as "behind and to the right" should be predictable by arithmetic on atomic directions.
Since steering the identified directions changes the model's next-token behavior, the subspace gives a practical handle for controlling spatial outputs in instruction-tuned models.
Objects occupy consistent positions in the same subspace, so the geometry supports tracking multiple object configurations in a shared frame, not just pairwise relation labels.
The atomic antipodal structure also appears in two smaller model families, suggesting the linear spatial organization is not unique to one scale or architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A fair test would hold out composed directions from the PCA fit; because the reported subspace is fitted to the six atomic vectors being compared, the near-perfect composition alignment in PCA space is likely inflated relative to the model's true internal geometry.
If the spatial code is as algebraic as the paper suggests, a natural next test is whether motion and temporal dynamics occupy the same subspace as linear transition operators, since the paper explicitly leaves transition functions out.
A stronger statement would follow if continuous coordinates, not just six categorical relations, were linearly decodable; the current dataset only asks for discrete directions.
The steering result hints at an alignment application: injecting spatial directions could correct or constrain object-relation behavior without fine-tuning, though the paper does not test downstream task performance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's central claim—that LLMs encode a linear spatial world model with compositional vector addition—is not supported by the evidence as presented. The composition test (Property 2) is evaluated in a PCA subspace fit to the six atomic probe directions that define the sums. In that subspace composed vectors align beautifully (mean cosine 0.99), but in the original 3072-dimensional activation space the same comparison gives mean cosine 0.40 and angle 66.7 degrees. The projection is doing the work. Table 1 makes this visible: at layer 24, 'in front'/'behind' go from cosine 0.11 in original space to 0.995 in PCA. A projection chosen from those same vectors inflates antipodal alignment even for relations that are nearly orthogonal in the model. So the abstract's 'linear spatial world models' claim rests on a projection artifact.

That said, there is real value here. The formal definition of a spatial world model (basis, antipodal inverses, composition) is clean and useful. The probing results are solid: linear probes hit 100% accuracy at multiple layers, matching nonlinear probes, which is genuine evidence that spatial relation information is linearly decodable. The antipodal structure in PCA—even if inflated—does appear consistently across models (Llama-3.2-3B, 1B, Qwen3), which is worth noting. The steering results (74.3% overall) are suggestive, but they too use the same PCA directions, and there are no controls with random directions or null steering.

The paper also omits a key experimental detail: how the composed relation sentences (e.g., 'above and right') were generated. That matters for reproducibility. The limitations section is honest about scope but does not flag the circularity in the composition analysis.

My take: the paper is a genuine attempt with a good formal skeleton, but the load-bearing composition result is methodologically unsound as reported. The fix is straightforward—evaluate composition in a held-out subspace or in the original space with proper baselines. The weaker claim (linear decodability of spatial relations, some antipodal structure) likely survives; the strong claim (Euclidean isomorphism with vector arithmetic) does not. I would send this to a serious referee, because the question is important and the flaws are fixable. A desk reject would miss the salvageable core.

Recommendation: engage with it as a borderline paper that needs a major revision: report original-space composition metrics, add a null-subspace control, describe composed sentence generation, and add steering baselines. Then the community can judge whether a linear spatial world model actually emerges.

Referee Report

4 major / 6 minor

Summary. This paper proposes a formal definition of a linear spatial world model as a subspace of the residual stream that is approximately isomorphic to R^3, with atomic spatial relations encoded as basis vectors, inverse relations as antipodal vectors, and composed relations as vector sums. The authors train linear and nonlinear probes on a synthetic dataset of object-relation sentences, apply PCA to the probe directions, and report that the resulting low-dimensional subspace exhibits antipodal inverse relations and compositional structure. They also perform activation steering in this subspace and report a 74.3% success rate, concluding that LLMs encode linear spatial world models that are causally used during generation.

Significance. If the central claim were established, this would be a meaningful contribution to interpretability, connecting the linear representation hypothesis with spatial reasoning in LLMs and extending prior work on world models in chess and Othello. The paper has several strengths: it reports original-space metrics alongside PCA metrics, tests multiple model families (LLaMA-3.2-3B, LLaMA-3.2-1B, Qwen3-1.7B), provides a formal definition, and attempts causal interventions. However, the main evidence for the world-model claim is compromised by a circular analysis: the PCA subspace used for the antipodal and composition tests is fitted to the same atomic relation vectors that are subsequently evaluated. The causal steering experiments lack null controls. As a result, the paper does not currently provide valid empirical support for its abstract claim that LLMs encode linear spatial world models.

major comments (4)

[Section 5.2, Tables 2 and 3] The composition analysis is circular. The PCA subspace into which vectors are projected is computed from the six atomic probe directions, as described in Appendix F. The sum of atomic vectors therefore lies in the span of the fitted directions, and the projection is chosen to maximize the variance of those exact directions. The reported PCA cosines (mean 0.9931, angle 6.02°) are measured between this sum and the projected composed vector, while the original-space comparison gives mean cosine 0.3952 and angle 66.72°. The inflation from 66.72° to 6.02° is a direct consequence of discarding the orthogonal components, not evidence of internal Euclidean structure. A proper test would use a subspace fitted on held-out relations or on an independent set of directions, or would compare against a null distribution of random subspaces. Without such a control, Property 2 and the composition claim are unsupported.
[Section 5.1, Table 1] The antipodal-structure claim suffers from the same circularity. In original activation space, only left/right shows a strong negative cosine (-0.964 at layer 24), while above/below has cosine 0.4465 and in front/behind has cosine 0.1130, which is nearly orthogonal. After projection into a PCA subspace fitted to these same six relation vectors, all pairs become near-antipodal (e.g., in front/behind goes from 0.1130 to 0.9950 at layer 24). This transformation is an expected property of projecting a small set of vectors onto their own principal subspace, not a discovery about the model's internal geometry. The paper needs an independent method for identifying the spatial subspace, such as cross-validated PCA on held-out relations or a subspace derived from object-position activations, before the basis property can be accepted.
[Section 5.4, Figure 4] The causal steering experiment lacks the necessary baselines. Steering vectors are constructed from the same PCA directions used elsewhere in the paper, and there is no control condition using random directions of matched norm or directions from unrelated semantic relations. Without such controls, the observed 74.3% success rate does not demonstrate that the spatial subspace is causally used; any direction with a consistent effect on the output could produce steering. In addition, the 'behind' condition succeeds in only 5% of trials, which is far below the 16.7% chance level for six alternatives. The paper does not discuss this result, but it suggests that the steering direction for 'behind' has an opposite or non-specific effect, further weakening the claim that all six basis directions are functionally active.
[Section 5.1 and Appendix E] The near-perfect linear probe accuracy may reflect direct decoding of the surface relation words rather than an abstract spatial world model. The synthetic prompts contain the explicit relation terms ('above', 'below', 'left of', etc.) in the input, and the probes are trained on the final token's residual stream. The model could be linearly separating the input words themselves. The paper does not include a control with paraphrased relation descriptions, non-spatial relations, or object positions that are not verbally stated. Without such controls, the probing results are consistent with lexical memory rather than with a grounded spatial representation.

minor comments (6)

[Figure 4 caption] The word 'In ront' appears in the steering success table; this should be 'In front'.
[Section 5.3 Motivation] The definition of a spatial world model in Section 3.1 is W = <R^3, O, S>, but Section 5.3 writes W = <R^3, O, S, T>, introducing an undefined T. Please align the notation.
[Section 5.1 Results] The sentence 'Results are summarized in Appendix 4' is unclear; there is no 'Appendix 4'. Presumably the authors mean Table 4 in Appendix E, but this should be corrected.
[Section 5.1 and 5.3] There are several typos, including 'strucutred' instead of 'structured' and 'structre' instead of 'structure'.
[Section 5.4 and Figure 4] The text says the authors tested 'six canonical spatial relations and their inverses', but Figure 4 lists only six conditions. Please clarify whether the reported rates are for the forward relations only or pooled over both directions, and why the table does not include inverse conditions separately.
[Appendix C] The Qwen3-1.7B model is referenced to Qwen2 and Qwen2.5 technical reports (refs [53, 54]); please provide the correct citation for the Qwen3 model actually used.

Circularity Check

2 steps flagged · score 6.0 of 10

Composition and antipodal-basis results are evaluated in a PCA subspace fitted from the same atomic relation vectors, so the high alignments are partly projection artifacts.

fitted input called prediction [Section 5.2, Tables 2-3; Appendix F]
"We perform this procedure in both the original activation space and in the PCA-reduced subspace identified earlier... Table 2: Mean Orig: 0.3952 PCA: 0.9931 ... Orig: 66.72 PCA: 6.02. Appendix F: We apply Principal Component Analysis (PCA) to the corresponding linear probe directions... We stack these into a matrix D... and compute the empirical covariance matrix."

The 'predicted' composed vector is the sum of atomic vectors, projected into a PCA subspace that was itself fitted from those same atomic probe directions (Appendix F). By linearity, the projected sum lies in the span of the atomic vectors used to define the subspace, so the projection cannot test whether the sum has a component outside that span. The paper's own original-space numbers show the composed mean is mostly orthogonal to that span: mean cosine 0.3952, angle 66.72 degrees. Projecting discards the orthogonal component, and the PCA cosine rises to 0.9931, angle 6.02 degrees. The high alignment is therefore largely an artifact of evaluating the sum in the subspace fitted from its own summands, not independent evidence that composition holds in the model's native activation geometry.
self definitional [Section 5.1, Table 1; Appendix F]
"dimensionality reduction identify a lower-dimensional subspace that captures the structure of these spatial relations. Within this subspace, we observe a geometric organization: relational pairs such as above and below are encoded as antipodal directions... Table 1: in front↔behind 0.1130 83.51 0.9950 5.75."

The subspace is obtained by PCA on the atomic probe directions themselves, so the projected arrangement of those same directions is a summary of the fitted subspace rather than an independent prediction. Projecting vectors into their own best-fit plane is chosen to maximize their spread, which inflates antipodal and orthogonal alignment. Table 1 exposes the effect: in front↔behind has cosine 0.1130 (83.51 degrees) in the original activation space but 0.9950 (5.75 degrees) after projection. The paper uses the projected numbers to conclude that 'the model represents spatial semantics within a subspace that is approximately isomorphic to R3,' but the R3-like geometry is substantially manufactured by selecting the projection from the very relation vectors being displayed.

full rationale

The paper's central claim depends on Properties 1 and 2: atomic spatial directions should be antipodal/orthogonal, and composed relations should equal vector sums. Both properties are evaluated primarily inside a PCA subspace, and the only subspace-construction description in the paper (Appendix F) fits that subspace by PCA on the atomic linear-probe directions. Consequently, the projected atomic configuration and the projected sums of atomics are not independent of the subspace used for evaluation. The original-space numbers reported in Tables 1-3 make the projection artifact visible: inverse pair in front/behind goes from cosine 0.1130 (83.51 degrees) to 0.9950 (5.75 degrees), and composition mean cosine goes from 0.3952 (66.72 degrees) to 0.9931 (6.02 degrees). Projecting onto the span of the atomic vectors removes exactly the orthogonal components that cause the poor original-space fit, so the high PCA alignments are partly by construction. This is load-bearing because the abstract's 'linear spatial world models' claim rests on these geometric properties. I find no load-bearing self-citation: prior work is cited externally, and the formal framework is the authors' own definition rather than a cited uniqueness theorem. The causal steering experiments are not circular in the same way, since they test whether the identified directions affect generation, but they use the same PCA-derived steering vectors and report no null or random-direction controls, so they do not independently validate the geometry. Overall, the central compositional and basis evidence is partially circular, giving a score of 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on a predefined 3-dimensional PCA subspace fit to probe directions, an unspecified steering scale, and post hoc layer selection. The definition of the world model is introduced by the paper itself, and the composition test is evaluated in the space that was fit to define it.

free parameters (3)

PCA subspace dimensionality = 3 (main), 2 (location analysis)
The number of spatial dimensions is set to 3 to match Euclidean space R^3, not determined by the data.
Steering scale alpha = not reported
Equation in Section 4.3 h' = h + alpha * v_r requires a scale, which is never specified in the paper.
Layer selection (8, 16, 24) = layers 8, 16, 24
Chosen post hoc based on the probe accuracy heatmap in Appendix E.1; not a predetermined hypothesis.

assumptions (3)

domain assumption Linear representation hypothesis: interpretable features correspond to linear directions in activation space.
Invoked in Section 2 and Section 5.1; the entire probing and steering framework presupposes this.
ad hoc to paper The PCA-projected subspace is isomorphic to R^3 and stable across prompts.
Section 5.1 assumes the top-3 PCA subspace of the probe directions represents the true spatial geometry; this is the core assumption that the composition test relies on.
domain assumption Sentence templates assign consistent Euclidean coordinates to objects.
The synthetic dataset generation (Appendix B) assumes that natural language relations map to fixed relative positions, which is the ground truth for the probes.

invented entities (1)

3D spatial world model subspace independent evidence
purpose: Claimed internal Euclidean coordinate frame for object positions in the residual stream.
The steering experiment is a falsifiable handle (interventions change outputs), but the same subspace is used for measurement and intervention, so independent confirmation is incomplete.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Linear Spatial World Models Emerge in Large Language Models." pith.science (2026). https://pith.science/paper/OZBJXRSP

@misc{pith2026250602996,
  author       = {Pith},
  title        = {Pith review of: Linear Spatial World Models Emerge in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZBJXRSP}},
  note         = {Machine review of arXiv:2506.02996}
}

read the original abstract

Large language models (LLMs) have demonstrated emergent abilities across diverse tasks, raising the question of whether they acquire internal world models. In this work, we investigate whether LLMs implicitly encode linear spatial world models, which we define as linear representations of physical space and object configurations. We introduce a formal framework for spatial world models and assess whether such structure emerges in contextual embeddings. Using a synthetic dataset of object positions, we train probes to decode object positions and evaluate geometric consistency of the underlying space. We further conduct causal interventions to test whether these spatial representations are functionally used by the model. Our results provide empirical evidence that LLMs encode linear spatial world models.

Figures

Figures reproduced from arXiv: 2506.02996 by the authors.

**Figure 1.** Projection on the plane of 3-D PCA vectors representing atomic spatial relations for layer 24 of Llama-3.2-8B-Instruct model. (a) {above, below, right, left}. (b) {left, right, in front, behind}. (c) {above, below, in front, behind} [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

**Figure 2.** Visualization of spatial relation basis vectors and their compositional structure in PCA space. Results Our results identify a low-dimensional subspace in which spatial relation representations exhibit clear compositional structure [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** The relative position of two objects can be retrieved from their embeddings. (a) Object 1 shows clear embedding clusters for each base spatial relation; (b) Object 2 approximates flipped configuration of object 1, yielding a mirrored embedding structure. This confirms that the learned spatial embeddings preserve relative positioning. 5.3 Location of Objects Motivation. We have identified a structured subspace isomor… view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Steering success by spatial relation with 95 % confidence intervals. We use the 3D subspace we have isolated above to steer the models. Experiments. As detailed in Section 4.3, we evaluate whether the spatial subspace basis of R 3 , identified via probing, is causally …

**Figure 5.** Figure 5: Probing Workflow. Causal Intervention using learned ith direction i.e., wℓ,i, which captures the spatial relation between two objects in the activation space. Sentence with spatial relation: the ball is above the table. wℓ,i LLM Intervention a (α) ℓ,i := xℓ + αw⊤ ℓ,iaℓ…

**Figure 6.** Figure 6: Causal Intervention Workflow. E Additional Linear Probe Analysis for Llama-3.2-3B-Instruction E.1 Trends in Linear Probe Accuracy Across Model Layers In this section, [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Heatmap of classification accuracy for linear probes trained across all layers of Llama-3.2-3B-Instruct. The x-axis corresponds to model layers, from the input-adjacent bottom layer ℓ1 to the top layer ℓ24. The y-axis indicates the four atomic spatial relations: above,…

**Figure 8.** Figure 8: Layer ℓ24 decision boundaries for inverse relation pairs in Llama-3.2-3B-Instruct, shown in the 2D-PCA subspace: (a) above ↔ below, (b) left ↔ right. Additional 2D and 3D PCA Compositional Relations [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: 2D PCA projection of atomic spatial relations: above, below, left, and right. Compositional relations are shown as dashed lines and closely align with the true diagonal directions, reflecting consistent geometric structure in the embedding space. relations are represen…

**Figure 10.** Figure 10: 2-dimensional normalized PCA projection of vectors representing atomic spatial relations {above, below, right, left} for the Llama-3.2-1B-Instruct model. (a) Data extracted from layer ℓ10. (b) Data extracted from layer ℓ20. Results for Qwen3-1B. For this model, we use…

**Figure 11.** Figure 11: Layer ℓ8 decision boundaries for inverse relation pairs in Llama-3.2-1B-Instruct, shown in the 2D-PCA subspace: (a) above ↔ below, (b) left ↔ right. dec. boundary x y −3 −3 3 3 above below left right (a) above ↔ below dec. boundary x y −3 −3 3 3 above below left right…

**Figure 12.** Figure 12: Layer ℓ16 decision boundaries for inverse relation pairs in Llama-3.2-1B-Instruct, shown in the 2D-PCA subspace: (a) above ↔ below, (b) left ↔ right. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: 2-dimensional normalized PCA projection of vectors representing atomic spatial relations {above, below, right, left} for the Qwen3-1B model. (a) Data extracted from layer ℓ16. (b) Data extracted from layer ℓ20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

**Figure 14.** Figure 14: Layer ℓ16 decision boundaries for inverse relation pairs in Qwen3-1B, shown in the 2D-PCA subspace: (a) above ↔ below, (b) left ↔ right. dec. boundary x y −1 −1 1 1 above below left right (a) above ↔ below dec. boundary x y −1 −1 1 1 above below left right (b) left ↔ …

**Figure 15.** Figure 15: Layer ℓ20 decision boundaries for inverse relation pairs in Qwen3-1B, shown in the 2D-PCA subspace: (a) above ↔ below, (b) left ↔ right. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 23 canonical work pages

[1]

Emergent abilities of large 9 language models.arXiv preprint arXiv:2206.07682, 2022

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large 9 language models.arXiv preprint arXiv:2206.07682, 2022

arXiv 2022
[2]

Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

arXiv 2023
[3]

World models.CoRR, March 2018

David Ha and Jürgen Schmidhuber. World models.CoRR, March 2018. doi: 10.5281/zenodo. 1207631. URLhttp://arxiv.org/abs/1803.10122

arXiv 2018
[4]

World-model interpretability is all we need.AI Alignment Forum, January 2023

Thane Ruthenis. World-model interpretability is all we need.AI Alignment Forum, January 2023. URL https://www.alignmentforum.org/posts/HaHcsrDSZ3ZC2b4fK/ world-model-interpretability-is-all-we-need

work page 2023
[5]

Ilker Yildirim and L. A. Paul. From task structures to world models: What do llms know?CoRR,

work page
[7]

Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970

Roger C Conant and W Ross Ashby. Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970

work page 1970
[8]

Francis and Walter Murray Wonham

Bruce A. Francis and Walter Murray Wonham. The internal model principle of control the- ory.Automatica, 12(5):457–465, 1976. URL https://www.sciencedirect.com/science/ article/pii/0005109876900066. Publisher: Elsevier

arXiv 1976
[9]

Robust agents learn causal world models.ICLR Oral, February 2024

Jonathan Richens and Tom Everitt. Robust agents learn causal world models.ICLR Oral, February 2024. URLhttp://arxiv.org/abs/2402.10877

arXiv 2024

Show all 56 references

[10]

Climbing towards nlu: On meaning, form, and understanding in the age of data

Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 5185–5198, 2020

2020
[11]

Experience grounds language.arXiv preprint arXiv:2004.10151, 2020

Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language.arXiv preprint arXiv:2004.10151, 2020

2004 arXiv
[12]

Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big?ACM FAccT, pages 610–623, March 2021. doi: 10.1145/3442188.3445922. URL https://dl.acm.org/doi/10.1145/ 3442188.3445922

2021
[13]

Actually, othello-gpt has a linear emergent world representation.Neel Nanda’s Blog, March 2023

Neel Nanda. Actually, othello-gpt has a linear emergent world representation.Neel Nanda’s Blog, March 2023. URL https://neelnanda.io/mechanistic-interpretability/ othello

2023
[14]

On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

2021
[15]

On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

2021 arXiv
[16]

The alignment problem from a deep learning perspective, 2023

Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023

2023
[17]

World models: The safety perspective

Zifan Zeng, Chongzhe Zhang, Feng Liu, Joseph Sifakis, Qunli Zhang, Shiming Liu, and Peng Wang. World models: The safety perspective. In2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 369–376. IEEE, 2024

2024
[18]

Evaluating the world model implicit in a generative model.Advances in Neural Information Processing Systems, 37:26941–26975, 2024

Keyon Vafa, Justin Chen, Ashesh Rambachan, Jon Kleinberg, and Sendhil Mullainathan. Evaluating the world model implicit in a generative model.Advances in Neural Information Processing Systems, 37:26941–26975, 2024. 10

2024
[19]

Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B

Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Z...

2023 doi arXiv
[20]

Toy models of superposition.Transformer Circuits Thread, 2022

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition.Transformer Circuits Thread, 2022. URL https://transformer-circuits. pub/2022/toy_model...

2022
[21]

Distributed representations: Composition & superposition.Trans- former Circuits Thread, 2023

Chris Olah. Distributed representations: Composition & superposition.Trans- former Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/ superposition-composition/index.html

2023
[22]

Taking features out of superposition with sparse autoencoders.AI Alignment Forum, 2022

Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders.AI Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/ interim-research-report-taking-features-out-of-superposition

2022
[23]

Mathematical models of computation in superposition.ICML MI Workshop, August 2024

Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition.ICML MI Workshop, August 2024. doi: 10.48550/arXiv.2408. 05451. URLhttp://arxiv.org/abs/2408.05451

2024 doi arXiv
[24]

John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations.NAACL HLT, pages 4129–4138, June 2019. doi: 10.18653/v1/N19-1419. URL https://aclanthology.org/N19-1419

2019 doi
[25]

Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022

Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022

2022
[26]

Probing the probing paradigm: Does probing accuracy entail task relevance?ACL, pages 3363–3377, 2021

Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance?ACL, pages 3363–3377, 2021. doi: 10.18653/v1/ 2021.eacl-main.295. URLhttps://aclanthology.org/2021.eacl-main.295

2021 doi
[27]

Zoom in: An introduction to circuits.Distill, 5(3):e00024–001, 2020

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 5(3):e00024–001, 2020

2020
[28]

Toward transparent ai: A survey on interpreting the inner structures of deep neural networks

Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 464–483. IEEE, 2023

2023
[29]

Mechanistic interpretability for ai safety - a review

Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety - a review. CoRR, April 2024. URLhttp://arxiv.org/abs/2404.14082

2024 arXiv
[30]

A comprehensive mechanistic interpretability explainer & glos- sary.Neel Nanda’s Blog, December 2022

Neel Nanda. A comprehensive mechanistic interpretability explainer & glos- sary.Neel Nanda’s Blog, December 2022. URL https://www.neelnanda.io/ mechanistic-interpretability/glossary

2022
[31]

Activation addition: Steering language models without optimization.CoRR, September 2023

Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization.CoRR, September 2023. doi: 10.48550/arXiv.2308.10248. URL http://arxiv.org/abs/2308. 10248

2023 doi arXiv
[32]

Attribution patching: Activation patching at industrial scale.Neel Nanda’s Blog, February 2023

Neel Nanda. Attribution patching: Activation patching at industrial scale.Neel Nanda’s Blog, February 2023. URL https://www.neelnanda.io/mechanistic-interpretability/ attribution-patching

2023
[33]

Chess as a testbed for language model state tracking

Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as a testbed for language model state tracking. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385–11393, 2022. 11

2022
[34]

Emergent world representations: Exploring a sequence model trained on a synthetic task.arXiv preprint arXiv:2210.13382, 2022

Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task.arXiv preprint arXiv:2210.13382, 2022

2022 arXiv
[35]

Emergent world models and latent variable estimation in chess-playing language models.COLM, July 2024

Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models.COLM, July 2024. doi: 10.48550/arXiv.2403.15498. URL http://arxiv. org/abs/2403.15498

2024 doi arXiv
[36]

Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

2023 arXiv
[37]

The linear representation hypothesis and the geometry of large language models.NeurIPS Workshop on Causal Representation Learning, November 2023

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models.NeurIPS Workshop on Causal Representation Learning, November 2023. URLhttp://arxiv.org/abs/2311.03658

2023 arXiv
[38]

Hazineh, Zechen Zhang, and Jeffery Chiu

Dean S. Hazineh, Zechen Zhang, and Jeffery Chiu. Linear latent world models in simple transformers: A case study on othello-gpt.CoRR, October 2023. doi: 10.48550/arXiv.2310. 07582. URLhttp://arxiv.org/abs/2310.07582

2023 doi arXiv
[39]

Language models represent space and time.ICLR, 2024

Wes Gurnee and Max Tegmark. Language models represent space and time.ICLR, 2024. doi: 10.48550/ARXIV .2310.02207. URLhttps://arxiv.org/abs/2310.02207

2024 doi arXiv
[40]

Language encodes geographical information.Cognitive Science, 33(1):51–73, 2009

Max M Louwerse and Rolf A Zwaan. Language encodes geographical information.Cognitive Science, 33(1):51–73, 2009

2009
[41]

Representing spatial structure through maps and language: Lord of the rings encodes the spatial structure of middle earth.Cognitive science, 36(8): 1556–1569, 2012

Max M Louwerse and Nick Benesh. Representing spatial structure through maps and language: Lord of the rings encodes the spatial structure of middle earth.Cognitive science, 36(8): 1556–1569, 2012

2012
[42]

Do language models know the way to rome?arXiv preprint arXiv:2109.07971, 2021

Bastien Liétard, Mostafa Abdou, and Anders Søgaard. Do language models know the way to rome?arXiv preprint arXiv:2109.07971, 2021

2021 arXiv
[43]

Ivanitskiy, Alexander F

M. Ivanitskiy, Alexander F. Spies, Tilman Rauker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Structured world representations in maze-solving transformers.CoRR, December 2023. URL...

2023 arXiv
[44]

Mapping language models to grounded conceptual spaces.ICLR,

Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces.ICLR,
[45]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[46]

Probing classifiers: Promises, shortcomings, and advances.CoRR, September

Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.CoRR, September
[47]

Sparse autoencoders match supervised features for model steering on the ioi task.ICML MI Workshop, June 2024

Aleksandar Makelov. Sparse autoencoders match supervised features for model steering on the ioi task.ICML MI Workshop, June 2024. URL https://openreview.net/forum?id= JdrVuEQih5

2024
[48]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLe...

2023
[49]

Improving activation steering in language models with mean-centring.CoRR, December 2023

Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring.CoRR, December 2023. doi: 10.48550/arXiv. 2312.03813. URLhttp://arxiv.org/abs/2312.03813. 12

2023 doi arXiv
[50]

The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024

2024 arXiv
[51]

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexand...

2024 doi arXiv
[52]

Risks from ai misalignment at different scales, July 2024

Jan Kulveit. Risks from ai misalignment at different scales, July 2024

2024
[53]

Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

2024 arXiv
[54]

book", "mug

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

2024 arXiv
[2021]

URLhttp://arxiv.org/abs/2102.12452

doi: 10.48550/arXiv.2102.12452. URLhttp://arxiv.org/abs/2102.12452

doi arXiv
[2022]

URLhttps://openreview.net/forum?id=gJcEM8sxHK
[2023]

URLhttps://arxiv.org/abs/2310.04276

doi: 10.48550/ARXIV .2310.04276. URLhttps://arxiv.org/abs/2310.04276

doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Emergent abilities of large 9 language models.arXiv preprint arXiv:2206.07682, 2022

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large 9 language models.arXiv preprint arXiv:2206.07682, 2022

arXiv 2022

[2] [2]

Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

arXiv 2023

[3] [3]

World models.CoRR, March 2018

David Ha and Jürgen Schmidhuber. World models.CoRR, March 2018. doi: 10.5281/zenodo. 1207631. URLhttp://arxiv.org/abs/1803.10122

arXiv 2018

[4] [4]

World-model interpretability is all we need.AI Alignment Forum, January 2023

Thane Ruthenis. World-model interpretability is all we need.AI Alignment Forum, January 2023. URL https://www.alignmentforum.org/posts/HaHcsrDSZ3ZC2b4fK/ world-model-interpretability-is-all-we-need

work page 2023

[5] [5]

Ilker Yildirim and L. A. Paul. From task structures to world models: What do llms know?CoRR,

work page

[6] [7]

Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970

Roger C Conant and W Ross Ashby. Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970

work page 1970

[7] [8]

Francis and Walter Murray Wonham

Bruce A. Francis and Walter Murray Wonham. The internal model principle of control the- ory.Automatica, 12(5):457–465, 1976. URL https://www.sciencedirect.com/science/ article/pii/0005109876900066. Publisher: Elsevier

arXiv 1976

[8] [9]

Robust agents learn causal world models.ICLR Oral, February 2024

Jonathan Richens and Tom Everitt. Robust agents learn causal world models.ICLR Oral, February 2024. URLhttp://arxiv.org/abs/2402.10877

arXiv 2024