pith. sign in

arxiv: 2605.30233 · v1 · pith:4ZOPKBUCnew · submitted 2026-05-28 · 💻 cs.CL · cs.AI

Do Language Models Track Entities Across State Changes?

Zilu Tang , Qiao Zhao , Gabriel Franco , Derry Wijaya , Aaron Mueller , Sebastian Schuster , Najoung Kim This is my paper

Pith reviewed 2026-06-29 07:33 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords entity trackinglanguage modelsstate changesmechanistic interpretabilitytransformer modelsremove operationglobal suppression tagnon-incremental computation
0
0 comments X

The pith

Language models aggregate relevant entity state information in parallel only at the final query token rather than tracking changes incrementally.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how language models perform entity tracking in natural language scenarios that include multiple operations changing object states. It establishes that models do not update states token by token or propagate query-relevant information layer by layer. Instead they collect and combine the needed facts only once the query appears at the end of the input. This non-incremental strategy is shown to produce specific, predictable failures especially around removal operations. The work also demonstrates that a targeted intervention can partially correct one class of those failures.

Core claim

LMs do not incrementally track world states across tokens or query-relevant states across layers, but simply aggregate relevant information in parallel at the last token when the query becomes evident. Individual operations are handled non-incrementally as well; in particular the REMOVE operation relies on a fragile global suppression tag that predicts several observed failure modes, and nullifying this tag partially restores correct behavior.

What carries the argument

Parallel aggregation of entity-state facts at the final query token, implemented in part through a global suppression tag for the REMOVE operation.

If this is right

  • Models will fail on entity-tracking problems whose state changes must be resolved before the query appears.
  • Nullifying the global suppression tag improves accuracy on removal operations and reveals the tag as a source of fragility.
  • Mechanistic findings can be used to design new behavioral tests that target the specific failure modes the mechanism predicts.
  • A fundamentally sequential task is solved by a non-sequential computation that waits until all information is present.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same late-aggregation pattern may limit performance on other multi-step reasoning problems that require maintaining changing states.
  • Architectural modifications that encourage incremental state updates could be tested by measuring whether they produce earlier activation changes.
  • The global-suppression mechanism might generalize to other operations that require deleting or overwriting information.
  • Evaluation suites could be expanded to include queries that appear before the final state change to expose the non-incremental strategy.

Load-bearing premise

The layer-wise probes and targeted interventions accurately reflect the model's internal mechanisms for entity tracking instead of capturing only task-specific patterns or probe artifacts.

What would settle it

An experiment in which entity-state activations are shown to update incrementally in intermediate tokens or layers on the same inputs would falsify the central non-incremental claim.

Figures

Figures reproduced from arXiv: 2605.30233 by Aaron Mueller, Derry Wijaya, Gabriel Franco, Najoung Kim, Qiao Zhao, Sebastian Schuster, Zilu Tang.

Figure 1
Figure 1. Figure 1: Local, global, and mention probe accuracy across layers in CODELLAMA-13B. The low global non-trivial accuracy shows that the model does not encode global states in the final token’s residual stream, supporting H2 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Probing for prior states in CODELLAMA-13B reveals that the model also does not build states sequentially across layers, supporting H4. Each subplot shows a subset of test examples with a fixed number of local operations. Each column within the subplots shows non-trivial accuracy of the final or prior state(s). As seen in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: DESCRIPTION and PUT circuits with one PUT operation (a); their overlap (b); and functional similarity across groups using LOO-analysis (c) in CODELLAMA-13B. Group A shows the most head overlap while Group D shares the most functional similarity. The cake is in Box 3, the orange is in Box 1, … Put the pear in Box 1. Put the pin in Box 3. Box 1 contains the ____ The apple is in Box 1, the jade is in Box 2, …… view at source ↗
Figure 4
Figure 4. Figure 4: Counterfactual design for DCM (a); result of subspace activation patching in CODELLAMA-13B (b,c). We include two PUT phrases, one of which is on the query box. In the counterfactual, all DESCRIPTION and PUT phrases are shuffled respectively in their groups, and objects mapped to a new set of objects. When patching from the counterfactual to the original sentence at the last token position (“the”), the posi… view at source ↗
Figure 5
Figure 5. Figure 5: Subspace overlap between DESCRIPTION and PUT circuits in CODELLAMA-13B are high around layers 15-25, where positional information is used. tools such as activation patching are not designed to cap￾ture. Moreover, counterfactuals for REMOVE often lead to illegal state changes (e.g., one cannot remove a non-existent object). These limitations make it non-trivial to apply ex￾isting techniques on REMOVE. Hence… view at source ↗
Figure 6
Figure 6. Figure 6: Logit and rank diff for REMOVE objects in CODELLAMA￾13B indicates their rank increase after REMOVE phrase and it is observed regardless of whether REMOVE targets the queried box. Black dotted-line is 0-baseline (i.e. no diff), and red dotted-line is the average diff across all other objects. We used 2-tail Mann￾Whitney test. ****: p ≤ 0.0001. 4.2.2. TERNARY PROBE FOR REMOVE TAG The logit and rank analysis … view at source ↗
Figure 7
Figure 7. Figure 7: High local box-object probe accuracy of CODELLAMA￾13B conditioned on the object and box ID tokens suggest both contain “remove tag” signal (around L5-10). In [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Local box-object probe accuracy (CODELLAMA-13B) conditioned on Object and Box ID across phrase index indicates the “tag” signal weakens across context, especially on Box ID. Probe accuracy across phrase index [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Single layer intervention results (CODELLAMA-13B) for nullifying the remove tag (in REMOVE/MOVE-OUT) and exist tag (in PUT/MOVE-IN) suggests that the causally relevant remove tag resides in object tokens, while the exist tag can be found in box ID tokens. adding a remove tag to the object, and an exist tag to the box it is moving into. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Local, global, and mention probes across layers in LLAMA-3.1-70B also reveal that model does not encode global state in the final token’s residual stream, supporting H2. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Hypothetical probing experiment evidence that would support H3 (left) and H4 (right) in example datapoints with two local operations (three different states). If model process states sequentially through layers H3, we expect earlier prior states probe to have higher accuracy earlier in the layer. Each column shows probe accuracy of the final or prior state(s). Probes were trained with the following set of… view at source ↗
Figure 12
Figure 12. Figure 12: Probing for prior states in LLAMA-3.1-70B reveals that the model does not build states sequentially across the layer, supporting H4. Each subplots shows a subset of test examples having fixed number of local operations. Each column within the subplots shows probe accuracy of the final or prior state(s). As an example, prior state=-2 is the next state for prior state=-3 and the prior state for final state.… view at source ↗
Figure 13
Figure 13. Figure 13: Probing for prior states in CODELLAMA-13B reveals that the model also does not build states sequentially across the layer, supporting H4. Each subplots shows a subset of test examples having fixed number of local operations. Each column within the subplots shows probe accuracy of the final or prior state(s). As an example, prior state=-2 is the next state for prior state=-3 and the prior state for final s… view at source ↗
Figure 14
Figure 14. Figure 14: GEMMA-2-2B path patching results: attention heads overlap and functional similarity across groups using LOO-analysis. All groups recover a good amount of performance except for group C. For each datapoint, we generate a counterfactual sentence where each objects are uniquely mapped to a new set of object not in the original context (Example G.2). For each group, after ranking each heads by the score menti… view at source ↗
Figure 15
Figure 15. Figure 15: Subspace patching results with GEMMA-2-2B confirms that the subspaces used to transmit positional information at the last token layer are highly similar between DESCRIPTION and PUT circuits. G.7. Subspace Patching for Token Identity Retrieval (Group A) Instead of patching with the hypothesis that the residual stream contains positional information about the object, we also use subspace patching to isolate… view at source ↗
Figure 16
Figure 16. Figure 16: Subspace patching results with CODELLAMA-13B confirms that the subspaces used to copy object content information at the last token layer are highly similar between DESCRIPTION and PUT circuits. H. Remove Mechanism Details H.1. Behavioral Accuracy CODELLAMA-13B 0-shot logit argmax accuracy is around 0.84 on the task (AltForm 1remove), and the full generation accuracy accuracy is 0.32 (recall=0.83, precisio… view at source ↗
Figure 17
Figure 17. Figure 17: Logit (left) and rank (right) diff of different objects before and after adding a REMOVE phrase (CODELLAMA-13B). Each examples are also split by model logit argmax correctness. We see similar trend between two metrics, and no significant effect of model correctness on either metric. Black dotted lines denote the 0-baseline (i.e. no diff), and red dotted lines denote average rank diff across all other obje… view at source ↗
Figure 18
Figure 18. Figure 18: Rank Difference of object with 1-REMOVE operation that is applied on either the query box or an irrelevant box for LLAMA models with/without 2-shot prompts. Black dotted-line is 0-baseline (i.e. no diff), and red dotted-line is average rank diff across all other objects in the correct cases. We used 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.0… view at source ↗
Figure 19
Figure 19. Figure 19: Rank difference of object with 1-REMOVE operation that is applied on either the query box or an irrelevant box for QWEN models with/without 2-shot prompts. Black dotted-line is 0-baseline (i.e. no diff), and red dotted-line is average rank diff across all other objects in the correct cases. We used 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.00… view at source ↗
Figure 20
Figure 20. Figure 20: Rank difference of object with 1-REMOVE operation that is applied on either the query box or an irrelevant box for GEMMA models with/without 2-shot prompts. Black dotted-line is 0-baseline (i.e. no diff), and red dotted-line is average rank diff across all other objects in the correct cases. We used 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.0… view at source ↗
Figure 21
Figure 21. Figure 21: Rank difference of object with 1-REMOVE operation that is applied on either the query box or an irrelevant box for MISTRAL model with/without 2-shot prompts. Black dotted-line is 0-baseline (i.e. no diff), and red dotted-line is average rank diff across all other objects in the correct cases. We used 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.… view at source ↗
Figure 22
Figure 22. Figure 22: REMOVE phrase rank diff split by query box position (CODELLAMA-13B, 0-shot) shows that the effect of rank increase in irrelevant REMOVE diffuses across box positions (similarly for other objects), but such effect is less pronounced when the query box is in the middle of the context. Black dotted-line is 0-baseline (i.e. no rank change), and red dotted-line is average rank diff across all other objects. We… view at source ↗
Figure 23
Figure 23. Figure 23: REMOVE phrase rank diff split by query box position (CODELLAMA-13B, 2-shot) shows that the effect of the diffused rank increase from irrelevant REMOVE is not present in 2-shot setting (right) but still somewhat present for other objects, suggesting few-shot learning could alter actual mechanisms. Black dotted-line is 0-baseline (i.e. no rank change), and red dotted-line is average rank diff across all oth… view at source ↗
Figure 24
Figure 24. Figure 24: REMOVE phrase rank diff split by query box position (GEMMA-2-9B, 2-shot) shows that the effect of the diffused rank increase from irrelevant REMOVE is still present in 2-shot setting, different from that of CODELLAMA-13B. This suggests the effect of few-shot learning on task mechanism could be different among models. Black dotted-line is 0-baseline (i.e. no rank change), and red dotted-line is average ran… view at source ↗
Figure 25
Figure 25. Figure 25: Logit (left) and rank (right) difference of object with 1-PUT operation that is applied on either the query box or an irrelevant box for CODELLAMA-13B zero-shot. There is a smaller increase in logit for irrelevant PUT than query PUT. Rank diffs are clipped between [−1000, 1000]. We use 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.001, and ****: … view at source ↗
Figure 26
Figure 26. Figure 26: Logit (left) and rank (right) difference of object with 1-PUT operation that is applied on either the query box or an irrelevant box for CODELLAMA-13B two-shot. Rank diffs are clipped between [−1000, 1000]. We use 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.001, and ****: p ≤ 0.0001 [PITH_FULL_IMAGE:figures/full_fig_p030_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Logit (left) and rank (right) difference of object with 1-PUT operation that is applied on either the query box or an irrelevant box for GEMMA-2-2B zero-shot. Rank diffs are clipped between [−1000, 1000]. We use 2-tail Mann-Whitney test. ns: 0.05 < p ≤ 1, *: 0.01 < p ≤ 0.05, **: 0.001 < p ≤ 0.01, ***: 0.0001 < p ≤ 0.001, and ****: p ≤ 0.0001 30 [PITH_FULL_IMAGE:figures/full_fig_p030_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Confusion matrix for CODELLAMA-13B (layer 8) ternary probes conditioned on the object token (top) and the box ID token (bottom). In [PITH_FULL_IMAGE:figures/full_fig_p031_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Structural analysis of Box ID probe (CODELLAMA-13B) reveals more antipodal representation between exist/removed and removed signal is in a subspace of exist. H.9. Ternary Probe Results and Analysis for LLAMA-3.1-70B In [PITH_FULL_IMAGE:figures/full_fig_p031_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Probe accuracy of LLAMA-3.1-70B conditioned on Object tokens and Box ID tokens suggests both locations contain signal (around layer 5-10) for the “remove-tag” with stronger signal on Box ID. 0 20 40 60 80 Layer 0.5 0.0 0.5 1.0 Cosine similarity pair Non-Exist vs. Exist Non-Exist vs. Removed Exist vs. Removed 0 20 40 60 80 Layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Reconstruction Norm Ratio Non-exist Non-ex… view at source ↗
Figure 31
Figure 31. Figure 31: Structural analysis of Box ID probe (LLAMA-3.1-70B) reveals closer (antipodal) representation for exist/removed probe weights and that removed signal is in a subspace of exist. H.10. 1-PUT Intervention Error Analysis In Sec. 4.2.3 ( [PITH_FULL_IMAGE:figures/full_fig_p032_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Ternary probe accuracy (LLAMA-3.1-70B) conditioned on Object and Box ID across phrase index indicates the “tag” signal weakens across context at Box ID token, and not at Object token. Does Not Exist Exist Removed Predicted Does Not Exist Exist Removed True 4.8e+06 3.9e+04 1 0 6.9e+03 0 0 0 0 DESCRIPTION Does Not Exist Exist Removed Predicted 1.5e+06 1.3e+04 2.4e+02 0 2.2e+03 37 0 0 0 PUT Does Not Exist Ex… view at source ↗
Figure 33
Figure 33. Figure 33: Confusion matrix for LLAMA-3.1-70B (layer 10) ternary probes conditioned on the object token (top) and the box ID token (bottom). across layers indicates that the model up-weights probabilities of all objects adjacent to the target object, and such effect diminishes as objects are located farther away from the target object. This gaussian-like effect on the up-weighting of entity probabilities across thei… view at source ↗
Figure 34
Figure 34. Figure 34: Single layer intervention results for nullifying the remove tag (in REMOVE/MOVE-OUT) and exist tag (in PUT/MOVE-in) in LLAMA-70B suggests that despite high probing accuracy, causally relevant remove tag resides in object tokens, while exist tag is in box ID tokens. Due to computational constraints, we use 100 examples for each operation in this experiment. 0 10 20 30 40 layer 0.0 0.2 0.4 0.6 0.8 1.0 Error… view at source ↗
Figure 35
Figure 35. Figure 35: Error types across intervening examples with a single PUT on object probe suggest that even when the “exist tag” from object tokens are removed, positional information (Error (OID)) continues to have big effect on the prediction, causing models to predict objects adjacent to the PUT object. Ravfogel et al. (2021) ( [PITH_FULL_IMAGE:figures/full_fig_p034_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Intervention results on CODELLAMA-13B on first or last n layers on “remove tags” at Box ID token. We are not able to successfully recover the model from predicting the removed object, supporting our claim that causally efficacious signals for “remove tags” are not at Box ID token. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Intervention results on CODELLAMA-13B negating the “remove tag” (left) or boosting the “exist tag” (right) at Box ID token. We are not able to successfully recover the model from predicting the removed object. 0.0 0.5 1.0 Object Exist Tag Remove Tag 7 8 9 10 11 12 13 14 15 16 17 18 Phrase Index 0.0 0.5 1.0 Box ID 7 8 9 10 11 12 13 14 15 16 17 18 Phrase Index Success Type Original Object(s) Target Object S… view at source ↗
Figure 38
Figure 38. Figure 38: Single-layer intervention (at layer three) success rates across phrase index of where the query operation occurs (CODELLAMA￾13B). Left plot shows full results and right plot shows results without MOVE operation. Results confirm the ineffectiveness of intervention for remove tag at the box ID token (bottom right at either side). It also shows that the success rate for exist tag is affected by where the que… view at source ↗
Figure 39
Figure 39. Figure 39: Probe accuracy of CODELLAMA-13B completion conditioned on Box ID tokens with local states (left) and cumulative states (right). Crucially, cumulative box-object line and local box-object line on the right are much lower than local box-object line on the left, suggesting models do not accumulate “remove-tags” across phrase boundaries. from Box 1. Remove the banana from Box 2. ..., “remove-tag” for (apple, … view at source ↗
Figure 40
Figure 40. Figure 40: DCM results patching part of the REMOVE phrase (CODELLAMA-13B) suggests that positional information is not the main signal used in communication with downstream components. Left to right columns are patching at a single layer, first-n layers, and last-n layers. would mean that the model is placing a “remove-tag” at the first object (which is orange in the original sentence), which would result in model pr… view at source ↗
read the original abstract

Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer language models (LMs) solve entity binding $\textit{without}$ state changes. However, there is limited understanding of how non-toy LMs address ET problems of realistic difficulties expressed in natural language. To this end, we investigate the mechanisms underlying ET in more complex scenarios featuring multiple state-changing operations. We find that LMs do not incrementally track world states across tokens or query-relevant states across layers, but simply aggregate relevant information in parallel at the last token when the query becomes evident. We further investigate mechanisms of individual operations ($\texttt{PUT}$, $\texttt{REMOVE}$, $\texttt{MOVE}$) to characterize this non-incremental ET mechanism. Surprisingly, LMs implement the $\texttt{REMOVE}$ operation with a fragile global suppression tag; this global removal mechanism predicts various failure modes that we confirm behaviorally. We provide a mechanistic solution of nullifying this tag to partially address this issue. Overall, our findings reveal that LMs solve a fundamentally sequential task using a non-sequential strategy. More broadly, our work illustrates how behavioral and mechanistic analyses can fruitfully interact. Behavioral results inform mechanistic hypotheses, and insights from mechanistic analyses help build stronger behavioral evaluations by predicting failure modes missing from existing evaluations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that non-toy LMs solve entity tracking (ET) tasks involving multiple state changes (PUT, REMOVE, MOVE) expressed in natural language by aggregating relevant information in parallel only at the final token once the query is evident, rather than maintaining incremental state tracking across tokens or query-relevant states across layers. Behavioral experiments and mechanistic analyses (layer-wise probes, activation interventions) support this non-incremental strategy; the REMOVE operation is implemented via a fragile global suppression tag whose predicted failure modes are confirmed behaviorally, and nullifying the tag provides a partial mechanistic fix. The work emphasizes the interaction between behavioral predictions and mechanistic insights.

Significance. If the central claim holds, the result is significant for understanding LM limitations on sequential reasoning: it shows that a fundamentally incremental task is solved non-incrementally, predicts specific failure modes from the REMOVE mechanism, and demonstrates how mechanistic interventions can improve behavioral robustness. The combination of behavioral tests informing mechanistic hypotheses (and vice versa) is a methodological strength.

major comments (2)
  1. [§4 (mechanistic analysis)] The central claim that LMs 'do not incrementally track world states across tokens or query-relevant states across layers' (abstract and §4) rests on layer-wise probes and targeted interventions. However, the paper does not report controls showing that these diagnostics would detect incremental tracking if it existed (e.g., via synthetic tasks with known incremental mechanisms or ablation of probe training data). Without such validation, the null result on incremental tracking risks being an artifact of the chosen probes rather than evidence against any incremental computation.
  2. [§5.2] §5.2 (REMOVE operation): the global suppression tag is presented as the implementation of REMOVE, with the intervention of nullifying the tag offered as a fix. The behavioral predictions from this tag are confirmed, but the manuscript does not test whether the same tag (or an analogous mechanism) appears in models or tasks outside the specific synthetic templates used; this limits the generality of the 'fragile global suppression' characterization and the proposed fix.
minor comments (2)
  1. [abstract] The abstract states that LMs 'simply aggregate relevant information in parallel at the last token'; this phrasing could be clarified to distinguish parallel aggregation from other forms of non-incremental computation (e.g., deferred computation that still depends on earlier tokens).
  2. [figures and §4] Figure captions and method descriptions should explicitly state the number of models, layers probed, and statistical tests used for the layer-wise analyses to allow readers to assess the strength of the 'no incremental tracking across layers' result.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below, providing our strongest honest response while noting where revisions are warranted.

read point-by-point responses

  1. Referee: [§4 (mechanistic analysis)] The central claim that LMs 'do not incrementally track world states across tokens or query-relevant states across layers' (abstract and §4) rests on layer-wise probes and targeted interventions. However, the paper does not report controls showing that these diagnostics would detect incremental tracking if it existed (e.g., via synthetic tasks with known incremental mechanisms or ablation of probe training data). Without such validation, the null result on incremental tracking risks being an artifact of the chosen probes rather than evidence against any incremental computation.

    Authors: We agree that explicit positive controls validating the probes' ability to detect incremental tracking (if present) would strengthen the null result. Our current evidence combines correlational probes with causal interventions that successfully predict and alter behavior in line with the non-incremental account. Nevertheless, we acknowledge the referee's point as a genuine methodological gap. In revision we will add a dedicated limitations paragraph in §4 discussing probe sensitivity and will include, where space permits, a brief control experiment on a simpler incremental task. revision: partial

  2. Referee: [§5.2] §5.2 (REMOVE operation): the global suppression tag is presented as the implementation of REMOVE, with the intervention of nullifying the tag offered as a fix. The behavioral predictions from this tag are confirmed, but the manuscript does not test whether the same tag (or an analogous mechanism) appears in models or tasks outside the specific synthetic templates used; this limits the generality of the 'fragile global suppression' characterization and the proposed fix.

    Authors: We accept that the global-suppression characterization and the proposed fix are demonstrated only within the controlled synthetic templates and model family examined. This design choice enabled the tight coupling between mechanistic discovery and behavioral predictions that the referee notes as a strength. We will revise §5.2 and the conclusion to state the scope limitation explicitly and to frame broader validation across models and naturalistic tasks as an important open question. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical mechanistic findings are self-contained

full rationale

The paper derives its claims about non-incremental entity tracking and the REMOVE operation's global suppression tag exclusively from behavioral tests, layer-wise probes, and intervention experiments on LMs. These are direct empirical observations rather than derivations that reduce by construction to fitted parameters, self-definitions, or self-citation chains. No equations, uniqueness theorems, or ansatzes are invoked that presuppose the target results. The confirmation of predicted failure modes follows from the discovered mechanism without evidence that the mechanism itself was defined in terms of those outcomes. This is a standard case of an independent empirical analysis.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No explicit free parameters, axioms, or invented entities are stated in the abstract; the work relies on standard assumptions from mechanistic interpretability research.

pith-pipeline@v0.9.1-grok · 5780 in / 1001 out tokens · 25269 ms · 2026-06-29T07:33:43.201045+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Localizing Model Behavior with Path Patching

    URL https://proceedings.mlr.press/ v236/geiger24a.html. Giulianelli, M., Harding, J., Mohnert, F., Hupkes, D., and Zuidema, W. Under the hood: Using diagnostic classi- fiers to investigate and improve how language models track agreement information. In Linzen, T., Chrupała, G., and Alishahi, A. (eds.),Proceedings of the 2018 EMNLP Workshop BlackboxNLP: An...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/w18-5426 2018

  2. [2]

    emnlp-main.1565/

    URL https://aclanthology.org/2025. emnlp-main.1565/. Meng, K., Bau, D., Andonian, A., and Belinkov, Y . Locating and editing factual associations in gpt.Advances in neural information processing systems, 35:17359–17372, 2022. Merrill, W. and Sabharwal, A. The parallelism tradeoff: Limitations of log-precision transformers.Transactions of the Association f...

    2025

  3. [3]

    Merrill, W., Petty, J., and Sabharwal, A

    URL https://openreview.net/forum? id=NjNGlPh8Wh. Merrill, W., Petty, J., and Sabharwal, A. The illusion of state in state-space models. InProceedings of the 41st In- ternational Conference on Machine Learning, pp. 35492– 35506, 2024. Merullo, J., Eickhoff, C., and Pavlick, E. Talking heads: Understanding inter-layer communication in transformer language m...

    work page doi:10.1162/jocn.2006.18 2024

  4. [4]

    Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K

    URL https://openreview.net/forum? id=SJzSgnRcKX. Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as a testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11385–11393, 2022. Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S. Investiga...

    2022

  5. [5]

    0-shot” in the paper is “0-shot

    URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper. pdf. Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for in- direct object identification in gpt-2 small. InThe Eleventh International Conference on Learning Representations, 2022. Wegne...

    work page arXiv 2020

  6. [6]

    the object token (secondapple) in theREMOVEphrase

  7. [7]

    the Box ID token (secondapple) in theREMOVEphrase

  8. [8]

    the period token at the end of theREMOVEphrase

  9. [9]

    0-shot” in the paper is “0-shot

    the entireREMOVEphrase. We show the results across 200 examples for CODELLAMA-13B for intervening on the entire REMOVE phrase in Fig. 40 as it showed the best intervention accuracy, which is still only around 15%. The peak of accuracy around layer 13 in the left plot (single-layer intervention) is similar to what we observe in Fig. 4b,c, which suggests th...

    2023

  10. [10]

    Description:

    Move the map in Box 6 to Box 2. Remove the bill from Box 4. Put the coat into Box 3. Statement: Box 2 contains the bag and the machine and the map. Description:{CONTEXT} Statement: Box{QUERY BOX}contains I.4. 2-shot with instruction enumerating all box contents Prompt for 2-shot evaluation with instruction Given the description after “Description:”, write...

  11. [11]

    Remove the bill from Box 4

    Move the map in Box 6 to Box 2. Remove the bill from Box 4. Put the coat into Box 3. Statement: Box 0 contains the plane, Box 1 contains the cross, Box 2 contains the bag and the machine and the map, Box 3 contains the coat, Box 4 contains nothing, Box 5 contains the apple and the cash and the glass, Box 6 contains the bottle. Description:{CONTEXT} Statem...