Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Decodable ≠ Causal: Transformers Store Hierarchy They Don't Use

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-04 19:46 UTC pith:VRIEB6CW

load-bearing objection Dyck-language transformers show a clean dissociation between decodable hierarchical structure and causal use — attention matters, residual stream doesn't — but the null result on residual ablation is the load-bearing weak point. the 3 major comments →

arxiv 2604.22128 v2 pith:VRIEB6CW submitted 2026-04-24 cs.CL cs.LG

Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

Aryan Sharma , Cutter Dawes , Shivam Raval This is my paper
classification cs.CL cs.LG
keywords transformer interpretabilityprobingcausal interventionDyck languagebracket sequencesresidual streamattention patternshierarchical structure
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when a transformer learns a bracket-matching task (the Dyck language of balanced parentheses), its internal representations make hierarchical variables like nesting depth, distance between brackets, and the top-of-stack position fully readable by a probe — but that decodability does not mean the model actually relies on those representations to do the task. The authors show this by comparing two interventions: masking the attention link to the correct top-of-stack position causes a sharp accuracy drop on long-distance bracket pairs, while ablating the low-dimensional residual stream subspaces that encode these same variables has comparatively little effect. The finding extends to a templated natural language setting, suggesting the gap between 'can be read off' and 'is functionally used' is not an artifact of a toy formal language. If correct, the paper establishes that probe-based interpretability — the standard method for asking 'what does this model know?' — can systematically overstate functional importance unless paired with causal intervention that actually disrupts the candidate mechanism.

Core claim

The central object is the divergence between two notions of representation in a transformer trained on Dyck bracket sequences: decodability (a linear or low-dimensional probe can extract depth, distance, and top-of-stack from the residual stream) and causal use (removing that information changes task performance). The paper finds that attention-based stack tracking is causally necessary — disrupting the attention link to the true top-of-stack position breaks long-distance accuracy — while the residual stream subspaces carrying the same hierarchical variables are not causally necessary in the same way, since ablating them barely affects performance. This dissociation is the core discovery: a)

What carries the argument

The Dyck language provides a formal setting where hierarchical structure is explicitly defined by balanced bracket sequences, giving the authors ground-truth labels for depth, distance, and top-of-stack position. The transformer trained on this task develops two candidate representations: (1) low-dimensional subspaces in the residual stream that encode these variables in a way probes can read, and (2) attention patterns that maintain a last-in-first-out stack ordering, linking each closing bracket to its matching opener. The paper intervenes on both — masking attention to the top-of-stack position versus ablating residual stream subspaces — and compares the behavioral consequences. The asym-

Load-bearing premise

The paper assumes that ablating low-dimensional residual stream subspaces is a fair test of their causal role — that if the subspace mattered, the ablation would have been strong and well-targeted enough to detect it. If the ablation was too coarse, too narrow, or missed distributed components of the representation, the 'little effect' result could be a false negative rather than genuine evidence of non-causal encoding.

What would settle it

A more fine-grained or higher-rank ablation of the residual stream — one that removes the full distributed encoding rather than a low-dimensional slice — causes a significant drop in long-distance bracket accuracy, showing that the residual stream representation is causally used but was simply more distributed than the ablation could capture.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • Probe-based interpretability results that report 'the model represents X' should be treated as evidence of information availability, not evidence that the model uses that information to produce its outputs.
  • Attention pattern analysis may be more diagnostic of functional mechanism than residual stream probing for tasks involving hierarchical or structured reasoning, since attention masking had the causal bite.
  • If residual stream representations are genuinely non-causal here, they may serve as a learned bookkeeping or logging of task variables rather than as the computational substrate — raising the question of why models invest capacity encoding information they do not use.
  • The finding could generalize beyond brackets: any task where a model's internal representations are probed for structure (syntax trees, entity tracking, logical reasoning) may show the same decodability-without-use pattern, requiring causal verification before functional claims are made.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the residual stream encodings are genuinely non-causal, they may be a byproduct of the attention mechanism's computation rather than an independent pathway — the model 'writes down' what it computed via attention but does not need to 'read back' from that written record.
  • The specific architecture of the transformer (attention vs. residual stream as parallel pathways) may predict which representations are causal: information used in attention-mediated computation will be causally dependent on attention, while the same information copied to the residual stream may be epiphenomenal.
  • A stronger test would be to check whether the residual stream encodings become causally necessary when attention is degraded or capacity-limited — if the model can 'fall back' on residual stream representations when attention is disrupted, this would show the encodings are a backup mechanism rather than truly unused.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript investigates whether hierarchical variables (depth, distance, top-of-stack) that are decodable from transformer representations are also causally used by the model. Using transformers trained on the Dyck language, the authors probe for these variables and perform two types of intervention: (1) masking attention to the true top-of-stack position and (2) ablating low-dimensional residual stream subspaces identified by probes. They report that attention masking causes a sharp drop in long-distance accuracy while residual stream ablation has comparatively little effect, leading to the conclusion that decodability does not imply causal use. The paper also claims extension to a templated natural language setting. The experimental design—combining probing with two distinct causal interventions—is well-motivated and addresses a genuine gap in the interpretability literature.

Significance. The central question—whether decodable representations are causally used—is important for the mechanistic interpretability community. The dual-intervention design (attention masking vs. residual stream ablation) is a reasonable framework for testing this question, and the choice of Dyck language provides an explicit ground-truth hierarchy, avoiding circularity concerns. If the findings hold under scrutiny, they would constitute a meaningful methodological contribution. However, this assessment is based solely on the abstract; the full text is unavailable, which severely limits verification of experimental rigor, statistical reporting, and the fairness of the ablation comparison.

major comments (3)
  1. The central claim rests on an asymmetry between two interventions: attention masking (large effect) and residual stream subspace ablation (small effect). For this asymmetry to support the conclusion that decodable residual stream representations are not causally used, the ablation must be a fair test—i.e., capable of detecting causal use if it existed. The abstract does not specify the dimensionality of the ablated subspaces, how many subspace selections were tested, or whether increasing ablation strength was explored. If the hierarchical variables are encoded redundantly across multiple subspaces or through a high-dimensional/nonlinear readout, ablating a single low-dimensional subspace would produce a null result despite genuine causal use. The attention masking intervention, by contrast, directly disrupts a specific mechanism and does not face the same granularity problem. The paper,
  2. The abstract does not specify whether error bars, confidence intervals, or multiple random seeds are reported for either intervention. Given that the central claim is comparative ('sharp drop' vs. 'comparatively little effect'), quantitative effect sizes with uncertainty estimates are load-bearing. Without this information, it is impossible to assess whether the reported asymmetry is statistically robust or an artifact of a single training run. This must be verified in the full text.
  3. The qualifier 'low-dimensional' in 'low-dimensional residual stream subspaces' is doing significant work in the argument. If the causal use operates through a higher-dimensional subspace than what was ablated, the null result is expected by construction and does not support the dissociation the paper aims to draw. The full text must clarify the relationship between the dimensionality of the ablated subspaces and the dimensionality required for accurate probing, and ideally show that the ablation was scaled until probe accuracy itself degraded.
minor comments (3)
  1. The abstract states results 'extend to a templated natural language setting' but provides no detail on the setting, the template structure, or the magnitude of effects. The full text should clearly describe this extension and whether the same dissociation holds quantitatively.
  2. The term 'causal use' is used without explicit definition in the abstract. The full text should provide a precise operational definition, as 'causal use' could mean necessity, sufficiency, or some intermediate notion, each with different implications.
  3. The abstract does not mention the model architecture (number of layers, heads, dimension), training data size, or training hyperparameters. These details are needed to assess generality beyond the specific setting studied.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and constructive review. The core concern—that our residual stream ablation must be a fair test capable of detecting causal use—is well-taken, and we address each point below. We note at the outset that this review was based on the abstract alone; the full manuscript contains substantially more detail on experimental design, dimensionality, and statistical reporting, which we summarize below and will further clarify in revision.

read point-by-point responses

  1. Referee: The central claim rests on an asymmetry between two interventions: attention masking (large effect) and residual stream subspace ablation (small effect). For this asymmetry to support the conclusion that decodable residual stream representations are not causally used, the ablation must be a fair test—i.e., capable of detecting causal use if it existed. The abstract does not specify the dimensionality of the ablated subspaces, how many subspace selections were tested, or whether increasing ablation strength was explored. If the hierarchical variables are encoded redundantly across multiple subspaces or through a high-dimensional/nonlinear readout, ablating a single low-dimensional subspace would produce a null result despite genuine causal use.

    Authors: This is a fair and important concern. The full text does specify the dimensionality of ablated subspaces and reports results across multiple subspace dimensionalities (we ablate subspaces of dimension 1 through 16 identified by linear probes for each hierarchical variable). We also report results for ablating the union of all probed subspaces across depth, distance, and top-of-stack simultaneously. However, the referee's deeper point—that causal use could operate through a high-dimensional or nonlinear readout not captured by our linear probes—is one we cannot fully rule out with the current experimental design. We agree that this limitation should be stated more explicitly in both the abstract and the discussion. We will revise the abstract to specify the range of dimensionalities tested and will add a dedicated limitations subsection addressing the redundancy/nonlinear-readout concern. We will also add the experiment the referee suggests in Comment 3: scaling ablation until probe accuracy itself degrades, to demonstrate that the ablation procedure is capable of disrupting the probed information. revision: partial

  2. Referee: The abstract does not specify whether error bars, confidence intervals, or multiple random seeds are reported for either intervention. Given that the central claim is comparative ('sharp drop' vs. 'comparatively little effect'), quantitative effect sizes with uncertainty estimates are load-bearing. Without this information, it is impossible to assess whether the reported asymmetry is statistically robust or an artifact of a single training run.

    Authors: The referee is correct that the abstract omits quantitative effect sizes and uncertainty estimates. The full text reports results across multiple random seeds (3 seeds for the Dyck experiments, 3 for the templated natural language setting) and includes standard deviations. The asymmetry between attention masking and residual stream ablation is consistent across seeds. We will revise the abstract to include representative quantitative effect sizes (e.g., accuracy drop magnitudes with standard deviations) so that the central comparative claim can be assessed from the abstract alone. revision: yes

  3. Referee: The qualifier 'low-dimensional' in 'low-dimensional residual stream subspaces' is doing significant work in the argument. If the causal use operates through a higher-dimensional subspace than what was ablated, the null result is expected by construction and does not support the dissociation the paper aims to draw. The full text must clarify the relationship between the dimensionality of the ablated subspaces and the dimensionality required for accurate probing, and ideally show that the ablation was scaled until probe accuracy itself degraded.

    Authors: We agree that this is the crux of the matter. In the current manuscript, we do report probe accuracy as a function of subspace dimensionality and show that the probed variables are accurately decodable from relatively low-dimensional subspaces (e.g., depth is decodable with >95% accuracy from a 4-dimensional subspace). We then ablate these same subspaces. However, the referee's suggestion to scale ablation until probe accuracy itself degrades is an excellent one that we have not yet performed. This would serve as a positive control demonstrating that our ablation procedure is capable of destroying the probed information when applied at sufficient strength. We will add this experiment in revision. If probe accuracy degrades while task performance remains largely unaffected, this would substantially strengthen our dissociation claim. If, conversely, task performance does degrade at higher ablation strengths, this would bound the scope of our claim and we would revise accordingly. revision: yes

standing simulated objections not resolved
  • We cannot fully rule out the possibility that causal use operates through a nonlinear or high-dimensional readout that our linear probing and ablation methodology does not capture. The scaled-ablation experiment (Comment 3) will address the linear case, but the nonlinear case remains an inherent limitation of the probing-based intervention framework. We will state this limitation explicitly but cannot fully resolve it within the scope of this paper.

Circularity Check

0 steps flagged

No circularity detected: the derivation chain is self-contained against an external ground truth (Dyck language).

full rationale

The paper trains transformers on the Dyck language, where the hierarchical ground truth (balanced brackets, depth, distance, top-of-stack) is defined externally by the formal language itself, not by the authors or the model. The probing step identifies which variables are decodable from representations; the intervention step (attention masking vs. residual subspace ablation) tests whether those representations are causally used. Neither step reduces to the other by construction: the probes are not fitted to the intervention outcomes, and the interventions target mechanisms (attention patterns, residual subspaces) that are distinct from the linear probes. No self-citation is visible in the abstract, no uniqueness theorem is invoked, and no fitted parameter is renamed as a prediction. The skeptic's concern about whether low-dimensional subspace ablation is a fair test of causal contribution is a legitimate correctness risk (false negative due to distributed/redundant encoding), but it is not a circularity problem — the conclusion is not forced by the inputs by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters, axioms beyond standard domain assumptions, or invented entities are evident from the abstract. The Dyck language, linear probes, attention masking, and subspace ablation are all established tools. Full text may reveal hyperparameters or methodological choices that function as free parameters, but none are visible at the abstract level.

axioms (3)
  • domain assumption Dyck language provides an unambiguous ground-truth hierarchical structure against which model representations can be compared.
    Standard in formal language theory; the paper uses Dyck as the training and evaluation domain. This is a reasonable domain assumption, not an ad hoc construction.
  • domain assumption Linear probes are a valid tool for detecting whether information is decodable from a representation.
    Standard assumption in mechanistic interpretability. The paper's contribution is partly to show this assumption does not extend to causal claims, but the probing methodology itself is taken as given.
  • domain assumption Attention masking and subspace ablation are valid causal intervention methods for isolating the functional role of representations.
    The paper's central experimental logic depends on these interventions being meaningful tests of causal use. The weakest assumption (above) concerns whether the ablation is performed at sufficient strength.

reviewed 2026-07-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Dissociating Decodability and Causal Use in Bracket-Sequence Transformers." pith.science (2026). https://pith.science/paper/VRIEB6CW

@misc{pith2026260422128,
  author       = {Pith},
  title        = {Pith review of: Dissociating Decodability and Causal Use in Bracket-Sequence Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VRIEB6CW}},
  note         = {Machine review of arXiv:2604.22128}
}
Share X LinkedIn Reddit HN
read the original abstract

When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention patterns maintaining a last-in, first-out ordering. However, it remains unclear whether these representations are causally used or merely decodable. We examine this gap in transformers trained on the Dyck language (a formal language of balanced bracket sequences), where the hierarchical ground truth is explicit. By probing and intervening on the residual stream and attention patterns, we find that depth, distance, and top-of-stack signals are all decodable, yet their causal roles diverge. Specifically, masking attention to the true top-of-stack position causes a sharp drop in long-distance accuracy, while ablating low-dimensional residual stream subspaces has comparatively little effect. These results, which extend to a templated natural language setting, suggest that even in a controlled setting where the relevant hierarchical variables are known, decodability alone does not imply causal use.

Figures

Figures reproduced from arXiv: 2604.22128 by Aryan Sharma, Cutter Dawes, Shivam Raval.

Figure 1
Figure 1. Figure 1: Decodability ̸= causal use. (A) At a closing bracket in a trained Dyck transformer, layer-1 attention routes sharply to the matching opener, while three latent variables (depth, distance, and top-of-stack) are well-defined at this position. (B) All three variables are decodable from the model. (C) Only top-of-stack is causally used: ablating the probe-aligned residual-stream subspaces for depth and distanc… view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics for the d = 64 base model on the first 50k steps. Task accuracy (red), depth probe Pearson IID (dark purple), and layer-1 stack-top attention mass (blue) all stabilize by step 5k. The full 500k-step trajectory is in view at source ↗
Figure 3
Figure 3. Figure 3: shows the corresponding picture through attention. Each row of the heatmap is an attention distribution for a sin￾gle query position, and the orange triangles mark the rows corresponding to closing brackets, i.e., the positions whose next-token prediction depends on retrieving the matching opener. In layer 0, these marked rows are diffuse, mean￾ing attention is spread broadly across source positions. In la… view at source ↗
Figure 4
Figure 4. Figure 4: Decodability across splits. Depth and distance (pur￾ple) report residual-probe Pearson r; top-of-stack (blue) reports layer-1 attention mass. Top-of-stack stays near-perfect across IID, OOD-length, and OOD-depth; depth and distance degrade but stay positive. Long-distance task accuracy is ≥ 0.994 across all splits. Depth Distance Top-of-stack 0.0 0.2 0.4 0.6 0.8 1.0 |Δ| L D a c c ura c y dro p 0.000 0.000 … view at source ↗
Figure 5
Figure 5. Figure 5: Causal intervention |∆| long-distance accuracy: resid￾ual ablation of depth (left), residual ablation of rank-2 distance subspace (middle), and layer-1 attention-edge knockout of top-of￾stack (right). Top-of-stack routing is causally necessary. In our base d = 64 models, layer 1 attention to the true stack-top po￾sition is ≈ 0.98–0.997, compared to a uniform baseline of ≈ 0.013 (Appendix A.10). This routin… view at source ↗
Figure 7
Figure 7. Figure 7: Full 500k-step training dynamics on a log x-axis. Task accuracy (red), depth probe Pearson IID (dark purple), and layer-1 stack-top attention mass (blue) all stabilize well before the end of training. Dashed vertical line marks step 50k. former block without standardization. For activation patch￾ing experiments, we train probes on clean sequences only and apply them to corrupted forward passes. Our depth p… view at source ↗
Figure 8
Figure 8. Figure 8: Full activation patching recovery heatmap across cache points (rows) and offsets from the closing bracket (columns). increases, yet the task performance still remains near ceiling, further emphasizing decodability without causality. Further￾more, the layer 1 attention directs over 98% of its weight to the true stack-top position in all variants, compared to a uniform baseline of 1.3%–1.7% (i.e., what equal… view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy drop from ablating the probe-aligned distance subspace (solid) vs. a random subspace of equal rank (dashed), across ranks r ∈ {2, 4, 8, 16, 32, 64}. causes substantial drops, showing the decoded subspace is not causally privileged. As rank grows, both ablations converge to near-total accuracy loss, but the probe-aligned subspace remains marginally less damaging than random even at full rank — inco… view at source ↗
Figure 11
Figure 11. Figure 11: shows our probe and task accuracy as a function of relative sequence position. The top-of-stack probe is near-perfect (≈ 100%) throughout, dropping only to 97.1% at sequence end. The depth probe Pearson r hovers around 0.40–0.45 with no positional trend, consistent with weak linear decodability. Next-token accuracy rises from 25% to 77% near sequence end, reflecting increasing predictability as sequences … view at source ↗
Figure 12
Figure 12. Figure 12: shows probe accuracy on corrupted inputs by offset from the corruption point. The top-of-stack probe 0 5 10 15 Relative sequence position (bin) 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Top-of-stack probe Depth probe Task accuracy view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LAWFUL: Law-Aligned Witness for Faithful Use of Latents

    cs.LG 2026-07 conditional novelty 7.0

    LAWFUL defines coverage-aware physical-consistency scores and circuit tests, reporting that a MoCap-to-Radar transformer's 9-component temporal circuit carries Doppler-law consistency via attention patterns.

This paper was first reviewed by glm-5.2 on July 4, 2026.