Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Structure Development in List-Sorting Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dataset statistic—the mean and variance of gaps between adjacent sorted numbers—predicts how a sorting transformer organizes its attention heads.

desk verdict Solid, honest toy-model study showing gap statistics correlate with head specialization; the causal claim outstrips the data, but it deserves referee time. read the letter →

arxiv 2501.18666 v1 pith:3Y6ZLO5X submitted 2025-01-30 cs.LG cs.AIcs.NE

Einar Urdshals , Jasmina Urdshals This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords mechanisticinterpretabilitydevelopmentaltransformercircuitslistsortingvocabulary-splittingcopy-suppressionlocallearningcoefficienttrainingdatadistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what in the training data makes a transformer organize its attention heads the way it does. Training a one-layer, attention-only transformer to sort lists of numbers, the authors identify three final circuit organizations: vocabulary-splitting, where heads take over disjoint ranges of the number vocabulary; copy-suppression, where one head copies tokens and the other fine-tunes that copy; and one-head sorting, where the second head is effectively switched off. The paper's central claim is that the deciding data feature is the distribution of gaps $\delta$ between adjacent numbers in the sorted lists: larger, more varied gaps favor vocabulary-splitting, smaller and less varied gaps favor copy-suppression, and very low variance turns the second head off. It also reports that vocabulary-splitting scores as a simpler solution on the Local Learning Coefficient and emerges even without weight decay, which it reads as evidence for an inductive bias toward simpler solutions. If correct, this gives a concrete, measurable way in which training-data statistics sculpt the internal circuits of transformers, a step toward understanding larger language models.

What carries the argument

The load-bearing object is the gap statistic $\delta = l_{i+1} - l_i$, the difference between adjacent elements of a sorted list, summarized across the dataset by its mean and variance. The paper's argument runs through two interpretability tools: the OV/QK circuit decomposition, which represents each attention head as a copying circuit (diagonal of the OV matrix) and an attention-selection circuit (QK matrix), and the Local Learning Coefficient (LLC), a singular-learning-theory measure of solution complexity estimated from the local loss landscape. The mechanism is that smaller typical gaps force the QK circuit to distinguish nearby tokens more sharply, so the mean QK gradient inside each active region rises; with a roughly constant product $\delta$ times the gradient, region sizes shrink and the model splits its vocabulary differently, while very small gap variance removes the need for the second head entirely.

What would settle it

Generate two datasets with identical mean and variance of $\delta$ but different generation mechanisms, for example fixed allowed $\delta$ values versus pruned lists, and train identical models on each; if the final specialization mode tracks the generation mechanism rather than the $\delta$ statistics, the central claim is false. The paper also names the missing experiment: directly increasing or decreasing the QK gradient during training should move the model between the predicted specialization modes, and a null result would falsify the attribution.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a developmental sequence for a one-layer, two-head transformer learning to sort lists: rapid initial learning, a stage of overlapping head roles, then head specialization. The final specialization comes in three modes characterized by the OV and QK circuits: vocabulary-splitting (contiguous non-overlapping diagonal regions across heads), copy-suppression (a negative OV diagonal in one head that calibrates the copying head), and one-head sorting (the sub-leading head's circuits become orders of magnitude smaller). The paper claims these modes are selected by the mean and variance of the gap distribution $\delta$ between neighboring elements of sorted lists, and supports this with a sweep of datasets varying list length, vocabulary size, and sampling procedure. It further claims that vocabulary-splitting is a genuinely simpler solution, with lower LLC than the overlapping stage, and that it arises without weight decay, while copy-suppression is an intermediate regime that increases model confidence when the copying head is already accurate. The result is presented as a concrete demonstration that a measurable feature of training data determines the internal organization of a transformer.

Load-bearing premise

The central claim depends on the assumption that the observed specializations are caused by the mean and variance of the gap distribution $\delta$, not by other dataset properties that change at the same time (list length, vocabulary size, or distribution shape); the paper itself notes that datasets with the same mean $\delta$ are not equivalent and that it performed no intervention studies.

Editorial extensions

If this is right

  • If the central claim is right, the mean and variance of $\delta$ become a practical predictor: before full training, one can estimate from the dataset which of the three head specializations a two-head sorting transformer will settle into.
  • Vocabulary-splitting forming without weight decay implies that simpler solutions can be a natural training outcome rather than an artifact of regularization.
  • Copy-suppression as a confidence-calibrating mechanism in a high-accuracy setting means the same circuit motif can have opposite effects on confidence depending on task accuracy.
  • The rough constancy of $\delta$ times the QK gradient across datasets suggests a simple scaling law for how fine-grained the attention circuit must be to sort a given gap distribution.
  • The link between data statistics and circuit organization supports the universality hypothesis, suggesting that similar data-driven specialization could be searched for in larger transformer models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves open is whether the same $\delta$ mean-and-variance rule holds in deeper transformer stacks; if it does, gap statistics could become a diagnostic for circuit regimes in real language-model data.
  • The observed ordering from one-head sorting to copy-suppression to vocabulary-splitting with increasing gap variance suggests a phase boundary that could be mapped precisely with a variance sweep at fixed mean $\delta$.
  • Because the paper's datasets vary list length and vocabulary size alongside $\delta$, an intervention that manipulates only the gap distribution, for example by re-weighting gaps within the same vocabulary and length, would isolate the causal direction.
  • If the LLC-based simplicity ordering is robust, training curricula that begin with compact gaps and then widen them could deliberately steer a model through copy-suppression into vocabulary-splitting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies a one-layer attention-only transformer trained to sort small lists of numbers and characterizes its developmental stages (initial learning, head-overlapping, vocabulary-splitting, copy-suppression) through circuit visualizations, the Local Learning Coefficient (LLC), and circuit rank. It identifies two end-of-training specializations—vocabulary-splitting and copy-suppression—and proposes that the mean and variance of the gap distribution δ between adjacent sorted list elements controls which specialization emerges: larger, more varied gaps favor vocabulary-splitting; smaller, less varied gaps favor copy-suppression; and very low variance leads to one-head sorting. The paper also reports a roughly constant product δ·∇QK across datasets, relates copy-suppression to a mechanism previously identified in GPT-2, and argues that vocabulary-splitting is a naturally preferred simpler solution even without weight decay.

Significance. If the main causal claim held, the paper would be a valuable and concrete demonstration that a simple, measurable property of training data—the gap distribution—shapes circuit-level organization in a transformer, with implications for interpretability and for the universality hypothesis. The study is unusually candid about its limitations (Sec. 6), includes extensive dataset variations and LLC-based complexity comparisons, and provides ablation evidence on copy-suppression. Its strengths include the breadth of controlled architecture variations (1–4 heads, LN/WD ablations) and the mechanistic connection to GPT-2 copy-suppression. However, the central data-to-specialization attribution currently rests on correlational evidence across heterogeneous dataset manipulations, and the quantitative 'constant product' claim is not yet supported by the reported fit.

major comments (4)
  1. [Sec. 3.2, Table 2, Fig. 16] The central claim that the mean and variance of δ determine which specialization emerges is not supported by the current experimental design. Within the cleanest family D^{[2,k]} (fixed list length and vocabulary), increasing k changes the variance together with the support/range of δ and the number of possible lists; across the other families, list length and vocabulary size also change. The paper explicitly concedes in Sec. 2.1 that datasets with the same mean δ are not equivalent and in Sec. 6 that no intervention studies were performed. The observed ordering 1-head sorting → copy-suppression → vocabulary-splitting with increasing variance can therefore not be attributed to variance per se. I ask for matched-moment or matched-support interventions—for example datasets with identical mean, support, and list/vocabulary cardinality but different variance—or at minimum a falsifiable prediction and additional controls, before the causal claim can be accepted.
  2. [Sec. 3.2, Fig. 6] The claim that δ·∇QK is approximately constant is not quantified convincingly. The fitted line in Fig. 6 is δ·∇QK = 0.004δ + 0.468, which has a nonzero slope, and no error bars, per-seed spread, or residuals are shown. Over the displayed range of δ the product varies by a nontrivial amount if the slope is real, so the visual 'flatness' needs to be backed by an uncertainty estimate and a statement of the acceptable tolerance. In addition, the active-region definition relies on visually determined boundaries (Fig. 5, Tab. 1) and the partition is admitted to be somewhat arbitrary; a robustness check of the region-partition rule is needed before this quantitative relationship is treated as established.
  3. [Fig. 16 caption, Sec. 3.2] The specialization labels are assigned by visual inspection of the circuits, and the same figure is used to show that the labels separate cleanly by relative head weight norm. Although the caption states that the labels were made without consulting the weight norms, a visual-only labeling protocol is hard to audit. Please provide a quantitative definition of each specialization class, or an inter-rater/robustness check, so that the reported clean separation in Fig. 16 is not dependent on the authors' subjective judgment. In addition, Table 2 includes an 'Other' class (for D^{ℓ=3}) and the perturbed-dataset model is described as a different specialization mode; these cases need to be reconciled with the three-way variance story or explicitly excluded with justification.
  4. [Sec. 4.3, Fig. 10] The conclusion that copy-suppression acts by increasing model confidence is based on head ablation and a Shannon-entropy measure, but the reported support is thin. The paper states that ablating the copy-suppressing head leaves accuracy unchanged while increasing loss, and that entropy increases after ablating either head; however, the entropy computation is not specified (over which token distribution and context) and no magnitudes are given. Since the functional-role claim is a main contribution, please specify the entropy measure and report the actual loss and entropy changes, including variance across seeds, so the 'slightly increases confidence' conclusion can be evaluated.
minor comments (5)
  1. [Sec. 2.1, Eq. (1)] The symbol δ is used both for the per-list gap values and for the dataset mean, which is confusing; please introduce a different notation for the random variable (e.g., Δ) and the mean (e.g., δ̄) throughout.
  2. [Sec. 5] There is a typo: 'modular arithmetric' should be 'modular arithmetic'.
  3. [Sec. 7] The conclusion states that the toy model has 100% accuracy, but earlier appendices report a no-LN model achieving only 20% accuracy; please qualify the statement to refer to the specific baseline models and not to all trained models.
  4. [Fig. 4] The histograms in Fig. 4 would be easier to read with explicit bin widths and axis labels for the probability mass; currently the y-axis labeled 'Count' is on a log scale without stating the binning.
  5. [Sec. 3.1] The stage boundaries are described as chosen by eye from a combination of measures (Sec. 4.1). Please state whether the reported stage transitions, especially the vocabulary-splitting to copy-suppression transition in Fig. 3, are robust to an automated rule such as local extrema of the LLC or loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: δ statistics are input features, specializations are observed circuit states, and the LLC comparisons are self-contained empirical checks.

full rationale

This paper is an empirical study rather than a derivation, and I find no load-bearing step that reduces to its own inputs. The mean and variance of δ are defined directly from the training data (Eq. 1) and are compared against end-of-training circuit states that are classified by visual inspection of OV and QK circuits (Figs. 2, 3, 16), so specialization mode is not constructed from δ. The LLC is an externally defined complexity measure (Lau et al. 2023), and even though the authors use LLC changes to help delimit developmental stages, the stages are also defined by circuit-level patterns (non-overlapping OV diagonals, negative diagonals, relative head weight norms); the claim that vocabulary-splitting is 'simpler' therefore reports a correlation between an independent complexity metric and an independently identified circuit state rather than restating the stage definition. The δ·∇QK product is fitted as a descriptive trend in Fig. 6, but it is presented as a supporting observation for the region-size hypothesis and is not used to 'predict' the specialization outcome, so it is not a fitted input renamed as a prediction. The copy-suppression identification relies on prior work by McDougall et al. (2023), which is external to this paper and not authored by the present authors; no uniqueness theorem or ansatz is smuggled in via self-citation. The paper's own caveat that datasets with the same mean δ are not equivalent, and the absence of intervention studies, point to potential confounds in the causal attribution, but confounding is a validity concern, not circularity. Consequently, the derivation chain is self-contained and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the LLC as a complexity measure, the OV/QK circuit decomposition, the approximate gradient argument, and the assumption that dataset manipulations isolate δ effects. These are domain assumptions from prior work and are acknowledged in the paper.

free parameters (3)
  • Linear fit for δ·∇QK = slope 0.004, intercept 0.468
    Fit to the data in Fig. 6 to emphasize approximate constancy of δ·∇QK; the nonzero slope indicates the product is not exactly constant.
  • Dataset pruning threshold = 70%
    Hand-chosen threshold in the iterative removal of high-δ lists to create datasets with specific mean δ (Sec. 2.1).
  • Perturbation swap probability divisor = 40%
    Hand-chosen constant in the perturbed dataset construction (App. C.3).
assumptions (5)
  • domain assumption The Local Learning Coefficient (LLC) measures model complexity and its relative ordering across training stages is trustworthy.
    The paper relies on LLC estimates from SLT (App. A) and cites Lau et al. (2023) for the relative-ordering claim; models are not at exact local minima, which the paper acknowledges.
  • domain assumption OV and QK circuit decomposition captures the functional behavior of attention heads.
    Standard in mechanistic interpretability (Elhage et al. 2021), invoked in Sec. 2.2.
  • ad hoc to paper Active regions in the QK circuit can be defined unambiguously and row gradients within regions approximate the attention behavior.
    The region definition is partly arbitrary (footnote 5) and the linear-gradient approximation is stated in Sec. 4.2.
  • domain assumption Dataset variations with different generation methods are comparable for attributing effects to δ.
    The paper relies on this despite acknowledging that datasets with the same mean δ are not equivalent (Sec. 2.1); the causal claim depends on δ statistics rather than confounds.
  • domain assumption The universality hypothesis: patterns found in this toy model are relevant to larger transformers.
    Motivates the study (Sec. 1) but is not established; the paper acknowledges generalization is uncertain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structure Development in List-Sorting Transformers." pith.science (2026). https://pith.science/paper/3Y6ZLO5X

@misc{pith2026250118666,
  author       = {Pith},
  title        = {Pith review of: Structure Development in List-Sorting Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Y6ZLO5X}},
  note         = {Machine review of arXiv:2501.18666}
}
read the original abstract

We study how a one-layer attention-only transformer develops relevant structures while learning to sort lists of numbers. At the end of training, the model organizes its attention heads in two main modes that we refer to as vocabulary-splitting and copy-suppression. Both represent simpler modes than having multiple heads handle overlapping ranges of numbers. Interestingly, vocabulary-splitting is present regardless of whether we use weight decay, a common regularization technique thought to drive simplification, supporting the thesis that neural networks naturally prefer simpler solutions. We relate copy-suppression to a mechanism in GPT-2 and investigate its functional role in our model. Guided by insights from a developmental analysis of the model, we identify features in the training data that drive the model's final acquired solution. This provides a concrete example of how the training data shape the internal organization of transformers, paving the way for future studies that could help us better understand how LLMs develop their internal structures.

Figures

Figures reproduced from arXiv: 2501.18666 by the authors.

Figure 1
Figure 1. Illustration of the transformer architecture and an idealized version of the sorting circuits based [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Baseline 2-head model trained on Dδ≈4.7 undergoes three stages characterized by: rapid learning (left), heads copying partly overlapping vocabularies, as can be seen from the diagonal OV circuits (middle), and vocabulary-splitting head specialization with diagonal OV circuits covering contiguous regions (right). The loss on Dd δ≈2.2 measures out-of-distribution loss on lists with closer elements. • The Circuit Rank … view at source ↗
Figure 3
Figure 3. 2-head model trained on Dd δ≈2.2 (the baseline model is trained on Dδ≈4.7 ). Initially, it evolves similar to the baseline model( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Distribution of δ (separation between neighbouring list elements) in Dd δ≈2.2 and Dδ≈4.7 . 1. For the first hundred steps, the models rapidly learn to sort, and we refer to this stage as Initial Learning. The loss decreases steeply on both Dδ≈4.7 and on Dd δ≈2.2 . The …
Figure 5
Figure 5. Figure 5: Illustration of the active regions in the QK circuits at the end of training for the baseline 2-head model trained on Dδ≈4.7 . The attention pattern in regions close to the diagonal diminishes from left to right for each row, establishing a gradient. We hypothesize: th…
Figure 6
Figure 6. Figure 6: (Left) The mean QK gradient of the active regions in the QK circuit ∇ˆ QK decreases proportionally with δ. To emphasize the flatness of the slope, we fit a linear line across all points with δ · ∇ˆ QK = 0.004 × δ + 0.468. (Right) The mean region size increases with δ. …
Figure 7
Figure 7. Figure 7: 1-head model trained on Dδ≈4.7 undergoes three stages characterized by: rapid learning (left), QK and OV circuits develop the expected patterns (middle) and off-diagonal patterns appearing in the OV circuit (right). The loss is evaluated on Dδ≈4.7 and Dd δ≈2.2 . in the…
Figure 8
Figure 8. Figure 8: Both OV and QK circuits of the baseline 2-head model trained on Dδ≈4.7 during the developmental stages. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Both OV and QK circuits of the baseline 2-head model trained on Dd δ≈2.2 during the developmental stages. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Head ablation loss and accuracy of the model trained on [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: 3-head model trained on Dδ≈4.7 . As the model learns how to sort (top left), at LLC peak (top right), three-way vocabulary-splitting after LLC decrease (bottom left) and head 3 performing copy￾suppression (bottom right). The loss is evaluated on Dδ≈4.7 and Dd δ≈2.2 . …
Figure 13
Figure 13. Figure 13: Early in training, at steps 71-348 (top row) the model goes through a transition in which The Circuit [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 12
Figure 12. Figure 12: 4-head model trained on Dδ≈4.7 . As the model learns how to sort (1st row), as the LLC decreases and heads specialize differently (2nd row), as heads 3 and 4 cover the same vocabulary regions (3rd row), as head 3 covers the entire range (4th row), and at the end of tr…
Figure 13
Figure 13. Figure 13: Baseline 2-head model trained without LN trained on Dδ≈4.7 . As the model simplifies but performs poorly (1st row), as relevant structure develops and performance improves rapidly (2nd row), as vocabulary-splitting appears before and after LLC decrease (3rd row). The …
Figure 14
Figure 14. Figure 14: Baseline 2-head model trained without WD trained on Dδ≈4.7 . As the model learns how to sort (upper left), as the LLC is at its peak (upper right), after the LLC drop (lower left) and at the end of training (lower right). The loss is evaluated on Dδ≈4.7 and Dd δ≈2.2 .…
Figure 15
Figure 15. Figure 15: Baseline 2-head model without LN and WD trained on Dδ≈4.7 . As the loss starts to drop (upper left), as loss is low and LLC peaks (upper right), after LLC drop (lower left) and at the end of training (lower right). The loss is evaluated on Dδ≈4.7 and Dd δ≈2.2 . 28 [P…
Figure 16
Figure 16. Figure 16: Distribution of relative weight norms for model heads at the end of training across different [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Baseline 2-head model with vocabulary size increased to 202, we find similar developmental stages as in the baseline model. Vocabulary region size increases. The model is trained on and the loss is evaluated on Dv=202 δ≈18.3 . 31 [PITH_FULL_IMAGE:figures/full_fig_p03…
Figure 18
Figure 18. Figure 18: Baseline 2-head model with list length increased to 20, we find similar developmental stages as in the baseline model, but without the copy suppression. The model is trained and the loss is evaluated on Dℓ=20 δ≈2.5 . 33 [PITH_FULL_IMAGE:figures/full_fig_p033_18.png]
Figure 19
Figure 19. Figure 19: Baseline 2-head model trained on a perturbed version of Dδ≈4.7 . The panels show different developmental stages and it is the only 2-head model where we observe copy-suppression. The loss is evaluated on Dδ≈4.7 and Dd δ≈2.2 . 34 [PITH_FULL_IMAGE:figures/full_fig_p034…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Influence Dynamics and Stagewise Data Attribution

    cs.LG 2025-10 conditional novelty 7.0 of 10

    Using Bayesian influence functions and singular learning theory, the authors show that a sample's influence on a model varies non-monotonically over training, peaking and flipping sign at phase transitions.

Reference graph

Works this paper leans on

25 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [3]

    Bilal Chughtai, Lawrence Chan, and Neel Nanda

    URLhttps://arxiv.org/abs/2310.06301. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networkslearngroupoperations. In International Conference on Machine Learning, pp.6243–6267.PMLR,

    arXiv

  2. [4]

    URLhttps://arxiv.org/abs/2309.08600. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kapl...

    arXiv

  3. [8]

    Michael Hanna, Ollie Liu, and Alexandre Variengien

    URLhttps://arxiv.org/abs/2312.09230. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,

    arXiv

  4. [9]

    Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet

    URLhttps://arxiv.org/abs/2305.00586. Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning,

    arXiv

  5. [10]

    Edmund Lau, Daniel Murfet, and Susan Wei

    URLhttps://arxiv.org/abs/2402.02364. Edmund Lau, Daniel Murfet, and Susan Wei. Quantifying degeneracy in singular models via the learning coefficient,

    arXiv

  6. [11]

    Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft

    URL https://arxiv.org/abs/2308.12108. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations?,

    arXiv

  7. [12]

    16 Ilya Loshchilov and Frank Hutter

    URLhttps://arxiv.org/abs/1511.07543. 16 Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations,

    arXiv

  8. [13]

    Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg

    URLhttps://arxiv.org/abs/2310.04625. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations,

    arXiv

Show all 25 references
  1. [14]

    Neel Nanda and Joseph Bloom

    URLhttps://arxiv.org/abs/2307.15771. Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens,

    arXiv

  2. [15]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter

    URLhttps://arxiv.org/abs/2301.05217. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill,

    arXiv

  3. [17]

    Nina Panickssery and Dmitry Vaintrob

    https://transformer-circuits.pub/2022/in-context- learning-and-induction-heads/index.html. Nina Panickssery and Dmitry Vaintrob. Investigating the learning coefficient of modular addition: hackathon project. https://lesswrong.com, October

    2022

  4. [18]

    Maartje E.J

    URLhttps://arxiv.org/abs/2201.02177. Maartje E.J. Raijmakers, Sylvester van Koten, and Peter C.M. Molenaar. On the validity of simulating stagewise development by means of pdp networks: Application of catastrophe analysis and an experimental test of rule-like network performan...

    arXiv

  5. [20]

    Stan van Wingerden, Jesse Hoogland, and George Wang

    URL https: //transformer-circuits.pub/2024/scaling-monosemanticity/index.html. Stan van Wingerden, Jesse Hoogland, and George Wang. Devinterp. https://github.com/ timaeus-research/devinterp,

    2024

  6. [22]

    org/abs/2410.02984

    URLhttps://arxiv. org/abs/2410.02984. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small,

    arXiv

  7. [23]

    Sumio Watanabe

    URLhttps://arxiv.org/ abs/2211.00593. Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory . Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press,

    arXiv

  8. [133]

    The LLC still has a large drop between steps 1985 and 10066 during which the heads specialize into splitting the vocabulary, and the loss decreases further

    Compared to the baseline model, the OV and QK circuits seem more noisy, and there is no drop in the Circuit Rank. The LLC still has a large drop between steps 1985 and 10066 during which the heads specialize into splitting the vocabulary, and the loss decreases further. This s...

    1985

  9. [1996]

    doi: https: //doi.org/10.1016/S0364-0213(99)80004-4

    ISSN 0364-0213. doi: https: //doi.org/10.1016/S0364-0213(99)80004-4. URL https://www.sciencedirect.com/science/article/ pii/S0364021399800044. Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. arXiv preprint arXiv:2312.06581 ,

    arXiv

  10. [2009]

    The LLC is defined in Definition 1 of Lau et al

    A Singular Learning Theory and the Local Learning Coefficient Our main tool for studying model development is the Local Learning Coefficient (LLC), a theoretically well- motivated measure of model complexity based on the learning coefficient from Singular Learning Theory (SLT)...

    2023

  11. [2016]

    Mateusz Bagiński and Gabin Kolly

    URL https: //arxiv.org/abs/1607.06450. Mateusz Bagiński and Gabin Kolly. One attention head is all you need for sorting fixed-length lists. https://apartresearch.com, January

    arXiv

  12. [2017]

    George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet

    URLhttps://arxiv.org/abs/1706.03762. George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient,

    arXiv

  13. [2020]

    https://distill.pub/2020/circuits/zoom-in

    doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Dan...

    2020 doi

  14. [2021]

    https://transformer- circuits.pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Mar...

    2021

  15. [2022]

    Zach Furman and Edmund Lau

    URL https://arxiv.org/abs/2209.10652. Zach Furman and Edmund Lau. Estimating the local learning coefficient at scale,

    arXiv

  16. [2023]

    Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra

    https://transformer-circuits.pub/2023/monosemantic-features/index.html. Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. InThe Twelfth Internation...

    2023

  17. [2024]

    Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy

    URL https: //arxiv.org/abs/2402.03698. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild,

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.