Pith. sign in

REVIEW 3 major objections 5 minor 32 references

The paper claims that a one-shot, pre-fine-tuning rank allocation computed from a task-conditioned interaction graph of a frozen model outperforms both uniform LoRA ranks and training-time adaptive rank methods under the same total rank bud

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:19 UTC pith:3QCN6H5J

load-bearing objection Novel and plausible one-shot pre-fine-tuning rank allocation via zero-ablation tracing, but the abstract overclaims and the central transfer assumption is not directly validated. the 3 major comments →

arxiv 2607.22251 v1 pith:3QCN6H5J submitted 2026-07-24 cs.LG cs.AI

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

Wei Zhang , Xinwu Liu , Yihang Cheng This is my paper
classification cs.LG cs.AI
keywords parameter-efficient fine-tuningLoRArank allocationinformation-flow centralityinteraction graphmechanistic interpretabilitylow-rank adaptationtask-conditioned topology
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

IFCLoRA tries to answer a practical question: given a fixed low-rank budget for fine-tuning a large language model, which Transformer modules deserve the most capacity? Its claim is that the answer can be read off before training from a small calibration set, by building a task-conditioned interaction graph of LoRA-insertable modules on the frozen model and scoring each module by information-flow centrality—how strongly it lies on paths from the input side to the output side of the task. Ranks are then assigned once, and training proceeds exactly like standard LoRA with non-uniform ranks. The paper reports consistent gains over uniform LoRA, AdaLoRA, and EVA across three 8B/14B models on GSM8K and SuperGLUE at ranks 4 and 8, arguing that global topology is a more reliable allocation signal than local training-time statistics. If true, this would make low-budget PEFT cheaper and its capacity placement interpretable.

Core claim

A single, pre-fine-tuning pass can convert a frozen pretrained model into a task-conditioned interaction graph whose nodes are the LoRA-insertable modules (attention projections and MLP projections). Zero-ablation tracing over a small calibration set estimates directed influence weights between modules within a depth window; a power-form fusion of forward/backward source-to-sink reachability with local gradient sensitivity yields an Information-Flow Centrality (IFC) score per module; a temperature-controlled softmax maps these scores into discrete integer ranks under a global budget. The central discovery is that this one-shot topology-aware allocation—without any training-time rank search—c

What carries the argument

The central object is the Information-Flow Centrality (IFC) score, computed from a task-conditioned sparse directed interaction graph. Nodes are LoRA-insertable modules; a directed edge i→j carries the average change in node j's activation when node i is zero-ablated on calibration data. The graph is row-normalized into a transition operator, and forward reachability from the shallowest module and backward reachability from the deepest module—each truncated with damping—are pointwise multiplied and normalized to form a topology prior. That prior is fused multiplicatively with type-wise normalized local gradient sensitivity, so a module must be both structurally on a source-to-sink path and l

Load-bearing premise

The load-bearing premise is that zero-ablation influence within a 15-module depth window on the frozen model faithfully identifies the paths along which fine-tuning capacity will matter—if task-relevant information enters through the embedding or LM head, or flows through paths longer than 15 modules, the rank allocation will be systematically misplaced.

What would settle it

Take a task known to depend on a long-range interaction spanning more than 15 Transformer module positions (e.g., a synthetic copying or induction task whose critical path crosses many layers). Run IFCLoRA with w=15 and compare against a variant with w equal to full depth: if the larger window yields a materially different allocation and higher accuracy, the window assumption is falsified for that task.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Under the matched total-rank-budget protocol, IFCLoRA improves GSM8K accuracy over LoRA on all three backbones tested at both r=4 and r=8, averaging +1.13 points and reaching +1.82 points on LLaMA3-8B at r=8.
  • The allocation is task-dependent and non-uniform: mid-to-late FFN modules and selected attention projections receive higher ranks than shallow modules, and the pattern changes with the task, indicating topology, not a fixed template, drives allocation.
  • Because ranks are fixed before training, the fine-tuning stage has the same runtime and peak memory as standard LoRA; the extra cost is a one-time preprocessing pass (about 5 minutes tracing on the reported setup).
  • The ablation shows both components matter: removing the topology prior drops accuracy toward gradient-only routing, and replacing zero-ablation edges with a cheaper substitute loses the advantage over simple FFN-biased routing.
  • The approach is designed for decoder-only Transformers, where module order follows forward depth; transfer to encoder-decoder or MoE architectures is not established.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The IFC prior could be recomputed cheaply at multiple tasks and then averaged to produce a multi-task rank profile, something the paper does not test; if the gains carry over, it would make the method a drop-in router for continual or multi-task fine-tuning.
  • Inference: The zero-ablation window w=15 is a strong structural assumption: if long-range dependencies (spanning more than 15 module positions) carry the task, IFCLoRA would systematically under-allocate to bridging layers. A direct test is to vary w and watch whether gains concentrate on long edges.
  • Inference: The paper's fairness metric is total rank budget, not total trainable parameters; because modules differ in shape, equal rank does not mean equal parameter count. A stricter parameter-matched comparison might shrink or enlarge the reported deltas, especially against AdaLoRA whose parameter count varies during training.
  • Inference: Because the graph is built on a frozen model, the prior is only as good as the assumption that the fine-tuning update follows the same routes as zero-ablation influence. Lightweight interventions during training (e.g., patching the allocated high-rank modules) could validate whether the chosen modules are indeed the bottlenecks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces IFCLoRA, a pre-fine-tuning rank-allocation method for LoRA. It uses 128 calibration samples and a frozen pretrained model to build a sparse directed interaction graph over LoRA-insertable modules, with edge weights from zero-ablation activation differences (Eq. 2-3) and edges restricted to a window w=15. It computes forward and backward reachability from one-hot anchors at the shallowest and deepest LoRA modules (Eq. 5), calibrates this topology prior with local gradient sensitivity (Eq. 6), fuses them via a power-form soft-AND (Eq. 7), and maps z-scored scores through a temperature softmax to integer ranks under a fixed total rank budget (Eq. 8). The resulting non-uniform allocation is then used for standard LoRA training. Experiments on GSM8K and SuperGLUE across LLaMA3-8B, Qwen3-8B, and Qwen3-14B at r=4 and r=8 compare against LoRA, AdaLoRA, and EVA. The paper claims consistent improvement, and reports ablations, hyperparameter sensitivity, and efficiency comparisons.

Significance. The method's distinctive contribution is a one-shot, topology-based allocation that avoids training-time rank search or dynamic pruning. If the empirical claims are robust, this is practically useful: Table 2 shows IFCLoRA best on all six GSM8K model/budget settings, with gains up to 1.82 points over LoRA; Table 8 shows training time and peak memory on par with LoRA; Figure 5 ablations support complementarity of the topology and gradient terms. The paper also ships reproducible experimental settings, seed reporting, and explicit limitations. However, the contribution is heuristic and the evidence is incomplete on two central points: the abstract's 'consistently outperforms' claim is contradicted by the paper's own SuperGLUE tables, and the graph-based prior is not validated as a measure of where LoRA rank matters. Both issues are fixable with revised claims or additional experiments.

major comments (3)
  1. [Abstract; Table 1; Table 6] The abstract and introduction claim IFCLoRA 'consistently outperforms LoRA, AdaLoRA, and EVA' under matched budgets. This is contradicted by the paper's own tables. Examples at r=4: LLaMA3-8B RTE (Table 1) LoRA 89.97±0.23 vs IFCLoRA 89.58±0.81; Table 6 WiC LoRA 76.30±1.26 vs IFCLoRA 75.00±0.68; Qwen3-8B CB LoRA 95.80±3.40 vs IFCLoRA 94.72±2.74; Qwen3-14B COPA LoRA 100.00 vs IFCLoRA 98.33. The consistent-outperformance claim should be restricted to GSM8K, with SuperGLUE described as task-dependent and competitive.
  2. [Appendix C; Table 4] The hyperparameters ρ and τ are selected on the same benchmark used for the headline result. Appendix C reports a 5×5 grid search on Qwen3-8B GSM8K r=4 and states that the default (ρ=0.25, τ=0.8) lies in a favorable region. If the same held-out GSM8K evaluation is used for selection and final numbers, part of the reported gain may be selection rather than method. Please state the split used for selection, whether the same hyperparameters were fixed for all backbones and tasks, and whether grid-search results for the other settings are available.
  3. [§3.2-3.3; §4.4; Eq. (5); Figure 5] The central transfer assumption is unvalidated: the zero-ablation interaction graph of the frozen model is assumed to predict where LoRA rank budget is best spent during fine-tuning. Zero ablation is a large off-distribution intervention, not a low-rank perturbation; the anchors are the shallowest and deepest LoRA modules, not the actual model input/output; and the w=15 window may miss longer dependency paths. Figure 5 only compares composite routing scores against Random/Depth/FFN/PageRank baselines and does not check whether IFC's top-ranked modules coincide with independently measured per-module adaptation importance. Please add a direct validation, e.g., measure per-module rank sensitivity by one-at-a-time rank increases, vary anchors and w, or compare IFC ranks with a training-time importance estimate.
minor comments (5)
  1. [Eq. (4) and Eq. (7)] The symbol ε is reused in Eq. (4) as a denominator smoothing constant and in Eq. (7) as an additive constant inside the fusion; please use distinct symbols or state explicitly that they are the same small constant.
  2. [Figure 4(b); Section 4.3] The text says EVA is included in quantitative comparisons, but Figure 4(b) shows only LoRA, AdaLoRA, and IFCLoRA. Please clarify that EVA is omitted from the visualization and why.
  3. [Throughout] There are typographical and formatting inconsistencies: 'EV A' is written with a space, references contain 'V olume' artifacts, and Figure 1 contains garbled sample labels ('sample1 sample2 sample 128'). Please proofread.
  4. [Appendix C, Figures 6-7] The sensitivity plots report averaged validation performance but no error bars or number of seeds in the caption text. Since the main tables report mean±std over three seeds, the sensitivity analysis should follow the same protocol.
  5. [Appendix A.3] Code is not released due to double-blind review. This is understandable, but the final version should include a public repository or executable package; the detailed experiment entry points and settings are a good start.

Circularity Check

0 steps flagged

IFCLoRA's rank-allocation pipeline is a self-contained heuristic; the reported gains are held-out empirical results, so no step reduces to its own inputs.

full rationale

The method's derivation chain (Eqs. 2-8) is self-contained and empirical. The interaction graph is constructed from zero-ablation responses on a frozen model, the topology prior is computed from one-hot anchors at the first and last LoRA-insertable modules, and the IFC score is defined as a power-form fusion of that prior with local gradient sensitivity; the softmax allocation then consumes only these internally defined scores and the fixed rank budget. No equation uses the final GSM8K/SuperGLUE accuracy as an input, and no 'adaptation importance' ground truth is assumed—Section 5 explicitly states 'The IFC score is a heuristic without formal optimality guarantees.' The central claim is therefore an empirical comparison, supported by held-out test evaluations across three backbones, two rank budgets, and ablations against Random/Depth/FFN/PageRank baselines. The only mild concern is model selection: ρ, τ, and calibration size were chosen using validation performance on the same benchmarks (Appendix C) and the 128 calibration samples are drawn from the training split, so part of the advantage could reflect selection rather than the topology prior; however, the paper's own sensitivity analysis shows the default configuration lies in a robust region, and this is standard hyperparameter selection, not a definitional reduction. There is also no load-bearing self-citation: the cited mechanistic-interpretability and centrality works are external, and no uniqueness theorem or ansatz is imported from the authors' prior work.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 2 invented entities

The method rests on a chain of unproven modeling choices: zero-ablation influence in a frozen model predicts where fine-tuning capacity is best spent; node ordering + windowed edges suffice to capture information flow; first/last LoRA modules are valid anchors; a row-normalized random-walk-style propagation with damping factors is a meaningful task topology; and multiplicative fusion of topology and gradient sensitivity is a valid importance score. These are domain assumptions, not theorems; the paper acknowledges the heuristic nature in Section 5.

free parameters (8)
  • ρ (rho) = 0.25
    Fusion weight between topology prior and gradient sensitivity in Eq. 7; selected by hand/grid in Appendix C.1.
  • τ (tau) = 0.8
    Softmax temperature in Eq. 8 converting scores to rank quotas; selected by hand/grid in Appendix C.1.
  • α and β (damping factors) = 0.85
    Damping factors in forward/backward reachability, Eq. 5; chosen by hand.
  • Graph window size w = 15
    Maximum edge distance in module depth order; controls graph sparsity, Section 3.2.
  • Kf, Kb (reachability truncation depths)
    Truncation depths in Eq. 5; values not specified in the text, another hand choice.
  • min_rank / max_rank = 2 / 32
    Clipping bounds for allocated ranks, Appendix A.2.
  • Calibration size = 128 samples
    Number of calibration samples used for tracing and gradient sensitivity, Appendix A.2.
  • ε (epsilon)
    Small constant in Eqs. 4 and 7 to avoid division by zero / zero scores; numerical stabilizer.
axioms (6)
  • domain assumption Zero ablation on the frozen model estimates task-relevant causal influence between modules.
    Section 3.2, Eqs. 2–3: measuring the change in target activation after zeroing a source module's output assumes this intervention captures how modules influence each other under the task.
  • domain assumption Edges exist only from earlier to later nodes within window w=15; the interaction graph is a directed acyclic approximation of the model's computation.
    Section 3.2: "retains only candidate edges satisfying j > i and j ≤ i+w"; assumes forward-depth propagation structure of decoder-only Transformers.
  • ad hoc to paper The shallowest and deepest LoRA-insertable modules are the input-side and output-side anchors of task information flow.
    Section 3.3: "The shallowest target node is used as the input-side anchor, and the deepest target node is used as the output-side anchor"; no validation that these coincide with actual input/output flow boundaries.
  • ad hoc to paper A row-normalized transition matrix with damping factors propagates task-relevant information in a way that matches adaptation importance.
    Eq. 5: source-to-sink reachability computed via damped random-walk-style propagation is assumed to be a meaningful topology prior for rank allocation.
  • ad hoc to paper The multiplicative power-form fusion of topology prior and gradient sensitivity is a valid importance score.
    Eq. 7: I_i = (f_i+ε)^(1-ρ) (g_i+ε)^ρ; the soft-AND behavior is asserted, not derived.
  • domain assumption The task loss L and calibration data used in Eq. 6 are representative of the downstream fine-tuning objective.
    Local gradient sensitivity is computed on 128 calibration samples from the training split; assumes this small sample is representative.
invented entities (2)
  • Information-Flow Centrality (IFC) score no independent evidence
    purpose: Ranks LoRA-insertable modules by combining source-to-sink reachability with local gradient sensitivity for one-shot rank allocation.
    The score is validated only by in-paper benchmark comparisons; it makes no externally falsifiable prediction outside the reported experiments.
  • Task-conditioned interaction graph no independent evidence
    purpose: Sparse directed representation of influence among LoRA-insertable modules under a task, built via zero-ablation tracing.
    The graph structure is not directly validated against ground-truth circuits; only its downstream rank allocations are evaluated.

pith-pipeline@v1.3.0-alltime-deepseek · 20566 in / 14362 out tokens · 124090 ms · 2026-08-01T05:19:13.466715+00:00 · methodology

0 comments
read the original abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules. Existing adaptive-rank methods usually rely on local gradient statistics collected during training, which introduces extra memory and computation and overlooks task-conditioned global information flow. We propose IFCLoRA, a topology-aware rank allocation method applied before fine-tuning. Using a small calibration set and a frozen pretrained model, IFCLoRA builds a sparse task-conditioned interaction graph whose nodes represent LoRA-compatible modules. It combines a global information-flow topology prior with local gradient sensitivity to compute Information-Flow Centrality scores, which estimate each module's adaptation importance under multi-hop propagation. Ranks are then assigned once under a global budget. Across multiple models, tasks, and low-rank settings, IFCLoRA consistently outperforms LoRA, AdaLoRA, and EVA under matched training configurations and total rank budgets, while retaining training costs comparable to standard LoRA. On mathematical reasoning with LLaMA 3 8B, IFCLoRA improves over LoRA by 1.36 percent at rank 4 and 1.82 percent at rank 8. Further analysis shows task-dependent, non-uniform rank profiles, indicating that global information-flow structure provides an informative and interpretable prior for low-budget parameter-efficient fine-tuning.

Figures

Figures reproduced from arXiv: 2607.22251 by Wei Zhang, Xinwu Liu, Yihang Cheng.

Figure 1
Figure 1. Figure 1: Comparison of LoRA, AdaLoRA, and IF￾CLoRA rank-allocation strategies [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of IFCLoRA. Given a small calibration set, IFCLoRA first performs [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: GSM8K accuracy under differ￾ent rank budgets [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Rank allocation analysis on Qwen3-8B with GSM8K under the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study and routing-control baselines on Qwen3-8B (GSM8K, r = 4). We ablate the routing score on Qwen3-8B (GSM8K, r = 4), covering routing-control base￾lines (Random, Depth-biased, and FFN-biased), PageRank, graph-construction ablation (w/o zero ablation), component ablations (w/o Gradient and w/o Topology), and the full IFCLoRA score. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Validation performance under different ρ and τ settings on Qwen3-8B with GSM8K under r = 4 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance of IFCLoRA under different calibration sizes on Qwen3-8B with GSM8K [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Rank allocation patterns across backbone models under the same task and rank budget. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Rank allocation patterns across tasks using the same backbone and rank budget. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Attention Is All You Need , author=. Advances in Neural Information Processing Systems , volume=. 2017 , url=

    2017

  2. [2]

    Proceedings of the 36th International Conference on Machine Learning , pages=

    Parameter-Efficient Transfer Learning for NLP , author=. Proceedings of the 36th International Conference on Machine Learning , pages=. 2019 , volume=

    2019

  3. [3]

    Prefix-Tuning: Optimizing Continuous Prompts for Generation , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=. 2021 , address=. doi:10.18653/v1/2021.acl-long.353 , url=

    doi:10.18653/v1/2021.acl-long.353 2021

  4. [4]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    The Power of Scale for Parameter-Efficient Prompt Tuning , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=. 2021 , address=. doi:10.18653/v1/2021.emnlp-main.243 , url=

    doi:10.18653/v1/2021.emnlp-main.243 2021

  5. [5]

    2022 , address=

    Ben Zaken, Elad and Goldberg, Yoav and Ravfogel, Shauli , booktitle=. 2022 , address=. doi:10.18653/v1/2022.acl-short.1 , url=

    doi:10.18653/v1/2022.acl-short.1 2022

  6. [6]

    Transactions on Machine Learning Research , year=

    Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey , author=. Transactions on Machine Learning Research , year=

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning , author=. Advances in Neural Information Processing Systems , volume=. 2022 , url=

    2022

  8. [8]

    International Conference on Learning Representations , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations , year=

  9. [9]

    International Conference on Learning Representations , year=

    AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning , author=. International Conference on Learning Representations , year=

  10. [10]

    2024 , address=

    Liu, Zequan and Lyn, Jiawen and Zhu, Wei and Tian, Xing and Graham, Yvette , booktitle=. 2024 , address=. doi:10.18653/v1/2024.naacl-long.35 , url=

    work page doi:10.18653/v1/2024.naacl-long.35 2024

  11. [11]

    Advances in Neural Information Processing Systems , volume=

    Parameter Efficient Fine-tuning via Explained Variance Adaptation , author=. Advances in Neural Information Processing Systems , volume=. 2025 , url=

    2025

  12. [12]

    TLoRA: Task-aware Low Rank Adaptation of Large Language Models

    Lin, Weicheng and Zhang, Yi and Dang, Jiawei and Zhang, Liang-Jie , journal=. 2026 , eprint=. doi:10.48550/arXiv.2604.18124 , url=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.18124 2026

  13. [13]

    2026 , eprint=

    Cui, Xuan and Li, Huiyue and Zeng, Run and Zhao, Yunfei and Qian, Jinrui and Duan, Wei and Liu, Bo and Zhou, Zhanpeng , booktitle=. 2026 , eprint=. doi:10.48550/arXiv.2603.13792 , url=

    doi:10.48550/arxiv.2603.13792 2026

  14. [14]

    2025 , address=

    Ed-dib, Abdessalam and Datbayev, Zhanibek and Aboussalah, Amine Mohamed , booktitle=. 2025 , address=. doi:10.18653/v1/2025.findings-emnlp.1372 , url=

    work page doi:10.18653/v1/2025.findings-emnlp.1372 2025

  15. [15]

    2025 , url=

    He, Haonan and Ye, Peng and Ren, Yuchen and Yuan, Yuan and Zhou, Luyang and Ju, Shucun and Chen, Lei , booktitle=. 2025 , url=

    2025

  16. [16]

    Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025) , year=

    Singh, Raul and Brunello, Nicol. Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025) , year=

    2025

  17. [17]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  18. [18]

    , booktitle=

    Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R. , booktitle=. 2019 , url=

    2019

  19. [19]

    arXiv preprint arXiv:2407.21783 , year=

    The Llama 3 Herd of Models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 Technical Report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  21. [21]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Transformer Feed-Forward Layers Are Key-Value Memories , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=. 2021 , address=. doi:10.18653/v1/2021.emnlp-main.446 , url=

    doi:10.18653/v1/2021.emnlp-main.446 2021

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Locating and Editing Factual Associations in GPT , author=. Advances in Neural Information Processing Systems , volume=. 2022 , url=

    2022

  23. [23]

    arXiv preprint arXiv:2211.00593 , year=

    Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small , author=. arXiv preprint arXiv:2211.00593 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2304.14997 , year=

    Towards Automated Circuit Discovery for Mechanistic Interpretability , author=. arXiv preprint arXiv:2304.14997 , year=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=

    Attribution Patching Outperforms Automated Circuit Discovery , author=. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=. 2024 , address=. doi:10.18653/v1/2024.blackboxnlp-1.25 , url=

    doi:10.18653/v1/2024.blackboxnlp-1.25 2024

  26. [26]

    Transformer Circuits Thread , year=

    A Mathematical Framework for Transformer Circuits , author=. Transformer Circuits Thread , year=

  27. [27]

    Page, Lawrence and Brin, Sergey and Motwani, Rajeev and Winograd, Terry , institution=. The. 1999 , url=

    1999

  28. [28]

    2018 , publisher=

    Networks , author=. 2018 , publisher=. doi:10.1093/oso/9780198805090.001.0001 , isbn=

    arXiv 2018

  29. [29]

    Proceedings of the 34th International Conference on Machine Learning , pages=

    Axiomatic Attribution for Deep Networks , author=. Proceedings of the 34th International Conference on Machine Learning , pages=. 2017 , volume=

    2017

  30. [30]

    2026 , eprint=

    Deng, Guanzhi and Li, Bo and Chen, Ronghao and Liu, Xiujin and Han, Zhuo and Wang, Huacan and Wen, Lijie and Song, Linqi , journal=. 2026 , eprint=. doi:10.48550/arXiv.2601.04823 , url=

    doi:10.48550/arxiv.2601.04823 2026

  31. [31]

    International Conference on Learning Representations , year=

    Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guarantees , author=. International Conference on Learning Representations , year=. doi:10.48550/arXiv.2602.16823 , url=. 2602.16823 , archivePrefix=

    work page doi:10.48550/arxiv.2602.16823

  32. [32]

    International Conference on Learning Representations , year=

    Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility , author=. International Conference on Learning Representations , year=. doi:10.48550/arXiv.2510.03358 , url=. 2510.03358 , archivePrefix=

    doi:10.48550/arxiv.2510.03358