REVIEW 1 major objections 1 minor 41 references
On Mechanistic Circuits for Extractive Question-Answering
T0 review · 1 major / 1 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that a small set of attention heads in a language model's context-faithfulness circuit performs reliable data attribution by default, and that a single such head suffices for state-of-the-art attribution in extractive QA.
desk verdict Useful zero-cost attribution method; the memory circuit story rests on an unverified probe and needs fixing before the mechanistic claims are trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is causal mediation analysis via activation patching: for each internal component (attention layer, attention head, or MLP), the paper corrupts the input, copies the corrupted activation into the clean model, and measures the drop in answer-token probability; greedy selection of the smallest set achieving a score threshold yields the circuit. The probe dataset has two partitions, Dcopy (context answer replaced by a semantically similar token, forcing context answering) and Dmemory (answer replaced by an unrelated placeholder), to elicit the context-faithfulness and memory-faithfulness circuits respectively. The attribution head is then characterized by low attention entropy over the context window.
What would settle it
Run the memory-probe examples through the model and record the generated tokens: if the model outputs the placeholder token or refuses instead of producing the original answer, then the memory-faithfulness circuit is not actually the circuit for parametric-memory answering, and the steering experiments built on it collapse. A second check: ablate the single attribution head used by ATTNATTRIB and verify that extractive QA accuracy drops substantially on held-out contexts, not just on the probe set.
Extended reading notes
Core claim
The central claim is that an extractive question-answering model that answers from the provided context relies on a compact circuit of about ten attention heads, whereas answering from parametric memory engages many more heads plus MLPs, with minimal overlap between the two sets. Within the context-faithfulness circuit, a few attention heads have low-entropy attention maps that concentrate on the answer span in the context, so their attention patterns constitute a causal data attribution by default. The paper packages this as ATTNATTRIB, which uses a single such head to rank context spans for each generated token and matches or beats attribution baselines across synthetic, NQ-Swap, Natural-Questions, and single-hop HotPotQA benchmarks without any extra forward pass or auxiliary model.
Load-bearing premise
The load-bearing premise is that the probe set meant to force memory-based answering actually works, so the model answers from parametric memory rather than copying the placeholder or refusing; the paper asserts this but reports no behavioral check, filter, or accuracy on that set.
Editorial extensions
If this is right
- Data attribution for extractive QA can be obtained during the ordinary forward pass, with no auxiliary model, gradient computation, or second prompting round.
- The context-faithfulness circuit generalizes across extractive QA benchmark datasets and across knowledge types (country, capital, language), so a circuit extracted on one probe set applies to new questions.
- Steering toward context faithfulness is possible by upweighting attention from the identified heads or by adding their attributions to the prompt, improving extractive QA accuracy by up to 9%.
- Because memory answering uses a much larger, partly distinct circuit, interventions that suppress memory-faithful heads or MLPs push the model toward the context path, offering a mechanism-level handle on hallucination.
Reading between the lines
- Extension: the same one-forward-pass attribution could be repurposed as a faithfulness or hallucination detector, flagging generations whose ATTNATTRIB spans have low attention mass.
- Extension: the circuit's separation suggests a testable hypothesis for retrieval-augmented generation: fine-tuning or prompting that strengthens the context-faithfulness heads might reduce parametric interference on conflicting-context benchmarks without explicit supervision.
- Extension: the paper does not check whether Dmemory examples are truly answered from memory; a reader should treat the memory-circuit findings as conditional on that unverified assumption until a behavioral pass-rate is reported or measured.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper extracts mechanistic circuits for extractive question-answering in Vicuna-7B, Llama-3-8B, Phi-3, and Llama-3-70B using activation patching on two probe datasets, Dcopy and Dmemory. It claims that context-faithful answering is driven by a small set of ~10 attention heads, that memory-based answering uses many more heads and relies more on MLPs, that a low-entropy attention head in the context circuit performs data attribution by default, and that the resulting ATTNATTRIB algorithm obtains state-of-the-art attribution results on extractive QA benchmarks without extra forward passes. It further proposes using ATTNATTRIB attributions as a prompt signal to steer models toward context faithfulness, reporting improvements of up to 9%. The paper's central mechanistic claims rest on the unverified assumption that Dmemory actually elicits parametric-memory answering.
Significance. If the results hold, this is a valuable step: it applies circuit analysis to a realistic language-modeling task and demonstrates two practical applications, one-forward-pass attribution and context-faithfulness steering. The ATTNATTRIB method is simple, inexpensive, evaluated across several benchmarks and model families, and its attribution head is selected on a probe set and then evaluated on held-out data. The scaling of circuit extraction to Llama-3-70B is also noteworthy. The main weakness is that the memory-faithfulness circuit, which underlies the paper's central mechanistic distinction and the steering motivation, is built on a probe dataset whose intended behavior is never verified. The paper also lacks error bars and significance tests, and the 'state-of-the-art' claim is made against a narrow baseline set. On balance the contribution is promising but the load-bearing memory circuit claim needs additional behavioral evidence.
major comments (1)
- [4.2] The 'state-of-the-art' claim is made against a limited set of baselines: self-attribution prompting, iterative prompting, sentence similarity with an auxiliary encoder, and input-gradient attribution. Recent attribution and grounding methods specifically designed for extractive QA (e.g., citation-based generation, RAG-specific attribution frameworks, or other attention-based span methods) are not compared. Please either broaden the baseline set or soften the claim to 'strong performance against the considered baselines' so that the conclusion is commensurate with the evidence.
minor comments (1)
- [Appendix P] The comparison-based reasoning result reports an attribution F1 of 0.14 when restricted to Yes/No answers, which is very low; the discussion attributes this to the format but should also acknowledge that this is a substantial limitation for such reasoning questions.
Circularity Check
Partial circularity: the ATTNATTRIB head is selected on Dcopy for answer-span attention and then re-scored on the same probe set (Synthetic I/II); the memory circuit's label rests on an unvalidated Dmemory construction.
-
fitted input called prediction
[Sec. 3.3.3; Sec. 4.2; Appendix G (Synthetic 1 and Synthetic 2)]
"we observe that a small subset of attention heads in the extracted circuit for context faithfulness achieves a low entropy score with respect to the normalized attention values over the context. Upon further inspection, we find that these low-entropy attention heads predominantly focus on the answer token spans in the context. ... Synthetic 1: Consists of the probe dataset D where the context is the one generated by Llama-3-70B."
The head used by ATTNATTRIB is selected on Dcopy for low entropy and for peaking on the answer-token span, and the algorithm returns exactly the max-attention span from that head. Scoring on Synthetic 1 (Dcopy) and Synthetic 2 (the Dcopy variant with the answer token replaced) therefore re-measures the selection criterion rather than testing an independent prediction. The NQ-Swap, Natural-Questions, and HotPotQA evaluations are on held-out data and provide genuine independent evidence, so the circularity is partial and does not collapse the main attribution claim.
full rationale
The central ATTNATTRIB claim has independent content: the single attention head is selected on the Dcopy probe, and the method is then evaluated on unseen NQ-Swap, Natural-Questions, and HotPotQA benchmarks, where it outperforms baselines. The one construction-level circularity is the inclusion of the same probe set (Synthetic 1 and Synthetic 2) among the attribution benchmarks, since the head's selection criterion (low-entropy attention peaking on answer spans) is nearly identical to the max-attention-span scoring used there. There is no load-bearing self-citation chain: the circuit-extraction method cites standard prior work (Wang et al., Pearl, Zhang and Nanda) and is not justified by the authors' own prior results. Separately, the memory-faithfulness circuit rests on Dmemory's asserted memory-forcing behavior without behavioral verification, and the Appendix M.1 prompt instructs strict context adherence; this is a validation or labeling gap rather than an equation-level circular reduction, so it does not raise the circularity score further. Overall, the core ATTNATTRIB contribution retains independent empirical content, so the circularity is partial and confined to the probe-set evaluation.
Assumptions & free parameters
free parameters (4)
- delta (circuit selection threshold) =
not reported
- beta (attention upweighting scalar) =
10
- slength (attribution span length) =
not reported
- k (number of attribution spans) =
not reported
assumptions (5)
- standard math Logit representation decomposes as a sum over attention heads and MLP contributions (Elhage et al., 2021).
- domain assumption Patching one node's activation while restoring all other nodes isolates that node's direct causal effect.
- domain assumption The probe context in Dcopy forces the model to answer only from context.
- domain assumption The probe context in Dmemory forces the model to answer from parametric memory.
- domain assumption The attention head with lowest entropy in the context window is the one that performs attribution.
Cite this review
Pith. "Pith review of On Mechanistic Circuits for Extractive Question-Answering." pith.science (2026). https://pith.science/paper/5ROWYYBM
@misc{pith2026250208059,
author = {Pith},
title = {Pith review of: On Mechanistic Circuits for Extractive Question-Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/5ROWYYBM}},
note = {Machine review of arXiv:2502.08059}
}
read the original abstract
Large language models are increasingly used to process documents and facilitate question-answering on them. In our paper, we extract mechanistic circuits for this real-world language modeling task: context-augmented language modeling for extractive question-answering (QA) tasks and understand the potential benefits of circuits towards downstream applications such as data attribution to context information. We extract circuits as a function of internal model components (e.g., attention heads, MLPs) using causal mediation analysis techniques. Leveraging the extracted circuits, we first understand the interplay between the model's usage of parametric memory and retrieved context towards a better mechanistic understanding of context-augmented language models. We then identify a small set of attention heads in our circuit which performs reliable data attribution by default, thereby obtaining attribution for free in just the model's forward pass. Using this insight, we then introduce ATTNATTRIB, a fast data attribution algorithm which obtains state-of-the-art attribution results across various extractive QA benchmarks. Finally, we show the possibility to steer the language model towards answering from the context, instead of the parametric memory by using the attribution from ATTNATTRIB as an additional signal during the forward pass. Beyond mechanistic understanding, our paper provides tangible applications of circuits in the form of reliable data attribution and model steering.
Figures
Figures from the paper (16 more)
Reference graph
Works this paper leans on
-
[3]
URL https://arxiv. org/abs/2404.14082. Buchmann, J., Liu, X., and Gurevych, I. Attribute or abstain: Large language models as long document assistants,
-
[4]
URL https://arxiv.org/abs/2407.07799. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y ., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y ., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March
-
[5]
URL https://lmsys.org/blog/ 2023-03-30-vicuna/ . Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y ., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah...
work page 2023
-
[6]
Gao, T., Yen, H., Yu, J., and Chen, D
URL https://arxiv.org/abs/2407.21783. Gao, T., Yen, H., Yu, J., and Chen, D. Enabling large language models to generate text with citations,
-
[7]
Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., and Wang, H
URL https://arxiv.org/abs/2305.14627. Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., and Wang, H. Retrieval-augmented generation for large language models: A survey,
-
[8]
Gould, R., Ong, E., Ogden, G., and Conmy, A
URL https://arxiv.org/abs/2312.10997. Gould, R., Ong, E., Ogden, G., and Conmy, A. Succes- sor heads: Recurring, interpretable attention heads in the wild,
-
[9]
Hanna, M., Liu, O., and Variengien, A
URL https://arxiv.org/abs/ 2312.09230. Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abili- ties in a pre-trained language model,
-
[10]
URL https: //arxiv.org/abs/2305.00586. Hermann, K. M., Kocisk ´y, T., Grefenstette, E., Espe- holt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. CoRR, abs/1506.03340,
Show all 41 references
-
[11]
We provide the patching steps as follows: Step 1: Copy the activation of a node (e.g., a12) from the corrupted model to the clean model to create the patched model
Different Steps of Patching with the Clean and Corrupted Model. We provide the patching steps as follows: Step 1: Copy the activation of a node (e.g., a12) from the corrupted model to the clean model to create the patched model. Step 2: Patching a12 also affects a13, m13 and m...
1966
-
[12]
Jiang, A
URL https://arxiv.org/abs/2307.02185. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and ...
-
[13]
Khalifa, M., Wadden, D., Strubell, E., Lee, H., Wang, L., Beltagy, I., and Peng, H
URL https: //arxiv.org/abs/2310.06825. Khalifa, M., Wadden, D., Strubell, E., Lee, H., Wang, L., Beltagy, I., and Peng, H. Source-aware training enables knowledge attribution in language models,
-
[14]
URL https://arxiv.org/abs/2404.01019. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., De- vlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Na...
-
[15]
Bob ’s your uncle
Removing the attributions obtained with ATTN ATTRIB from the context leads to a large relative change in the log probability of the responses. We measure the relative change in the log probabilities of the original response (with the original context and context where the attr...
2010
-
[16]
Lieberum, T., Rahtz, M., Kram´ar, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V
URL https://arxiv.org/abs/ 2311.03731. Lieberum, T., Rahtz, M., Kram´ar, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V . Does circuit analysis inter- pretability scale? evidence from multiple choice capabili- ties in chinchilla,
-
[17]
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., and Singh, S
URL https://arxiv.org/ abs/2307.09458. Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., and Singh, S. Entity-based knowledge conflicts in question answering,
-
[18]
Across various extractive QA benchmarks, we obtain improved performances over different attribution baselines
Attribution through one attention head in our circuit via ATTN ATTRIB obtains strong attribution results. Across various extractive QA benchmarks, we obtain improved performances over different attribution baselines. For HotPotQA, we measure the F1-score due to it being single...
2024
-
[19]
McDougall, C., Conmy, A., Rushing, C., McGrath, T., and Nanda, N
URL https://arxiv.org/abs/ 2212.10511. McDougall, C., Conmy, A., Rushing, C., McGrath, T., and Nanda, N. Copy suppression: Comprehensively understanding an attention head,
-
[20]
Meng, K., Bau, D., Andonian, A., and Belinkov, Y
URL https: //arxiv.org/abs/2310.04625. Meng, K., Bau, D., Andonian, A., and Belinkov, Y . Locat- ing and editing factual associations in gpt,
-
[21]
Niu, C., Wu, Y ., Zhu, J., Xu, S., Shum, K., Zhong, R., Song, J., and Zhang, T
URL https://arxiv.org/abs/2202.05262. Niu, C., Wu, Y ., Zhu, J., Xu, S., Shum, K., Zhong, R., Song, J., and Zhang, T. Ragtruth: A hallucination corpus for de- veloping trustworthy retrieval-augmented language mod- els,
-
[22]
Shi, W., Han, X., Lewis, M., Tsvetkov, Y ., Zettlemoyer, L., and tau Yih, S
URL https:// arxiv.org/abs/2402.14811. Shi, W., Han, X., Lewis, M., Tsvetkov, Y ., Zettlemoyer, L., and tau Yih, S. W. Trusting your evidence: Hallucinate less with context-aware decoding,
-
[23]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C
URL https: //arxiv.org/abs/2305.14739. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Ful...
-
[24]
Turner, A
URL https://arxiv.org/abs/2307.09288. Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization,
-
[25]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A
URL https://arxiv.org/abs/2308.10248. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR, abs/1706.03762,
-
[27]
Wu, K., Wu, E., and Zou, J
URL https://arxiv.org/abs/2211.00593. Wu, K., Wu, E., and Zou, J. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence,
-
[28]
Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y ., and Xu, W
URL https://arxiv.org/abs/ 2404.10198. Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y ., and Xu, W. Knowledge conflicts for llms: A survey, 2024a. URL https://arxiv.org/abs/2403.08319. Xu, S., Pang, L., Shen, H., Cheng, X., and Chua, T.- S. Search-in-the-chain: Interact...
-
[30]
org/abs/2311.09533
URL https://arxiv. org/abs/2311.09533. 10 On Mechanistic Circuits for Extractive Question-Answering Yin, K. and Neubig, G. Interpreting language models with contrastive explanations,
-
[31]
org/abs/2202.10419
URL https://arxiv. org/abs/2202.10419. Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods,
-
[32]
Zhang, S., Pan, L., Zhao, J., and Wang, W
URL https://arxiv.org/abs/2309.16042. Zhang, S., Pan, L., Zhao, J., and Wang, W. Y . The knowl- edge alignment problem: Bridging human and exter- nal knowledge for large language models,
-
[33]
Zheng, C., Yin, F., Zhou, H., Meng, F., Zhou, J., Chang, K.-W., Huang, M., and Peng, N
URL https://arxiv.org/abs/2305.13669. Zheng, C., Yin, F., Zhou, H., Meng, F., Zhou, J., Chang, K.-W., Huang, M., and Peng, N. On prompt-driven safeguarding for large language models,
-
[34]
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M
URL https://arxiv.org/abs/2401.18018. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendry...
-
[35]
11 On Mechanistic Circuits for Extractive Question-Answering A
URL https://arxiv.org/abs/2310.01405. 11 On Mechanistic Circuits for Extractive Question-Answering A. Qualitative Examples on Data Attribution A.1. Vicuna Figure
-
[36]
use a similar smaller size probe dataset to find a circuit for entity tracking. E. More Details on the Interventional Algorithm © 2024 Adobe. All Rights Reserved. Adobe Confidential. Steps for Obtaining the Circuit Components Logit a13 a12 m13 m12 a11 m11 a0 m0 ……. Logit a13 a...
2024
-
[38]
17 On Mechanistic Circuits for Extractive Question-Answering H
with zero-hop or single-hop extractive QA questions. 17 On Mechanistic Circuits for Extractive Question-Answering H. Qualitative Study of Attributions using AttnAttribute Question Context Attribution via Attention Head GT Attribution who won the icc under 19 world cup 2018 Ans...
2018
-
[39]
In this example, Russia (which is the answer) is present at multiple places
ATTN ATTRIB can select the right attribution span containing the answer, even if the answer token is present at multiple locations. In this example, Russia (which is the answer) is present at multiple places. We find that ATTNATTRIB can infact pick out the correct causal locat...
1901
-
[2015]
Huang, J
URL http://arxiv.org/ abs/1506.03340. Huang, J. and Chang, K. C.-C. Citation: A key to building responsible and accountable large language models,
-
[2017]
Wang, F., Mo, W., Wang, Y ., Zhou, W., and Chen, M
URL http://arxiv.org/abs/1706.03762. Wang, F., Mo, W., Wang, Y ., Zhou, W., and Chen, M. A causal view of entity bias in (large) language mod- els,
-
[2018]
Ye, X., Sun, R., Arik, S
URL http: //arxiv.org/abs/1809.09600. Ye, X., Sun, R., Arik, S. O., and Pfister, T. Effective large language model adaptation for improved grounding and citation generation,
-
[2021]
URL https://arxiv.org/abs/2005. 11401. 9 On Mechanistic Circuits for Extractive Question-Answering Li, D., Sun, Z., Hu, X., Liu, Z., Chen, Z., Hu, B., Wu, A., and Zhang, M. A survey of large language models attribution,
2005
-
[2022]
org/abs/2109.05052
URL https://arxiv. org/abs/2109.05052. Mallen, A., Asai, A., Zhong, V ., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Inves- tigating effectiveness of parametric and non-parametric memories,
-
[2023]
Bereska, L
URL https://arxiv.org/ abs/2310.11511. Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety – a review,
-
[2024]
Asai, A., Wu, Z., Wang, Y ., Sil, A., and Hajishirzi, H
URL https://arxiv.org/abs/2406.11717. Asai, A., Wu, Z., Wang, Y ., Sil, A., and Hajishirzi, H. Self- rag: Learning to retrieve, generate, and critique through self-reflection,
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.