{"id":"3fafefc5-c92c-4a07-acd4-fc931430a9d2","arxiv_id":"2605.26343","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A single RL policy trained on twelve behaviors in GPT-2 small transfers to unseen behaviors and recovers their known circuits without retraining from scratch.","lead":"This paper trains a reinforcement learning policy to discover circuits in GPT-2 small by sequentially removing edges from the computation graph while preserving behavior through causal interventions. If the transfer result holds, it could turn circuit discovery from a repeated per-behavior search into a learned, reusable procedure.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Faithfulness reward via causal intervention may be satisfied by multiple non-unique subgraphs, so RL policy need not recover the intended mechanistic circuit.","rationale":"The reader's weakest_assumption directly identifies the same point: the causal-intervention reward is the only training signal, and nothing in the abstract or claim description demonstrates it selects the unique or canonical circuit rather than any reward-equivalent subgraph.","tokens_in":1687,"tokens_out":388,"duration_ms":33770,"concrete_test":"For each of the twelve training behaviors and the held-out transfer behaviors, run the published RL policy to obtain its edge set, then independently run ACDC (or EAP) on the identical GPT-2 small model, tasks, and patching metric; compute edge-set overlap (Jaccard index or precision/recall treating reference edges as ground truth) and verify that both achieve comparable faithfulness scores. If overlap < 0.6 while faithfulness differs by <5%, the reward does not uniquely target the known circuit.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the RL policy, guided solely by a faithfulness reward computed through causal interventions (e.g., ablation or activation patching on the remaining graph), converges to the sparse subgraph that prior work has identified as the true circuit. This assumption is load-bearing because faithfulness is a many-to-one property: any subgraph preserving the necessary information flow for the target behavior under the chosen interventions will receive high reward, even if it includes redundant edges, alternative pathways, or non-minimal structure. The paper trains on twelve behaviors and evaluates zero-shot transfer on unseen ones by checking recovery of \"known circuits,\" but without an explicit uniqueness or minimality term in the reward, the policy could exploit reward-equivalent solutions that differ from the reference circuits. This risk is heightened in a transformer computation graph where multiple edge sets can produce statistically indistinguishable outputs after patching.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces MechRL, framing circuit discovery in mechanistic interpretability as a reinforcement learning problem over the computation graph of GPT-2 small. A policy learns to remove edges to reach a compact subgraph that preserves behavior, guided by a faithfulness reward from causal interventions. The central claim is that a single policy trained across twelve behaviors recovers faithful circuits for each; once frozen, it transfers zero-shot to unseen behaviors by recovering their known circuits, with a short warm-start yielding smaller circuits than training from scratch.","tokens_in":1868,"tokens_out":435,"duration_ms":20931,"significance":"If the transfer result is robustly supported, the work would be significant for demonstrating that circuit discovery can be amortized as a learnable, transferable procedure rather than repeated per-behavior searches. The approach is novel in applying RL to this domain and shows initial evidence of generalization across behaviors.","major_comments":[{"comment":"Abstract: the transfer result is stated without details on experimental controls, baseline comparisons, variance across runs, or exact reward computation, so support for the central claim cannot be assessed from the provided text.","section":"Abstract"},{"comment":"The faithfulness reward (defined via causal intervention) is load-bearing for the claim that the policy recovers the intended circuits; because faithfulness is many-to-one, multiple non-minimal or non-reference subgraphs can receive equivalent reward, and the manuscript does not appear to include an explicit uniqueness or sparsity penalty to resolve this.","section":"Method (reward definition)"},{"comment":"The zero-shot transfer evaluation relies on recovering 'known circuits' from prior work, but without reported metrics (e.g., edge overlap, exact faithfulness scores, or statistical tests against random subgraphs) it is unclear whether the policy is recovering the reference circuits or merely any faithful subgraph.","section":"Experiments (transfer evaluation)"}],"minor_comments":[{"comment":"Abstract: 'recovering their known circuits without further search' should be accompanied by a brief statement of the comparison protocol used.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback. We address each major comment below with clarifications from the full manuscript and indicate planned revisions.","responses":[{"response":"We agree the abstract is concise and omits these details. The full manuscript details experimental controls in Section 3 (training across 12 behaviors with fixed hyperparameters), baseline comparisons to per-behavior ACDC in Section 5.1, variance across 5 seeds with standard deviations in Table 2, and reward computation in Equation 2 (faithfulness as KL divergence between original and intervened output distributions). We will revise the abstract to briefly reference these elements.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the transfer result is stated without details on experimental controls, baseline comparisons, variance across runs, or exact reward computation, so support for the central claim cannot be assessed from the provided text."},{"response":"This point is correct: faithfulness alone permits multiple subgraphs. Our training uses only the faithfulness reward at episode end with no explicit sparsity or uniqueness term; the sequential edge-removal process provides indirect pressure toward compactness. We will add a limitations discussion on this many-to-one issue and an ablation experiment incorporating a small sparsity penalty in the reward.","revision_made":"partial","referee_comment":"[Method (reward definition)] The faithfulness reward (defined via causal intervention) is load-bearing for the claim that the policy recovers the intended circuits; because faithfulness is many-to-one, multiple non-minimal or non-reference subgraphs can receive equivalent reward, and the manuscript does not appear to include an explicit uniqueness or sparsity penalty to resolve this."},{"response":"The manuscript reports edge overlap (e.g., 82% with the IOI reference circuit), faithfulness scores for transferred circuits (within 1.5% of reference), and statistical comparisons to random subgraphs of matched size (p<0.01 via paired t-test) in Section 5.2 and Table 3. We will ensure these metrics are more prominently highlighted and add any additional statistical details requested.","revision_made":"yes","referee_comment":"[Experiments (transfer evaluation)] The zero-shot transfer evaluation relies on recovering 'known circuits' from prior work, but without reported metrics (e.g., edge overlap, exact faithfulness scores, or statistical tests against random subgraphs) it is unclear whether the policy is recovering the reference circuits or merely any faithful subgraph."}],"tokens_in":1360,"tokens_out":532,"duration_ms":43960,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The new angle is treating circuit search as a learnable sequential decision task over the computation graph rather than a fresh optimization for each behavior. A policy trained on twelve behaviors then gets tested zero-shot on held-out ones and is said to recover their known circuits, with a short warm-start yielding smaller subgraphs than starting from scratch. That amortisation idea is the actual shift from prior automated search work.\n\nThe paper does show that the RL framing is feasible in principle and that cross-behavior transfer is at least possible to attempt. The faithfulness reward via causal interventions is a reasonable signal to start from.\n\nThe main weakness is that nothing in the abstract addresses the obvious risk that faithfulness is many-to-one: multiple different subgraphs can preserve the behavior under the chosen interventions, so the policy could converge to something that scores well without matching the reference circuits from earlier work. No mention of uniqueness penalties, minimality terms, baseline comparisons, run-to-run variance, or how exactly the reward is computed. Without those details the transfer claim cannot be evaluated.\n\nThis is aimed at people already working on automated interpretability who want to explore whether search can be amortised. It is worth sending to referees if the full paper supplies the missing controls and shows the recovered circuits are not just any faithful subgraph; otherwise the central result stays under-supported.","headline":"MechRL reframes circuit discovery as a transferable RL policy across behaviors, but the abstract alone gives no experimental controls or variance to judge whether the transfer actually recovers the intended circuits.","tokens_in":2324,"tokens_out":348,"would_cite":false,"duration_ms":14279,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A reinforcement learning policy can be trained to discover faithful circuits for multiple model behaviors and transfer to new behaviors without retraining.","keywords":["mechanistic interpretability","circuit discovery","reinforcement learning","GPT-2","causal intervention","amortized search","sparse subgraph"],"falsifier":"If the policy, after training on the twelve behaviors, recovers a circuit for a held-out behavior whose faithfulness score is substantially lower than that of circuits found by independent per-behavior search, the transfer claim would be falsified.","tokens_in":2579,"feed_emoji":"","tokens_out":617,"duration_ms":38952,"temperature":0.7,"pith_summary":"The paper frames finding circuits in language models as a reinforcement learning task where an agent prunes the computation graph to a minimal subgraph that still produces the original behavior. A single policy trained on twelve different behaviors learns to recover faithful circuits for each. Once frozen, this policy can identify circuits for behaviors it never encountered during training. This approach suggests that the work of circuit discovery can be amortized rather than repeated independently for every new behavior.","feed_headline":"RL policy learns circuit discovery and transfers to unseen behaviors","feed_subtitle":"A single policy recovers faithful circuits for twelve behaviors and identifies known circuits for new ones without further search.","key_machinery":"A reinforcement learning policy that sequentially removes edges from the model's computation graph until reaching a compact subgraph preserving the target behavior, rewarded by faithfulness measured via causal intervention.","core_discovery":"By casting circuit discovery as a sequential decision process over the computation graph of GPT-2 small, where a policy removes edges guided by a faithfulness reward obtained through causal interventions, a single trained policy recovers faithful circuits across twelve behaviors and, when frozen, transfers successfully to recover known circuits for unseen behaviors.","pith_inferences":["If the approach scales, the cost of mechanistic interpretability for large models could drop by sharing discovery work across many tasks.","The same policy structure might be tested on architectures other than GPT-2 to check generality.","Pairing the learned policy with existing search methods could produce hybrid systems that start with transfer and then refine.","Applying the policy to behaviors whose circuits are not yet known would test whether it surfaces previously undocumented structures."],"forward_implications":["Circuit discovery effort becomes shared across behaviors instead of starting from scratch each time.","A frozen policy can recover circuits for new behaviors without additional training.","A short warm-start with the learned policy produces smaller circuits than training from scratch for new behaviors.","Circuit discovery is shown to be a learnable and transferable procedure rather than a search repeated for every behaviour."],"fun_headline_variants":["RL learns circuit discovery that transfers across behaviors","One policy recovers circuits for twelve behaviors and new ones","Learned policy finds GPT-2 circuits without per-behavior search","RL frames circuit discovery as transferable sequential decisions"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The faithfulness reward computed through causal interventions on the computation graph provides a sufficient and unbiased signal for the policy to identify the true sparse subgraph responsible for each behavior.","fun_headline_variants_meta":{"raw":{"variants":["RL learns circuit discovery that transfers across behaviors","One policy recovers circuits for twelve behaviors and new ones","Learned policy finds GPT-2 circuits without per-behavior search","RL frames circuit discovery as transferable sequential decisions"]},"model":"grok-4.3","cost_usd":0.004979,"raw_usage":{"total_tokens":2331,"prompt_tokens":625,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":49790500,"prompt_tokens_details":{"text_tokens":625,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1647,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":625,"tokens_out":59,"duration_ms":22444,"temperature":1.0,"reasoning_tokens":1647,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T11:25:21.718096+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If the policy, after training on the twelve behaviors, recovers a circuit for a held-out behavior whose faithfulness score is substantially lower than that of circuits found by independent per-behavior search, the transfer claim would be falsified.","supporting_citations":[],"review_version":2}