{"id":"98645aca-08a3-4725-bbec-252c47f3a855","arxiv_id":"2505.12636","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Edits made by ROME, MEMIT, PMET and other methods often leave the original fact intact inside later-layer attention heads, so a contextual prompt makes the edited model fall back to the old answer.","lead":"When you edit a fact inside a large language model, the model can still secretly remember the old fact and use it when asked in a roundabout way. This paper names that phenomenon 'superficial editing', measures it across popular editing methods, and traces it to specific attention heads in later layers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attack prefixes contain the original answer itself, so 'reversion' may be in-context copying rather than stored original knowledge; the causal mechanism lacks controls for this.","rationale":"The reader's verdict correctly flags the selection bias in CF-a/ZsRE-a: Appendix A filters to samples that already revert, so the 70% figure does not estimate prevalence on unselected data. I agree with that concern. However, I think an even more load-bearing problem sits upstream: the attack prefixes themselves embed the original answer. All three attack types (Wiki(o), Rep(o), Que(o)) include o, and the superficial-editing definition in Eq. (3) only requires that the model output o when o is present in the prompt. Any model with standard context-copying behavior will satisfy this, regardless of whether the edit actually 'stuck'. The mechanistic analysis in Section 4.3 then attributes the o-output to later-layer attention heads and left singular vectors, but these could be the same heads any model uses to move an entity from the prefix to the final position. Without a distractor control or an unedited-model baseline, the causal claim that these vectors 'encapsulate the original knowledge' in the edited parameters is not established. The concrete test I propose, replacing o with a distractor in the prefix and measuring whether the model still outputs o, would discriminate between stored-knowledge reversion and in-context copying. If the model outputs the distractor, the central mechanism claim would need reframing; the paper might still contribute an evaluation of context override, but not of 'superficial editing' in the sense of retained original knowledge. Therefore, until such a control is run, I would hold the verdict as UNVERDICTED rather than CONDITIONAL.","tokens_in":39654,"tokens_out":7268,"duration_ms":83063,"concrete_test":"Re-run the OM/OP evaluation and the LOPH/SVD analysis of Section 4.3 on a modified attack set where the original object o in each prefix is replaced by a matched distractor d (e.g., for the Toronto/London edit, prefix 'Montreal is the most populous city in Canada... The headquarter of CTV Television Network is located in'), keeping edited models and baseline prompts identical. Also compute the unedited model's OM/OP on the original attack prefixes. If the edited model outputs d or if OM/OP drops to near zero, the 'reversion' is in-context copying and the stored-knowledge mechanism is unsubstantiated; if it still outputs o with the distractor prefix while the unedited model does not, the paper's interpretation survives.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The most load-bearing weakness is that the attack probes used to define and measure superficial editing explicitly contain the original answer o. Equation (2) defines A = {Wiki(o), Rep(o), Que(o)}; Appendix A shows Wiki(o) is a Wikipedia summary of o, Rep(o) repeats o, and Que(o) poses a question that names o. Equation (3) then declares an edit 'superficial' when f'(a⊕x)=o. Under this construction, a model that simply copies the most salient entity from the prefix satisfies the definition. The unedited model already produces o on these prefixes, so the edited model's behavior may reflect ordinary in-context copying rather than a reversion to knowledge stored in the edited parameters. This confound undermines the prevalence claim and the mechanism: the later-layer attention heads and left singular vectors identified in Section 4.3 could be the generic circuit that moves an entity from context to the final position, not components that 'encapsulate the original knowledge' in the edited model. The paper reports no control where the prefix's entity is replaced with a distractor and no unedited-model baseline, so the causal attribution to residual original knowledge is not established. The reader's selection-bias concern is valid and related, but the entity-in-prefix confound is more fundamental: even on unfiltered data, the metric may be measuring prompt-following rather than superficial editing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper defines 'superficial editing' as a failure mode in which a knowledge-edited language model answers the original object on attack probes that contain that original object, despite answering the new object on baseline prompts. It constructs two stress datasets (CF-a, ZsRE-a) by retaining samples that already revert under three attack types, evaluates seven editing methods on three instruction-tuned LLMs, and reports OM/OP metrics. The mechanistic study uses residual-stream patching, logit-lens analysis, and attention-head and left-singular-vector ablations to argue that earlier-layer residual streams are implicated in new-knowledge enrichment (H1) and that later-layer attention heads encode the original knowledge (H2). It extends the analysis to 'superficial unlearning' on a small set of RWKU-derived targets.","tokens_in":39857,"tokens_out":8047,"duration_ms":74594,"significance":"If the central claims were established, this would be a useful contribution: it provides a large-scale evaluation of editing algorithms under adversarial prompts and a concrete interpretability framework (residual-stream patching, head-level LOPH, SVD-based vector ablation) for a practically important failure mode. The paper is transparent about its dataset construction and limitations, and it claims to ship datasets and code. However, the validity of the headline prevalence numbers and the causal mechanism depends on controls that are currently missing, in particular a demonstration that the attack probes are not being satisfied by generic in-context copying. With those controls, the paper's stress-testing methodology would be a valuable addition to the knowledge-editing literature.","major_comments":[{"comment":"The attack prefixes defined in Eq. (2) are Wiki(o), Rep(o), and Que(o), each of which explicitly contains the original answer o. Equation (3) then classifies an edit as superficial whenever f'(a⊕x)=o, so any model that copies the most salient entity from the prefix satisfies the definition. Because the paper reports no control with a distractor entity in the prefix and no unedited-model baseline for the same probes, the OM/OP values in Table 1 and the mechanistic claims in §4.3.2–4.3.3 may reflect generic in-context copying rather than reversion to stored original knowledge. I request (i) unfiltered prevalence estimates on the original CounterFact/ZsRE inputs, (ii) a distractor-prefix control, and (iii) unedited-model baselines for the same probes.","section":"§2, Eq. (2)-(3); Appendix A"},{"comment":"Appendix A states that CF-a and ZsRE-a are built by first editing with ROME, MEMIT, or MEND and then retaining only the samples that already elicit the original answer under the attack probes ('Samples that elicit the original answers from the model are retained'). The OM/OP numbers in Table 1 and the abstract's 'over 70% of cases' are therefore measured on a stress set that is guaranteed to contain the phenomenon; they do not estimate how often superficial editing occurs on unselected CounterFact or ZsRE inputs. Please report the rates on the unfiltered datasets, or explicitly qualify every prevalence claim as conditional on samples that were selected for revertibility.","section":"Appendix A; §3.1; Abstract"},{"comment":"The causal identification of attention heads and singular vectors is circular because both are selected by their effect on the original answer's latent probability on the very same set of attack probes. Heads are chosen by LOPH (Eq. 9), and the top singular vectors are chosen by the drop in PLL(o|z) under ablation (Eq. 11); Tables 2 and 4 then show that ablating these selected components lowers OAP and raises NAP. Since the selection criterion is the same as the outcome, the result is expected by construction. Please validate the identified heads/vectors on a held-out set of attack types or on edits that do not exhibit superficial editing, and include an ablation of the same number of randomly chosen components as a control.","section":"§4.3.2–4.3.3, Eq. (9), Eq. (11)"},{"comment":"The abstract and Section 1 state that PMET and AlphaEdit 'exhibit superficial editing in over 70% of the cases.' In Table 1, this holds only for the Wiki attack on LLaMA3-8B-Instruct (PMET OM=70.33, AlphaEdit OM=72.53); for the Que attack the corresponding OM values are 39.29 and 34.52, and the appendix tables contain many lower values. The prevalence claim should be tied to the specific attack and model, and it should be based on the unfiltered datasets once those are computed.","section":"Abstract; §1; Table 1"}],"minor_comments":[{"comment":"The locality definition is labelled 'Eff.' and the inequality uses o* instead of o, so the formula contradicts the prose definition of locality; please fix the label and the token in the inequality.","section":"Appendix B, Eq. (15)"},{"comment":"'Dissection of Attenion Head' should read 'Dissection of Attention Heads'.","section":"§4.3.3 heading"},{"comment":"The header 'OM ↓ OP↓' is ambiguous; please add a note that downward arrows indicate lower values are better, and ensure the arrow convention is applied consistently in the appendix tables.","section":"Table 1 and tables in Appendix B"},{"comment":"The code link is given as the word 'here' with no visible URL in the manuscript text; please provide the full URL in a footnote or reference.","section":"Abstract and §1"},{"comment":"The unlearning experiments use only 26 targets and 50 samples; the main text should acknowledge this limited scale when claiming 'robustness and broader applicability.'","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"I agree with the conditional verdict and the stress-test note: the entity-in-prefix confound is real and, combined with the outcome-selected data construction in Appendix A, it undercuts the headline prevalence and mechanistic claims. The paper has merit as a stress-testing methodology, and the central claims could in principle be rescued with unfiltered prevalence numbers, distractor/unedited baselines, and held-out validation of the identified components. If the authors cannot supply these controls, the paper would not be acceptable. The 'over 70%' statement in the abstract should also be corrected regardless."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version: the 70% superficial-editing figure should not be read as a prevalence estimate, and the attack probes used to define the phenomenon all contain the original answer, so the reversion may be ordinary in-context copying rather than leakage of stored knowledge. The mechanistic story about later-layer attention heads and singular vectors is plausible and worth a careful look, but it is currently under-controlled.\n\nThe paper does real work. It formalizes superficial editing, evaluates seven editing algorithms on three models across two datasets, and then runs a layered mechanistic investigation: residual-stream interventions, logit lens, head-level latent probabilities, SVD of head output matrices, and ablations. The consistency across ROME and MEMIT on three model families is a genuine strength, and the extension to unlearning is a sensible generalizability check. Releasing code is a plus.\n\nThe soft spot is load-bearing. Equation (2) defines attack prefixes as a Wikipedia summary of o, a repetition of o, or a question about o; Equation (3) then labels the edit superficial when the edited model outputs o on those prefixes. Because the unedited model already outputs o on the same strings, any model with a working copy circuit satisfies the definition. There is no distractor control where the prefix contains a different entity, and no unedited baseline, so the later-layer heads identified in Section 4.3 could simply be the circuit that moves an entity from context to the answer position, not components that encapsulate the original knowledge in the edited parameters. The dataset construction in Appendix A makes this worse: CF-a and ZsRE-a keep only samples that already revert under ROME, MEMIT, or MEND, so the OM/OP numbers in Table 1 are computed on a set guaranteed to contain the phenomenon. That makes the 'over 70%' claim partly circular. The causal sections would also benefit from random-head and matched-vector ablation controls, and the thresholds (tau=0.1, top-5%/10%) need sensitivity analysis. These are fixable, not fatal.\n\nThe paper is for people building knowledge-editing benchmarks and for interpretability researchers studying where parameter edits fail. As is, I would treat the mechanistic direction as promising rather than established. I would send it to peer review: the question is important, and a serious referee can push for the missing controls that would make the claims credible.","headline":"Attack probes that contain the original answer make the headline prevalence claim uninterpretable, but the mechanistic investigation is substantial and worth salvaging with controls.","tokens_in":40469,"tokens_out":3680,"would_cite":false,"duration_ms":40885,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge editing is often superficial: the original answer persists in later-layer attention heads, whose left singular vectors causally drive reversion, so ablating them weakens the old fact.","keywords":["superficial editing","knowledge editing","mechanistic interpretability","attention heads","singular value decomposition","logit lens","residual stream","unlearning"],"falsifier":"Recompute OM and OP on a random, unfiltered sample of CounterFact or ZsRE edits (no pre-filtering by reversion) and check whether the reversion rate is still above 70%; if it drops below, the claimed prevalence is an artifact of the stress-set construction.","tokens_in":39371,"feed_emoji":"🧠","tokens_out":7224,"duration_ms":63113,"temperature":0.7,"pith_summary":"Knowledge editing can look flawless on standard metrics while the model still produces the original (pre-edit) answer when the prompt mentions the old fact; the paper calls this 'superficial editing' and shows it is widespread: several methods such as PMET and AlphaEdit revert on over 70% of the constructed stress-set cases. The paper's central claim is mechanistic: the old knowledge is not erased but stored in specific later-layer attention heads, and within those heads it lives in particular left singular vectors of the attention output matrix. Treating a head's output as a weighted sum of singular vectors, the paper shows that a few of these vectors alone decode to the original answer, and ablating them lowers the probability of the original answer while raising the probability of the new one. It also finds the same attention-head and singular-vector pattern in a superficially unlearned model, suggesting a shared circuit for residual knowledge. A sympathetic reader should care because the result identifies exactly where current editing algorithms fail and gives a concrete target for fixing them.","feed_headline":"Edited LLMs revert because old facts stay in attention heads","feed_subtitle":"Removing a handful of left singular vectors in later-layer attention heads weakens the old answer and restores the edit.","key_machinery":"The central object is the attention head's output matrix $W_O^{(l,h)}$ in later layers. The paper rewrites the head's contribution at the last token as $z = \\sum_i \\lambda_i u_i$, where $u_i$ are the left singular vectors of $W_O^{(l,h)}$ and $\\lambda_i = \\sigma_i v_i^\\top x$ is a scalar coefficient set by the input; it claims a small subset of these $u_i$ carries the original answer. The accompanying mechanism is the 'Reversal of the Residual Stream' (RRS): from a late layer onward, the residual stream at the last token assigns higher probability to the original answer than to the edited answer, a reversal that is a prerequisite for superficial editing. A second load-bearing tool is the logit lens, which projects internal activations through the unembedding matrix to read latent probabilities of the original and new answers at each layer.","core_discovery":"On the paper's own terms, the discovery is: current parameter-editing algorithms (ROME, MEMIT, MEND, PMET, r-ROME, AlphaEdit) leave the original knowledge functionally intact inside the Transformer. When the input is a baseline prompt, the edited model outputs the new fact; when the input is prefixed with a Wikipedia summary of the old answer, a repetition of the old answer, or a question containing the old triple, the model reverts to the old answer. This 'superficial editing' is mediated by two factors the paper validates: the residual stream at the last subject position in early layers, where enrichment of the new knowledge is suppressed, and, decisively, certain attention heads in later layers that inject the original answer into the final position. Decomposing those heads' output matrices by SVD, the paper isolates specific left singular vectors whose linear combination encodes the original answer; ablating the top 5–10% of these vectors decreases the original-answer probability and increases the new-answer probability across ROME- and MEMIT-edited LLaMA and Qwen models. The same vector pattern appears in a superficial-unlearning setup, supporting the conclusion that this is a general locus of 'knowledge that the model was supposed to lose.'","pith_inferences":["A testable design consequence: an editing or unlearning method that explicitly zeroes the coefficients of the identified left singular vectors at edit time should achieve lower OM/OP than algorithms that only adjust MLP weights; this is my proposal, not the paper's.","Because the datasets CF-a and ZsRE-a were filtered to samples that already revert, the over-70% figure is likely an upper bound for ordinary workloads; running the same metrics on an unfiltered random sample would give the true practical frequency.","The same SVD dissection could be applied to other residual behaviors, for example jailbreak susceptibility or fact contamination after safety fine-tuning, to test whether later-layer left singular vectors are a general repository of pre-intervention knowledge.","The three attack types are all built around the original object (Wikipedia summary, repetition, or a question with the old triple); testing paraphrasal or multi-hop attack prefixes would reveal whether the identified heads are the general locus of reversion or just one trigger."],"forward_implications":["Standard efficacy, generalization, and locality scores are not enough to certify an edit: near-perfect scores coexist with over 70% reversion on attack probes for PMET and AlphaEdit on the constructed stress sets.","The original fact is never fully overwritten by current parameter-editing methods; it is stored in later-layer attention heads, so editing at the MLP level alone leaves a live circuit for the old answer.","Because ablating the identified left singular vectors lowers the original-answer probability and raises the new-answer probability, a targeted intervention on those vectors is a concrete route to making edits stick.","The same attention-head and singular-vector signature appears in superficial unlearning, implying that 'forgetting' procedures also leave the targeted knowledge recoverable in the same circuit."],"supporting_citations":[{"why":"Introduces ROME and the CounterFact dataset; ROME is the primary edited model used in the attack-probe construction and mechanistic analysis.","marker":"(Meng et al., 2022)"},{"why":"Introduces MEMIT, the second edited model used to build the evaluation sets and to validate the attention-head and singular-vector findings.","marker":"(Meng et al., 2023)"},{"why":"Introduces MEND, used as a baseline and as one of the three methods applied when filtering samples into CF-a and ZsRE-a.","marker":"(Mitchell et al., 2022a)"},{"why":"Supplies the subject-enrichment process that motivates the paper's H1 about the last subject position in earlier layers.","marker":"(Geva et al., 2023)"},{"why":"Provides the logit lens technique the paper uses to read latent probabilities of the original and new answers at every layer.","marker":"(nostalgebraist, 2020)"},{"why":"Supports the logit-lens reading of intermediate MLP and attention vectors as promoting concepts in vocabulary space.","marker":"(Geva et al., 2022)"},{"why":"Provides the gradient-ascent unlearning procedure used to build the superficial-unlearning models.","marker":"(Jang et al., 2022)"},{"why":"Supplies the RWKU dataset whose targets are used for the superficial-unlearning data construction.","marker":"(Jin et al., 2024)"},{"why":"Provides the GCG attack used to train adversarial suffixes that make unlearned models reveal original knowledge.","marker":"(Zou et al., 2023)"}],"fun_headline_variants":["Superficial LLM edits: old facts hide in attention heads","Editing LLMs is superficial: original knowledge stays in heads","Left singular vectors in attention heads defeat knowledge edits","Ablating head vectors makes LLM edits stick","Knowledge editing masks, doesn't remove, old facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The over-70% superficial-editing rate assumes that the filtered CF-a and ZsRE-a test sets, which keep only samples where ROME, MEMIT, or MEND already revert to the original answer, represent the real frequency of the phenomenon on ordinary knowledge edits; if those samples are not representative, the headline rate is an upper bound on a stress set, not a general prevalence estimate.","fun_headline_variants_meta":{"raw":{"variants":["Superficial LLM edits: old facts hide in attention heads","Editing LLMs is superficial: original knowledge stays in heads","Left singular vectors in attention heads defeat knowledge edits","Ablating head vectors makes LLM edits stick","Knowledge editing masks, doesn't remove, old facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3146,"prompt_tokens":988,"completion_tokens":2158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":604,"tokens_out":2158,"duration_ms":17067,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:29:49.794404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute OM and OP on a random, unfiltered sample of CounterFact or ZsRE edits (no pre-filtering by reversion) and check whether the reversion rate is still above 70%; if it drops below, the claimed prevalence is an artifact of the stress-set construction.","supporting_citations":[],"review_version":1}