{"id":"bde0bb06-4b92-453c-98f4-b2174cafd3e8","arxiv_id":"2506.16548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adaptive RMU with layer selection at decoder layers 12-14 (1B) and 24-26 (7B) ranked 4th in SemEval-2025 Task 4 unlearning.","lead":"This paper applies Adaptive RMU, an existing unlearning method, to erase memorized personal data from 1B and 7B OLMo models for SemEval-2025 Task 4. The practical takeaway is that later decoder layers are the best place to apply such edits, and the system placed 4th on both leaderboards.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"7B layer recommendation (24,25,26) is asserted without a reported 7B sweep, contradicting Section 4.2's statement that experiments ran only on the 1B model; the two-model ideal-layer claim lacks direct support.","rationale":"The reader's weakest_assumption points to exactly the gap I consider most load-bearing: the 7B layer recommendation is asserted with no reported sweep. I agree with that diagnosis. The 1B sweep is internally plausible, and the external leaderboard rank is positive evidence of competitiveness, so I would not reject the paper. However, the strongest claim includes a definite optimum for the 7B model, and that half of the claim is unsupported; verifying it requires either an actual 7B sweep or a downgrade of the statement to an extrapolation. I also noticed that Table 3's Adaptive RMU final score (0.376) does not match the paper's stated arithmetic mean of the three preceding columns (0.581); this is a separate reporting error that should be corrected, but it does not replace the missing 7B sweep as the primary issue. The appropriate verdict remains conditional.","tokens_in":6672,"tokens_out":7565,"duration_ms":72994,"concrete_test":"Run the same three-consecutive-layer adaptive-RMU sweep used for Table 2 on OLMo-7B-0724-Instruct-hf, i.e., evaluate all triples (0,1,2) through (29,30,31) with the official SemEval-2025 Task 4 final score; if the argmax is not (24,25,26), or if such a sweep cannot be reproduced from the released code, the Section 6 7B layer recommendation is an extrapolation rather than a measured result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central two-model claim is Section 6's assertion that the ideal adaptive-RMU layers are 12,13,14 for the 1B model and 24,25,26 for the 7B model. The 1B half is supported by the full sweep in Table 2, but no corresponding 7B sweep is presented anywhere in the paper. Section 4.2 states that experiments were confined to the 1B model, so the definite 7B optimum appears to be extrapolated from the 1B pattern (both triples sit near 75% of the decoder depth). A fourth-place leaderboard finish is real external evidence that the overall submission was competitive, but it does not test whether 24,25,26 is optimal for OLMo-7B; the ranking only establishes that one configuration performed well. The unsupported 7B layer claim is the load-bearing part of the strongest claim, not a cosmetic detail. A separate quantitative inconsistency in Table 3 (the Adaptive RMU final score should be 0.581 under the paper's stated arithmetic mean, not 0.376) reinforces the need for careful re-reporting but is secondary to the missing 7B measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies Adaptive Representation Misdirection Unlearning (Adaptive RMU) to the SemEval-2025 Task 4 unlearning benchmark, using the task-provided OLMo-1B and OLMo-7B models. On the 1B model, the authors sweep all 14 consecutive triples of decoder layers, report the task-aggregate, MIA, and MMLU components of the final score, and conclude that layers 12,13,14 are optimal. They also state without presenting measurements that layers 24,25,26 are ideal for the 7B model. The submission ranked fourth on both official leaderboards. The paper argues that later decoder layers are more effective targets for unlearning factual/PII knowledge, in contrast to earlier-layer targets in original RMU work, and it releases code on GitHub.","tokens_in":6942,"tokens_out":2544,"duration_ms":25557,"significance":"If the central claim were fully supported, this would be a useful empirical datapoint for layer-selective unlearning, with the 1B sweep being a genuinely systematic comparison and the fourth-place leaderboard finish providing external validation that the chosen configuration is competitive. The paper's release of code and its use of the task's official evaluation metric are strengths. However, the two-model claim in Section 6 is only directly supported for the 1B model; the 7B optimum is asserted rather than measured, and a displayed arithmetic inconsistency in Table 3 undermines confidence in the reported numbers. The layer conclusion is also a tuning result over the final metric, so its status as a mechanistic finding is weaker than the text suggests.","major_comments":[{"comment":"The paper states in Section 4.2 that experiments were conducted exclusively on the 1B parameter model, yet Section 6 asserts that the ideal layers for the 7B model are 24,25,26. No 7B layer sweep is reported anywhere. This is not a cosmetic omission: the strongest claim of the paper, as stated in Section 6, is a two-model conclusion, and the 7B half has no direct empirical support. The optimal 7B triple appears to be extrapolated from the 1B pattern (both around 75% of decoder depth), which is a reasonable hypothesis but not a measurement. I ask the authors to either report the 7B sweep (or at least a comparison of a few candidate triples on the 7B model) or to explicitly label the 7B recommendation as an extrapolation rather than an empirical result.","section":"Section 4.2 and Section 6"},{"comment":"The Adaptive RMU row in Table 3 reports a final score of 0.376, but the displayed components (task aggregate 0.387, MIA 0.872, MMLU 0.485) have arithmetic mean (0.387 + 0.872 + 0.485)/3 = 0.581, not 0.376. The other four rows in the table are consistent with the stated arithmetic-mean definition in Section 3.3. This inconsistency must be corrected, and the authors should double-check all reported final scores against their running code, since a single transcription error in a headline table raises doubts about other numbers.","section":"Table 3"},{"comment":"The headline claim that later layers are ideal is selected by sweeping all layer triples against the task's own final score. This makes the conclusion a post-hoc selection on the evaluation metric rather than a falsifiable prediction. The fourth-place leaderboard rank is an external check on the overall configuration, but it does not test whether the chosen layer triple is optimal. Please clarify in Section 6 that the 'ideal layers' conclusion is a tuning outcome on the development/evaluation split, and ideally report the ranking stability across the triples relative to the spread of final scores in Table 2.","section":"Section 5.2 and Section 6"},{"comment":"No variance, seeds, or repeated runs are reported. Each configuration appears to be evaluated once, and the paper describes randomized sampling during training. The layer sweep in Table 2 shows several close scores (e.g., 11,12,13 at 0.442 vs. 12,13,14 at 0.554 is less close, but the middle-layer cluster is dense), so without variance information the separation between neighboring triples may not be reliable. Please report run-to-run variability or at least state the number of runs and the seed handling.","section":"Section 5.2 / Table 2"}],"minor_comments":[{"comment":"The conclusion expands RMU as 'Rank-One Model Update,' but the paper's own Section 4.1 defines RMU as Representation Misdirection Unlearning. This acronym expansion is inconsistent and should be corrected.","section":"Section 7"},{"comment":"Equation (3) is followed by 'Where α controls' with a capital W; please make the prose style consistent. Also, the sentence 'RMU updates model weights iteratively, focusing on layers ℓ−2, ℓ−1, and ℓ' is ambiguous about whether the loss is applied at one layer or three; please clarify.","section":"Section 4.1"},{"comment":"The appendix text says 'middle layers achieve a balance ... but it remains more susceptible to MIA' while the figure caption says later layers are more robust to MIA. The narrative is clear, but the phrasing in the appendix could be sharpened to avoid the impression that middle layers are recommended.","section":"Figure 2 caption / Appendix"},{"comment":"Several references are incomplete or inconsistently formatted: the Robins entry is in all caps, the Huu-Tien et al. reference gives no venue, and the SemEval task description reference is listed as 'arXiv preprint' without an identifier. Please normalize all bibliography entries.","section":"References"},{"comment":"A few rows in Table 2 have merged or missing spaces (e.g., '5,6,70.6370.357' in the extracted text, and '0.367 0.3620.2650.419' in row 8,9,10). Please ensure the table is typeset cleanly so each column is readable.","section":"Table 2 formatting"}],"recommendation":"major_revision","confidential_remarks":"The missing 7B sweep is the main issue. The paper's own Section 4.2 admits experiments were only on 1B, so the 7B layer claim in Section 6 is unsupported as written. This is fixable by adding experiments or by reframing the claim, so I do not recommend rejection, but the revision needs to address it substantively. The Table 3 arithmetic error is the kind of detail that should have been caught in proofreading and should be checked against the submission package. The leaderboard result is real evidence and should be kept, but it should not be used as a substitute for a measured 7B layer sweep. I would also suggest the authors consult the official SemEval task report to make sure they use the same final-score definition and to cite it properly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a quick look if you care about layer selection in LLM unlearning. The 1B result is solid; the 7B claim is not.\n\nThe genuinely new thing here is the full sweep over consecutive decoder-layer triples for OLMo-1B under Adaptive RMU. The authors show that later layers (12,13,14) outperform earlier ones on the SemEval-2025 Task 4 final score, mainly because MIA robustness jumps from ~0.35 to ~0.84 while MMLU stays roughly flat. That is a concrete, useful observation. They also swap the generic retain set (WikiText) for the task-specific retain set, which is a sensible adaptation, and they release code. Fourth place on both official leaderboards is external evidence that the submission was competitive, not just self-report.\n\nThe soft spots are real but local. Section 4.2 says experiments were run exclusively on the 1B model, yet Section 6 states the ideal 7B layers are 24,25,26 with no 7B sweep anywhere. The leaderboard rank does not show that this triple is optimal; it only shows that one configuration scored well. That claim needs either a measured 7B sweep or a clearly labeled extrapolation. Table 3 also has a numeric inconsistency: the Adaptive RMU row lists final score 0.376, but the arithmetic mean of 0.387, 0.872, 0.485 is 0.581. That is the kind of re-reporting error the authors should fix. There are no error bars or repeated runs, and the training hyperparameters (learning rate, steps, batch size, alpha) are not given, so independent reproduction would require digging into the GitHub repo. The layer sweep is a tuning result, not a mechanistic prediction, because it sweeps against the task's own final metric; that framing is fine for a system paper but should be explicit.\n\nThe paper is what it is: a competent system-description for a competition, with one solid empirical finding and one overreach. The 1B layer result deserves to be in the literature. The 7B part should be either measured or softened. I'd engage with it if I were working on unlearning layer selection, and I'd encourage a serious referee to ask for the missing 7B experiment and the corrected numbers. Accept with revisions, not desk reject.","headline":"A useful 1B layer-sweep result for unlearning, with an unsupported 7B extension and a table typo that need fixing.","tokens_in":7459,"tokens_out":2742,"would_cite":false,"duration_ms":23341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying Adaptive RMU to decoder layers 12–14 of the 16-layer OLMo-1B gives the best tested unlearning score, and the paper scales that recipe to layers 24–26 for the 7B model.","keywords":["machine unlearning","large language models","Adaptive RMU","representation misdirection","decoder layer selection","privacy","personally identifiable information","SemEval-2025 Task 4"],"falsifier":"Run the same consecutive-three-layer sweep on the seven-billion-parameter model: if any triple other than 24, 25, 26 scores higher, the paper's central two-model layer claim fails.","tokens_in":6473,"feed_emoji":"🔒","tokens_out":10152,"duration_ms":89044,"temperature":0.7,"pith_summary":"The paper tries to establish that factual knowledge in LLMs—names, phone numbers, SSNs, addresses—is most cleanly unlearned from later decoder layers, and that the Adaptive RMU variant of representation-misdirection unlearning can do this without crippling retained knowledge. On the 16-layer OLMo-1B, it sweeps every consecutive three-layer block and finds layers 12, 13, 14 give the best final score, driven almost entirely by a jump in membership-inference robustness. It then transfers that three-quarter-depth recipe to the 32-layer OLMo-7B as layers 24, 25, 26 and reports 4th place on both task leaderboards. If right, the practical takeaway is a cheap rule of thumb: target roughly the last quarter of the decoder for factual unlearning, rather than the early layers earlier RMU work stressed.","feed_headline":"Later decoder layers are the best place to unlearn facts","feed_subtitle":"A 1B-parameter sweep finds personal data vanishes most cleanly near three-quarter depth; recipe ranks 4th on both SemEval-2025 boards.","key_machinery":"Adaptive RMU's forget loss replaces RMU's fixed steering target with an adaptively scaled random unit vector: the random direction is multiplied by the frozen model's activation norm at each token, so the push to misdirect forget-set representations adapts to how strongly the original model would have activated. The retain loss keeps unlearned activations close to the frozen model on retain data, and the combined loss is optimised only at the chosen layer triple. The layer sweep carries the argument: by testing every consecutive three-layer combination in the 16-layer model, the paper maps where in the decoder unlearning succeeds, and that map determines the recommended depth.","core_discovery":"The central claim is that unlearning factual and PII content with Adaptive RMU is best done in the later-middle decoder layers—specifically, layers 12, 13, and 14 among all triples from (0,1,2) to (13,14,15) for OLMo-1B, with the matching depth-scaled triple 24, 25, 26 recommended for OLMo-7B. The deciding factor is privacy: the MIA score jumps from 0.489 at layers 11–13 to 0.835 at layers 12–14, while task aggregate and MMLU stay roughly flat, so the later location buys privacy without sacrificing retention or general ability. This layer choice produced a 4th-place finish on both official leaderboards, and the paper reads it as evidence that factual memorisation concentrates at different depths than the hazardous conceptual knowledge targeted by original RMU, which favoured earlier layers.","pith_inferences":["If the 7B layer triple were verified with its own sweep, the three-quarter-depth rule would amount to a scaling statement about where factual memorisation concentrates, worth testing on other model families.","The later-layer concentration may be tied to this task's fine-tuned memorisation: the same sweep on a model that was not fine-tuned on the forget documents could separate intrinsic factual storage from instruction-tuning artifacts.","The MIA jump at layers 12–14 suggests membership signals live in a low-rank subspace around that depth; probing activations there before and after unlearning could reveal where PII is encoded.","Because adaptive RMU scales its steering by activation norm, the apparent layer preference could partly reflect activation magnitudes growing with depth; re-running the sweep with per-layer norm normalisation would test whether content, not scale, drives the result."],"forward_implications":["For unlearning factual or PII content, later decoder layers near three-quarters of the model's depth should be the first intervention point, not the early layers favoured by the original RMU recipe.","The task-specific retain set can replace the external corpus used in original RMU, so the recipe needs only data the unlearning task already provides.","Layer choice can be pinned down with a small-model sweep and then scaled by depth to a larger model, assuming the 7B extrapolation holds.","The main benefit of moving unlearning to later layers is membership-inference privacy: the MIA score rises sharply while task aggregate and MMLU stay approximately flat."],"supporting_citations":[{"why":"Supplies the original RMU forget/retain loss and random-vector steering that Adaptive RMU modifies.","marker":"(Li et al., 2024)"},{"why":"Introduces the adaptive scaling coefficient that is the method this paper applies.","marker":"(Huu-Tien et al., 2025)"},{"why":"Defines the task, subtasks, evaluation protocol, and leaderboards the paper reports.","marker":"(Ramakrishna et al., 2025b)"},{"why":"Provides the retain/forget dataset and evaluation suite used in the experiments.","marker":"(Ramakrishna et al., 2025a)"},{"why":"Describes the OLMo-1B and OLMo-7B decoder-only architecture whose layer triples are the intervention target.","marker":"(Groeneveld et al., 2024)"},{"why":"Supplies the MMLU accuracy component of the final score.","marker":"(Hendrycks et al., 2021)"},{"why":"Supplies the membership-inference metric whose jump at later layers drives the layer recommendation.","marker":"(Shokri et al., 2017)"}],"fun_headline_variants":["Forget PII at layers 12-14: privacy jumps, MMLU steady","Adaptive RMU: later layers erase PII without losing smarts","Unlearning sweet spot: layers 12-14 for 1B, 24-26 for 7B","PII vanishes best near three-quarter depth, 4th place on two boards","Later-middle layers: privacy spikes, task scores flat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the ideal layers for the seven-billion-parameter model can be inferred from the one-billion-parameter sweep (24–26 by proportional depth), because the paper says its experiments ran exclusively on the one-billion-parameter model.","fun_headline_variants_meta":{"raw":{"variants":["Forget PII at layers 12-14: privacy jumps, MMLU steady","Adaptive RMU: later layers erase PII without losing smarts","Unlearning sweet spot: layers 12-14 for 1B, 24-26 for 7B","PII vanishes best near three-quarter depth, 4th place on two boards","Later-middle layers: privacy spikes, task scores flat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4210,"prompt_tokens":889,"completion_tokens":3321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3214}},"tokens_in":505,"tokens_out":3321,"duration_ms":25226,"temperature":1.0,"reasoning_tokens":3214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:23:29.289918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same consecutive-three-layer sweep on the seven-billion-parameter model: if any triple other than 24, 25, 26 scores higher, the paper's central two-model layer claim fails.","supporting_citations":[],"review_version":2}