{"id":"a0c6d0cc-383e-433d-aac3-7f836736e79a","arxiv_id":"2412.16469","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning a safety-tuned LLM on a capability task erases safety behavior more than the reverse order, and this forgetting is worse for specific groups such as Muslim people in the authors' tests.","lead":"This paper shows that when a language model is first trained to be safe and then trained on a regular task, it loses more of its safety behavior than the reverse order, and the loss hits some demographic groups harder than others. The result gives model developers a concrete warning: the order of fine-tuning steps changes what the model forgets, and that forgetting is not evenly spread.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline asymmetry may reflect forgetting of the restricted-output QA template rather than safety content; no matched neutral-content control isolates task format from safety, and the group-level bias metric lacks baseline normalization and confidence intervals.","rationale":"The reader's weakest assumption is the most load-bearing: the safety tasks were reformatted into restricted-output QA, so the measured forgetting could reflect template or format forgetting rather than loss of safety knowledge. This is not just a measurement nitpick; it targets the abstract's central claim that 'safety tuning' is disproportionately forgotten. Internal evidence supports the concern: SaFeRDialogues, the only generative task not scored by exact-match format, shows much smaller forgetting, and the paper's own Section 7 acknowledges task-similarity and format confounds without experimentally separating them. A matched-control experiment would settle whether the asymmetry is about safety content or about surface form. Because the paper's 3x3 design already includes BBQ, which shares a template with the capability tasks and still shows large forgetting, the confound is not sufficient to reject the paper outright; the verdict should remain conditional on additional controls and significance testing. The reader's verdict is therefore unchanged, and the reader identified the correct weak point.","tokens_in":16681,"tokens_out":7882,"duration_ms":76005,"concrete_test":"Run a matched-control chain in which a neutral-content task uses the exact output template of each safety task. For ToxiGenQA, train a binary sentiment or truthfulness task with the identical instruction ending in 'This is [answer]' and the same 2,261 training examples; for BBQ, train a non-safety three-choice task with the same 'The best answer is [letter]' template. Chain each control through ARC/CQA/CQA2 under the same hyperparameters. If control forgetting is statistically indistinguishable from ToxiGenQA/BBQ forgetting, the safety-specific claim is not supported; if controls forget significantly less, the safety claim survives. Additionally, recompute BiasedForgetting from relative or baseline-normalized group forgetting with bootstrap confidence intervals across the three seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that accuracy drops on ToxiGenQA and BBQ reflect loss of safety-relevant knowledge rather than loss of the response format introduced by the authors' QA recasting in §3.2. ToxiGenQA uses a unique verbalizer ('This is toxic/not toxic') while capability tasks use 'The best answer is [letter]' or 'The answer is yes/no'; if evaluation is exact-match, a model that still recognizes toxicity but emits the wrong template is scored as unsafe. The paper's own data are consistent with this confound: SaFeRDialogues, the only task evaluated by a classifier on free-form generations, shows far smaller forgetting (0.84-4.36%) than ToxiGenQA (3.98-38.05%) or BBQ (13.52-46.85%). The reverse ordering is not an adequate control because it compares different tasks, and the 3x3 design lacks a neutral-content task with the same template, number of choices, and difficulty. BBQ shares the 'The best answer is [letter]' template with ARC/CQA and still forgets more when trained first, so template mismatch alone cannot explain all results; however, without a matched non-safety control, the conclusion that 'safety tuning' rather than task-specific properties is disproportionately forgotten remains unestablished. The biased-forgetting claim is also under-specified: group-level forgetting is an absolute accuracy gap with no baseline normalization and no confidence intervals, so higher-baseline groups have more room to drop, creating a possible floor/ceiling artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies catastrophic forgetting in chained supervised fine-tuning of LLaMA-2 7B, comparing sequences in which a capability task (ARC, CommonsenseQA, CommonsenseQA 2.0) is followed by a safety/bias task (ToxiGenQA, BBQ, SaFeRDialogues) with the reverse order. It introduces a metric called biased forgetting, defined as the gap between group-level and average forgetting, and reports three main findings: (i) safety tasks are forgotten more when they are tuned first and then followed by a capability task than capability tasks are forgotten when the order is reversed; (ii) forgetting is unevenly distributed across demographic groups, with certain groups (e.g., Muslim, Jewish, Native American in ToxiGenQA; Age and Disability Status in BBQ) consistently suffering more; and (iii) first-task loss curvature (spectral radius of the Hessian) correlates with downstream forgetting, and replaying a small portion of the safety data or using a low second-task learning rate mitigates forgetting. The paper frames these results as evidence that task ordering is a safety-relevant training decision and that safety evaluations should be group-disaggregated.","tokens_in":16966,"tokens_out":6083,"duration_ms":55772,"significance":"If the central claims hold, the paper is a valuable empirical contribution to LLM continual learning and safety: it systematically compares nine task pairs with fixed hyperparameters, uses an openly available base model, repeats experiments with three seeds, and proposes a concrete group-level metric. The finding that safety tuning is disproportionately forgotten by subsequent capability tuning, and that the effect varies by demographic group, would have direct implications for how open-weight models are fine-tuned downstream. The paper also gives a practical mitigation (replay with as little as 5% of safety data) and connects the results to flat-minima theory. However, the strength of the conclusions currently depends on ruling out a format/content confound and on making the group-level metric statistically robust; these are addressable within the manuscript's scope.","major_comments":[{"comment":"The central asymmetry may partly reflect forgetting of the response template rather than loss of safety-relevant knowledge. ToxiGenQA is reformatted with a unique verbalizer ('This is toxic/not toxic') while ARC/CQA/BBQ use 'The best answer is [letter]' and CQA2 uses 'The answer is yes/no'; under exact-match evaluation, a model that still recognizes toxicity but emits the wrong format is scored as unsafe. The authors acknowledge task similarity as a confound in §7 but do not isolate it experimentally: SaFeRDialogues, the only generative task scored by a classifier, shows much smaller forgetting (0.84–4.36%) than ToxiGenQA (3.98–38.05%) or BBQ (13.52–46.85%), which is consistent with a format-based account. The reverse ordering is not an adequate control because it compares different tasks. I request a matched neutral-content control with the same template, number of choices, and difficulty, or a robustness analysis that evaluates ToxiGenQA/BBQ with semantic equivalence or accepts both template variants.","section":"§3.2, §7; Table 1"},{"comment":"The biased forgetting metric is an absolute difference between group forgetting and average forgetting, with no normalization by baseline accuracy. Groups with higher baseline accuracy have more room to drop, so the reported BiasedForgettingmax values could partly be a floor/ceiling artifact. The paper should report per-group baseline accuracies and the group-level relative forgetting (defined in §3.2 but not used for groups), and provide confidence intervals via bootstrapping with a multiple-comparison correction across the 13 ToxiGenQA groups and 9 BBQ groups. Without these, the claim that forgetting disproportionately impacts certain groups is not yet quantified in a way that rules out trivial scale effects.","section":"§3.2, Eq. (2); Figure 3"},{"comment":"The headline ordering asymmetry is reported as point estimates averaged over three seeds, without standard deviations or confidence intervals. This is a comparative claim across nine task pairs, and the paper should report variability; a paired or mixed-effects test of the ordering effect would strengthen the claim. The absence of error bars in Figures 5–7 also makes it difficult to assess whether the learning-rate trends are reliable, especially where the paper itself states the picture is unclear for the first-task learning rate.","section":"§4.1, Table 1"},{"comment":"The curvature explanation is a post-hoc regression fitted to the same six tasks used in the main experiments, not an independent test of the hypothesis. With six task-level points, or with repeated observations if each task contributes multiple downstream pairs, the reported OLS p ≤ 0.01 and R² = 0.85 are fragile; the spectral radius is estimated from 50 training samples with no variance or convergence diagnostics reported. This section should be explicitly framed as exploratory, and the predictive claim should be validated, e.g., by leave-one-task-out prediction or by pre-registering the curvature hypothesis on held-out task pairs.","section":"§4.3, Eqs. (3)–(4); Figure 4(b)"}],"minor_comments":[{"comment":"The text 'Section 4.3 shows a markedly sharper minima...' should refer to Figure 4(a), not to the current section; the same self-reference appears later in the paragraph.","section":"§4.3"},{"comment":"The formula for relative forgetting is ambiguous: it writes RelativeForgettingAB with a denominator AccA(θ∗A,g), but the left-hand side has no group subscript. Please clarify whether relative forgetting is defined per group or overall.","section":"§3.2"},{"comment":"The table title says 'ToxigenQA accuracy across different settings' but the table lists accuracy for ToxiGenQA, BBQ, and SaFeRDialogues; the title should reflect all three tasks.","section":"Table 3"},{"comment":"The caption states that the blue dotted vertical line denotes the average forgetting on ToxiGenQA, but Figure 10 shows BBQ results; it should say BBQ.","section":"Figure 10"},{"comment":"The footnote says 'see footnote 1' for the ToxiGen classifier, but footnote 1 is about the definition of safety; the reference to the classifier should be a separate footnote describing the model and scoring procedure.","section":"Footnote 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a CS.CL venue and the core empirical design is systematic and reproducible in principle. The main barrier to acceptance is the format/content confound affecting the central asymmetry claim and the lack of statistical rigor in the group-level metric; both are fixable with additional experiments or re-analysis. I would not reject at this stage, but I would require the authors to address the matched-control and baseline-normalization issues before the claims can be considered established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result is solid: across all nine (capability, safety) task pairs, forgetting of the first task is larger when the first task is a safety/bias task than when it is a capability task. Table 1 makes that hard to argue with. The biased-forgetting metric is a simple and useful way to quantify group-level unevenness, and the repeated finding that certain groups (Muslim, Jewish, Native American for ToxiGenQA; Age, Disability Status for BBQ) land at the top is safety-relevant and worth taking seriously.\n\nWhat is genuinely new here is the systematic ordering experiment with controlled data sizes and fixed hyperparameters, plus the group-disaggregated measurement. The curvature explanation (sharper minima forget more) and the replay mitigation are re-applications of known ideas, but applied cleanly to a question that matters for deployment.\n\nNow the soft spots, in proportion.\n\nFirst, the headline numbers have no confidence intervals. Three seeds were run but Table 1 reports averages only. The biased-forgetting values have no error bars and no multiple-comparison control across 13/9 groups. The consistency across nine pairs keeps the central claim intact, but specific magnitudes should be read as rough.\n\nSecond, the format confound is real but not fatal. ToxiGenQA and BBQ are recast as restricted-output QA, and capability tasks are also QA. If a model loses the response template rather than the safety knowledge, it would be scored as unsafe. However, BBQ and ARC share the same 'The best answer is [letter]' template, and BBQ still forgets more when trained first. So template mismatch alone cannot explain the asymmetry. Still, without a matched neutral-content control, the claim that 'safety tuning' per se is disproportionately forgotten is not fully isolated from task difficulty or other task properties. The paper acknowledges this in Section 7 but does not experimentally separate format from content. SaFeRDialogues, the one generative/classifier-evaluated task, shows much smaller forgetting, which is consistent with either interpretation.\n\nThird, the curvature-forgetting regression is six points and one OLS line. R^2=0.85 on the same data that motivated the hypothesis is a fitted correlation, not an independent test. It is plausible but should not be over-interpreted.\n\nFinally, no code or seeds are released, which weakens reproducibility for a paper that presents itself as controlled.\n\nBottom line: this paper is worth engaging with seriously. The core ordering asymmetry is robust, the metric is useful, and the group-level results justify follow-up. Who is it for? Anyone designing chained fine-tuning pipelines, especially safety-then-capability, and anyone evaluating safety metrics that aggregate across groups. It deserves a proper peer review; a good referee would ask for confidence intervals, a matched non-safety control, and a less overfit curvature analysis.\n\nRecommendation: send it to review.","headline":"Chained tuning order matters: safety tasks forget more when trained first, consistently across nine pairs; the biased-forgetting magnitudes need error bars and a neutral control before the group-level claims are treated as firm.","tokens_in":17489,"tokens_out":3503,"would_cite":true,"duration_ms":31101,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chained fine-tuning of LLMs causes order-dependent, group-uneven loss of safety knowledge.","keywords":["catastrophic forgetting","continual learning","safety tuning","biased forgetting","task ordering","large language models","Hessian spectral radius","demographic groups"],"falsifier":"A decisive test: retrain the safety→capability chains, then evaluate on ToxiGenQA and BBQ with the response format re-supplied, e.g., reminding the model to end with 'The best answer is...' or 'This is toxic'; if accuracy recovers to near pre-capability levels whenever the format is restored, the observed forgetting is mostly template loss rather than safety-knowledge loss.","tokens_in":16497,"feed_emoji":"⚠️","tokens_out":8624,"duration_ms":68197,"temperature":0.7,"pith_summary":"This paper tries to establish that when a large language model is fine-tuned first on a safety or bias task and then on an ordinary capability task, the safety behavior is forgotten much more heavily than the capability is forgotten in the reverse order. It introduces a metric called biased forgetting to capture a second, less visible effect: the safety knowledge that is lost is not spread evenly across demographic groups, and some groups lose considerably more than the average. The authors argue that the asymmetry is tied to the sharpness of the loss landscape at the end of the first training task, with safety tasks converging to sharper minima that are more easily overwritten, and they show that a brief replay of a small slice of safety data can recover most of the lost behavior. A sympathetic reader would care because real deployment pipelines routinely chain safety tuning and downstream capability tuning, so the claim implies that task order is a safety-relevant decision and that safety evaluation needs to be reported per group.","feed_headline":"Safety tuning fades up to 43% after a reasoning task follows","feed_subtitle":"When a reasoning task follows safety tuning, safety knowledge drops most—and specific groups lose the most.","key_machinery":"The central measuring device is the forgetting metric $\\mathrm{Forgetting}_{AB} = \\mathrm{Acc}_A(\\theta^*_A) - \\mathrm{Acc}_A(\\theta^*_{AB})$, with a group-level version and the biased-forgetting gap $\\mathrm{BiasedForgetting}_{A,g} = \\mathrm{Forgetting}_{A,g} - \\mathrm{Forgetting}_A$. The explanatory mechanism is the curvature of the first-task minimum: using the Taylor bound $L_A(\\theta^*_{AB})-L_A(\\theta^*_A) \\approx \\frac12 \\Delta\\theta^{*\\top} H \\Delta\\theta^* \\le \\frac12 \\rho(H) \\|\\Delta\\theta^*\\|^2$, the paper measures the spectral radius $\\rho(H)$ of the first task's Hessian by power iteration and finds that safety tasks like ToxiGenQA and BBQ end in sharper minima than capability tasks, so a fixed parameter move in the second task costs them more loss.","core_discovery":"On a 7-billion-parameter open language model, the paper reports that for every one of nine pairs of one capability task and one safety task, forgetting of the first task is far larger when the safety task was trained first. Representative numbers from Table 1: ARC→BBQ forgetting is 1.02 percentage points while BBQ→ARC forgetting is 43.45, and CQA2→ToxiGenQA is 9.29 while ToxiGenQA→CQA2 is 38.05. The authors define forgetting as the accuracy drop on the first task after the second task, and biased forgetting as the gap between a demographic group's forgetting and the average forgetting. They report maximum biased forgetting values up to 10.09 on ToxiGenQA→CQA2, with Muslim, Jewish, and Native American groups consistently among the worst-affected in ToxiGenQA and Age and Disability Status among the worst in BBQ. They also show that the Hessian spectral radius after the first task correlates with later forgetting, and that a third fine-tuning stage on as little as 5% of the safety data, or about five optimization steps, restores most of the lost safety accuracy.","pith_inferences":["If format loss rather than content loss drives the numbers, the true safety deficit from chained tuning may be smaller than the headline table suggests; the same confound could inflate the apparent group disparity if some groups' examples are easier to answer from surface patterns.","The curvature result suggests a practical rule the paper does not state outright: when chaining tunings, measure the first task's Hessian spectral radius and put the wider-minimum task first; this could be tested cheaply before committing to a full fine-tuning run.","A natural extension is to repeat the ordering experiment on a more recent base model and on non-QA capabilities like code or summarization; persistence of the ordering asymmetry outside QA formats would make the safety implication much broader."],"forward_implications":["Chained tuning should be treated as a safety decision: for all nine (capability, safety) task pairs, safety-task forgetting is larger when the safety task is tuned first, with BBQ→ARC reaching 43.45% forgetting versus 1.02% for ARC→BBQ.","Safety evaluations need group-level reporting: maximum biased forgetting reaches 10.09 points on ToxiGenQA→CQA2, and the groups losing most are not the average groups.","First-task minima width predicts forgetting: tasks that end in sharper minima, such as BBQ and ToxiGenQA, are forgotten more, so curvature measurements could guide task ordering.","Rehearsal is a cheap mitigation: re-tuning on 5% of the safety data or about five optimization steps nearly restores safety accuracy with only a minor capability drop.","Capability tuning after safety tuning can degrade safety even when the capability task is benign, so downstream fine-tuning pipelines should audit for this rather than only tracking final-task accuracy."],"supporting_citations":[{"why":"Supplies the 7B open base model on which all chained fine-tuning runs are performed.","marker":"Touvron et al., 2023"},{"why":"Supplies ToxiGen, recast as ToxiGenQA, and the classifier used to score SaFeRDialogues safety.","marker":"Hartvigsen et al., 2022"},{"why":"Supplies BBQ, whose nine demographic groups make group-disaggregated biased forgetting measurable.","marker":"Parrish et al., 2022"},{"why":"Supplies SaFeRDialogues, the generative safety task whose lower forgetting anchors the format-dependence discussion.","marker":"Ung et al., 2022"},{"why":"Supplies the Taylor-expansion and Hessian-curvature framework linking first-task minima width to forgetting.","marker":"Mirzadeh et al., 2020b"},{"why":"Provides the revised ToxiGen v2 filtering used to construct the ToxiGenQA dataset.","marker":"Hosseini et al., 2023"},{"why":"Establishes the prior result that benign fine-tuning can degrade safety, which the paper extends to task ordering and groups.","marker":"Qi et al., 2023"},{"why":"Motivates the replay/rehearsal mitigation that the paper applies as a third fine-tuning stage.","marker":"Rolnick et al., 2019"}],"fun_headline_variants":["Chained tuning: order decides what LLMs forget","Safety first, reasoning after: 43-point forgetting","Forgetting is biased: some groups lose safety more","Retrain on 5% safety data to undo forgetting","Task order skews forgetting; groups hit unevenly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central numbers assume that a drop on ToxiGenQA, BBQ, and SaFeRDialogues measures loss of safety knowledge, not simply loss of the response template or scoring format the model was using for those tasks.","fun_headline_variants_meta":{"raw":{"variants":["Chained tuning: order decides what LLMs forget","Safety first, reasoning after: 43-point forgetting","Forgetting is biased: some groups lose safety more","Retrain on 5% safety data to undo forgetting","Task order skews forgetting; groups hit unevenly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3928,"prompt_tokens":921,"completion_tokens":3007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2930}},"tokens_in":537,"tokens_out":3007,"duration_ms":20250,"temperature":1.0,"reasoning_tokens":2930,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:33:09.635458+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: retrain the safety→capability chains, then evaluate on ToxiGenQA and BBQ with the response format re-supplied, e.g., reminding the model to end with 'The best answer is...' or 'This is toxic'; if accuracy recovers to near pre-capability levels whenever the format is restored, the observed forgetting is mostly template loss rather than safety-knowledge loss.","supporting_citations":[{"cited_title":"SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures","cited_arxiv_id":"2110.07518","evidence_quote":"Supplies SaFeRDialogues, the generative safety task whose lower forgetting anchors the format-dependence discussion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the replay/rehearsal mitigation that the paper applies as a third fine-tuning stage."}],"review_version":1}