Pith. sign in

REVIEW 2 major objections 5 minor 52 references

TRACE learns a single low-rank adapter on simulated corruption trajectories and shows it restores near-100% safety to fine-tuned LLMs while leaving task utility within ±1.7% of the undefended model, without any per-user calibration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:53 UTC pith:OIKG4XUS

load-bearing objection A genuinely different approach to post-FT safety recovery with unusually careful evaluation, but the universal-patch claim rests on an unmeasured simulation-to-deployment match. the 2 major comments →

arxiv 2607.16242 v1 pith:OIKG4XUS submitted 2026-06-26 cs.LG cs.AIcs.CR

TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

Changyue Li , Jiaming He , Youliang Yuan , Jialin Wu , Boxi Yu , Zhicong Huang , Pinjia He This is my paper
classification cs.LG cs.AIcs.CR
keywords safety alignmentfine-tuning-as-a-serviceparameter mergingLoRAsafety patchtrajectory simulationpost-training realignmentsafety-utility trade-off
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning-as-a-Service platforms let users customize LLMs, but the fine-tuning process can erase safety alignment. Existing post-hoc defense methods merge a safety patch with the tuned model, yet the patch and the user's update overlap in parameter space, forcing a hard safety-utility trade-off that must be recalibrated per user. This paper proposes TRACE, which shifts the problem from online merge calibration to offline patch learning: it simulates a harmful fine-tuning trajectory on the base model, producing progressively corrupted states, then optimizes a single LoRA adapter that restores refusal behavior across all these states while preserving benign task performance. The paper reports that TRACE achieves at least 94% safety on every benchmark/model combination (100% in most), keeps utility within ±1.7% of the undefended baseline, and occupies a safety-utility region no baseline reaches even with oracle knowledge of the user's training intensity. If correct, this means providers could restore safety with one fixed, low-rank patch and no per-user tuning.

Core claim

The central claim is that task-safety update entanglement—the directional overlap between a user's fine-tuning update and a safety-recovery patch—is the structural bottleneck of merging-based realignment, and that it can be circumvented by learning the patch offline over a simulated corruption trajectory. TRACE alternates between stepping a base model on harmful and benign data to create progressively corrupted checkpoints, and training a LoRA adapter (rank 64) to recover safety on each fresh state. The resulting adapter is 'disentangled' from task-update directions (saliency overlap drops from ~0.36 to below 0.079) and 'decisive' against harmful shifts of unknown intensity. The paper argues

What carries the argument

The alternating simulate-and-learn loop (Algorithm 1): Phase A applies gradient steps on a response-only causal-LM loss over paired harmful and benign batches to walk the base model along a corruption trajectory; Phase B freezes the corrupted state and updates a zero-initialized LoRA adapter on refusal responses plus benign-task examples. This exposes the adapter to a distribution of corruption intensities, forcing it to find safety directions orthogonal to task updates while remaining strong enough to dominate harmful drift. The ablation shows removing the trajectory collapses safety to near-zero, identifying the trajectory as the load-bearing mechanism.

Load-bearing premise

TRACE's universal transfer rests on the assumption that its simulated corruption trajectory—the base model stepped on one harmful and one benign surrogate dataset at a fixed learning rate—faithfully represents the corruption intensities of real user fine-tuning, including purely harmful 1,000-sample, 30-epoch training.

What would settle it

Fine-tune a held-out model outside the simulated regime—for example, on 5,000 harmful examples or for 50 epochs, or with a learning rate several times the simulation's ηθ—then apply the TRACE adapter and measure safety and utility. If safety falls below the claimed 94% or utility deviates by more than ±1.7%, the calibration-free universal-transfer claim collapses. A lighter check: measure the saliency overlap between the TRACE patch and a real task update; if it exceeds the reported ~0.079, the disentanglement evidence is an artifact of the chosen surrogate data.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If TRACE's results hold, FTaaS providers can deploy a single precomputed LoRA adapter (0.41 s online overhead) to any user-tuned model, removing per-user coefficient search from the deployment pipeline.
  • The patch remains effective across 5–30 user-training epochs, so safety coverage no longer degrades with fine-tuning intensity, a regime where fixed-strength baselines collapse.
  • Existing merging operators (arithmetic and projection-based) can be improved by substituting their conventional safety patch with TRACE's adapter, shifting their Pareto frontiers toward higher safety and utility.
  • Safety recovery after full-parameter fine-tuning is achievable within a low-rank subspace, contradicting the previous assumption that full-parameter patches are required for FFT-corrupted models.
  • The simulation trajectory is the indispensable ingredient: without it the patch's safety rate falls to 9–16%, comparable to an undefended model, so the corruption-diversity of training is what buys generalization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the disentanglement property is generic rather than an artifact of the two test models, it suggests safety alignment in LLMs may be storable as a separable, low-rank module—so safety could be treated as a plug-in capability that persists through task adaptation, with implications for how we audit and repair third-party models.
  • One testable extension: apply trajectory-based patch learning to other safety dimensions (bias, hallucination refusal, privacy) and to larger or smaller models; the paper's own assumption that larger models have 'more structured' representations is promising but untested.
  • A second extension: probe the boundary of the 'bounded fine-tuning regime' the paper concedes—e.g., empirically map at what harmful-corpus size or epoch count the patch starts to fail, which could yield the formal bound the authors list as future work.
  • Because the patch never sees real user data, its transfer owes to the surrogate trajectory; a practical deployment would need a monitoring check on the actual fine-tuned model to detect when user training has left the simulated regime.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes TRACE, an offline trajectory-based safety patch learning framework for post-fine-tuning realignment in Fine-Tuning-as-a-Service (FTaaS). TRACE first simulates a user fine-tuning trajectory on surrogate harmful and benign data, producing progressively corrupted model states. It then optimizes a single low-rank LoRA adapter across those states so that simple weight addition restores safety while preserving task utility, without per-user calibration at deployment. The method is evaluated on Llama-3.1-8B-Instruct and Qwen3.5-9B, using OOD harmful benchmarks (ShadowAlignment, PureBad, SafeRLHF) and utility benchmarks (SamSum, SQLCreate, GSM8K), including mixed benign/harmful user corpora. The authors report that TRACE achieves at least 94% safety across all benchmark/model combinations, keeps utility within ±1.7% of the undefended baseline, and reaches a safety-utility operating point that baseline methods cannot attain even when their repair strengths are swept with oracle knowledge of the user's training intensity.

Significance. If the central claim holds, TRACE would be a substantial step forward for post-FT safety recovery: a single, calibration-free, low-rank patch that dominates the safety-utility frontier across unseen user fine-tuning outcomes. The evaluation design is a genuine strength. Unlike much of the literature, the online evaluation uses datasets held out from offline training, so the headline results do not reduce to fitted values. The Pareto sweep in §5.4 gives baselines oracle knowledge of the user's training intensity, the trajectory ablation in Table 5 shows that the simulation component is load-bearing rather than decorative, and the epoch robustness study in §5.3 goes beyond a single operating point. However, the 'universal patch' conclusion rests on the faithfulness of the simulated corruption trajectory to real user fine-tuning states. The authors themselves concede in §7.2 that there is no formal bound on the regime in which the patch remains effective. The paper is therefore scientifically promising but requires additional evidence before the universality claim can be accepted.

major comments (2)
  1. [§4.2 (Eqs. 8–9), §5.1] The universal, calibration-free claim hinges on the simulated trajectory Tsim being representative of real user FT states. The simulation fixes a single learning rate ηθ and always mixes benign batches into harmful updates, while §2.1 says users control LR, batch size, optimizer, and data composition. §5.3 varies only epochs, and the recipe used to produce the user models in Tables 2–3 is never reported. If that recipe matches the offline simulation, the evaluation is in-distribution in the optimization-state space: only the datasets are OOD, not the corruption states. Please report the exact user fine-tuning hyperparameters and add a robustness sweep over user LR/batch size/optimizer (or at least a few held-out recipes) to substantiate the claim.
  2. [§7.2] The paper states that 'extremely prolonged training... can fundamentally reshape the model's representational geometry, potentially invalidating the safety directions encoded in the patch' and provides no formal bound on corruption intensity. Since the headline claim is that one patch restores safety for any unseen user fine-tuned model without calibration, this bounded regime is load-bearing rather than a routine limitation. Please add experiments that push beyond the current epoch range (e.g., more epochs, higher LR, larger harmful corpora) to map the failure boundary, or explicitly state the applicable intensity regime as part of the central claim.
minor comments (5)
  1. [§5.2.1, Table 2] The phrases 'improves the Llama safety rate from 23% to 100%' and 'Qwen safety rate from 38% to 100%' refer to the second-best baseline's rate, not the undefended model's rate. Please rephrase to avoid confusion.
  2. [Appendix A] Please report the user fine-tuning recipe used to generate the online evaluated models (optimizer, learning rate, batch size, number of steps). This is essential for interpreting the OOD claim.
  3. [Figure 6] The TRACE point is shown as a single fixed point. Given that safety and utility are stochastic, please report the variance over replications or seeds, or state the number of runs.
  4. [§3.1, Eq. (6)] The saliency overlap Scos is reported for a single user/task pair. Please include the spread across multiple datasets and seeds so readers can judge the stability of the entanglement measurement.
  5. [§7.2] The adaptive-attacks discussion says the user cannot observe model internals, but FTaaS users can query the deployed API and observe outputs. The claimed information asymmetry should be stated more carefully.

Circularity Check

0 steps flagged

No circular derivation: headline results are OOD and uncalibrated; only a minor non-load-bearing self-citation to EnchTable for the offline dataset protocol.

full rationale

This derivation is self-contained. TRACE's patch is trained offline on a simulated trajectory (Eqs. 8–9) using BeaverTails + WizardLM; all headline safety results are measured on unseen harmful benchmarks (ShadowAlignment, PureBad, SafeRLHF) and utility on SamSum/SQL/GSM8K, none of which appear in the training objective (Eq. 10). The 'universal, calibration-free' claim is therefore not a re-statement of the training loss: it is an OOD generalization test. No fitted parameter is renamed as a prediction: the learned LoRA is applied as θ+ϕ without per-user tuning, while baselines are given oracle repair-strength sweeps (§5.4), so TRACE's Pareto point is not an artifact of hindsight calibration. The trajectory is a training distribution, not a test-set fit; the epochs used in §5.3 are not used to optimize ϕ. The main caveat—that the surrogate trajectory may not cover extreme real FTaaS corruption states—is a genuine external-validity limitation, explicitly conceded in §7.2, but it is not circular: the paper does not define the evaluation states as the training states. The only self-citation is to [6] (EnchTable, with overlapping authors) for the offline dataset choice and as a baseline ('For offline, we align with the prior work [6] and use BeaverTails and WizardLM as the harmful, and benign dataset'); that citation is not load-bearing and does not force any result. Hence the low score reflects only this minor, non-load-bearing self-citation; there is no circular step.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

TRACE's contribution is a training procedure, not a derivation: the paper adds one hand-tuned objective (Eq. 10 with ω=2), one compression choice (r=64), and one simulation protocol (Eqs. 8-9) on top of standard supervised fine-tuning. The load-bearing domain assumptions are that task-vector arithmetic is a valid model of LLM parameter behavior, that refusal lives in a low-rank subspace, and that simulated corruption transfers to real users; the first is inherited from cited work, the second from [25], and the third is asserted and left bounded. None of these makes the headline evaluation circular, since online datasets are OOD. No new physical or architectural entities are postulated: the safety patch φ is an optimized artifact (a LoRA adapter), not an invented mechanism with independent-evidence requirements.

free parameters (6)
  • utility balancing weight ω = 2
    Eq. 10 weights benign-task preservation vs refusal recovery during patch training; hand-selected (Appendix 1); directly shapes the claimed disentanglement/utility behavior.
  • LoRA rank r = 64
    Adapter rank for the decisive safety subspace (Appendix 1); hand-chosen capacity; the paper's mechanism claim depends on r being sufficient to span safety-critical directions.
  • trajectory simulation learning rate ηθ = 5e-5 (reported as general lr)
    Eq. 9 step size controlling how corrupted the simulated states become; the appendix reports one lr (5e-5) without cleanly separating ηθ from ηϕ; determines the corruption intensity distribution the patch trains on.
  • patch learning rate ηϕ = 5e-5 (reported as general lr)
    Eq. 13 update step for the adapter; same reporting ambiguity as ηθ.
  • training iterations E / steps per iteration T = E=10, T unstated
    Controls the trajectory length and number of corrupted states; iteration count is ablated (Fig. 9) but T is not reported.
  • evaluation fine-tuning intensity grid = 5-30 epochs
    The user-FT depth range over which robustness is claimed; the LR/batch recipe for these user checkpoints is not specified.
axioms (5)
  • domain assumption Parameter-space linear superposition of task and safety vectors is a valid approximation for LLM behavior
    The entanglement analysis and merging equations (Eqs. 1-3, §3.1) inherit Task Arithmetic [16]/TIES [22]; the paper adds no justification beyond citing prior work.
  • domain assumption Refusal capability remains anchored along a small number of critical parameter directions even after full-parameter fine-tuning
    §4.4.2 justifies the low-rank patch via [25] ("refusal... mediated by a single direction"); load-bearing for the choice of LoRA r=64 instead of a full-parameter patch.
  • domain assumption Trajectories simulated on surrogate data (BeaverTails harmful + WizardLM benign, Eqs. 8-9) are representative of real user fine-tuning trajectories
    The OOD generalization claim (§5.2) depends on the simulated corruption distribution covering unseen datasets and intensities; the authors bound this regime verbally in §7.2 without a formal criterion.
  • domain assumption Guard-classifier judgments (LlamaGuard / Qwen3Guard on StrongReject prompts) are a valid safety measure
    All safety numbers, including the 100% claims, are discriminator classifications (§5.1); a guard-specific bias or bypass would change the reported values uniformly.
  • standard math Standard deep-learning background: response-only causal LM loss (Eq. 7), AdamW, LoRA behave as assumed
    Used throughout §4; standard, uncontroversial.

pith-pipeline@v1.3.0-alltime-deepseek · 22703 in / 19619 out tokens · 174192 ms · 2026-08-02T09:53:47.765279+00:00 · methodology

0 comments
read the original abstract

Fine-Tuning-as-a-Service (FTaaS) platforms let users train large language models (LLMs) on customized tasks, but this pipeline could erode models' safety alignment. In practice, service providers need to recover models' safety without re-running full alignment, or destroying the utility gained from customized tasks. A line of existing work refers to model parameter merging, which adds a safety patch on the fine-tuned model parameters to shift the model away from unsafe tendencies. However, this merging-based paradigm is fundamentally bottlenecked by task-safety update entanglement: downstream task updates and the safety patch often overlap in their dominant directions, so the merge strength is intrinsically hard to calibrate. If the safety vector is scaled too weakly, harmful components could still dominate, preventing the model from returning to a safe region; if it is scaled too aggressively, it suppresses task-relevant directions and degrades utility. To solve this problem, we shift the focus of merging-based methods from designing online merging operators to offline patch learning, and seek a safety patch that minimally interferes with task-relevant directions while retaining decisive control over unsafe behaviors. We propose TRACE, a trajectory-based safety patch learning framework that (i) simulates harmful tuning trajectories to generate progressively corrupted states, and (ii) optimizes a plug-in patch to recover safety while maintaining utility across varying corrupted base states. Across six benchmarks and two models, TRACE consistently dominates the safety-utility frontier. TRACE reaches nearly 100% safety on all settings, while maintaining comparable utility to the undefended fine-tuned model.

Figures

Figures reproduced from arXiv: 2607.16242 by Boxi Yu, Changyue Li, Jialin Wu, Jiaming He, Pinjia He, Youliang Yuan, Zhicong Huang.

Figure 1
Figure 1. Figure 1: Post-training safety recovery in Fine-Tuning-as-a-Service deploy [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Global safety-utility trade-offs across methods. Baselines trace unstable frontiers when their repair strength is swept, while TRACE occupies a [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of TRACE. Offline, the provider alternates between simulated harmful fine-tuning on the base model and adapter-only safety recovery [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Safety rate across malicious user fine-tuning intensity (5–30 epochs). TRACE remains stable under progressively stronger corruption. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task accuracy deviation (∆) between TRACE and No Defense across benign user fine-tuning intensity. increased dissimilarity triggers SafeDelta’s projection mech￾anism, and suppresses update components that deviate from the safety patch direction, thereby removing more harmful content. However, this stronger suppression simultaneously destroys co-trained task representations: SafeDelta drops MixedGSM8K on Ll… view at source ↗
Figure 6
Figure 6. Figure 6: Safety-utility Pareto frontiers. Each baseline curve is traced by [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Integrating existing baselines with TRACE by substituting their standard safety patch with the learned patch. Dashed curves denote the original [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Task utility across fine-tuning depth for all baselines on three benchmarks (SamSum, SQL, GSM8K) and two models (Llama and Qwen). TRACE [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Safety rate on PureBad as the number of TRACE training [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 19 linked inside Pith

  1. [1]

    Vaccine: Perturbation-aware align- ment for large language models against harmful fine-tuning attack,

    T. Huang, S. Hu, and L. Liu, “Vaccine: Perturbation-aware align- ment for large language models against harmful fine-tuning attack,” Advances in Neural Information Processing Systems, vol. 37, pp. 74 058–74 088, 2024

    2024

  2. [2]

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,

    T. Huang, S. Hu, F. Ilhan, S. Tekin, and L. Liu, “Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,” inInternational Conference on Learning Representa- tions, vol. 2025, 2025, pp. 67 202–67 226

    2025

  3. [3]

    Salora: Safety-alignment preserved low-rank adaptation,

    M. Li, W. M. Si, M. Backes, Y . Zhang, and Y . Wang, “Salora: Safety-alignment preserved low-rank adaptation,”arXiv preprint arXiv:2501.01765, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Understanding and preserving safety in fine-tuned llms,

    J. Zhang, Y . Hu, K. Chen, L. He, J. Ma, J. Lou, D. Li, J. Liu, X. Yang, and R. Jia, “Understanding and preserving safety in fine-tuned llms,” arXiv preprint arXiv:2601.10141, 2026

    arXiv 2026

  5. [5]

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,

    R. Bhardwaj, D. A. Do, and S. Poria, “Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 14 138–14 149

    2024

  6. [6]

    Enchtable: Unified safety alignment transfer in fine-tuned large language models,

    J. Wu, K. Li, Z. Huang, X. Li, X. Wang, and C. Hong, “Enchtable: Unified safety alignment transfer in fine-tuned large language models,” arXiv preprint arXiv:2511.09880, 2025

    arXiv 2025

  7. [7]

    Safe lora: The silver lining of reducing safety risks when finetuning large language models,

    C.-Y . Hsu, Y .-L. Tsai, C.-H. Lin, P.-Y . Chen, C.-M. Yu, and C.-Y . Huang, “Safe lora: The silver lining of reducing safety risks when finetuning large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 65 072–65 094, 2024

    2024

  8. [8]

    Safe delta: Consistently preserving safety when fine-tuning llms on diverse datasets,

    N. Lu, S. Liu, J. Wu, W. Chen, Z. Zhang, Y .-S. Ong, Q. Wang, and K. Tang, “Safe delta: Consistently preserving safety when fine-tuning llms on diverse datasets,”arXiv preprint arXiv:2505.12038, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Supervised fine-tuning,

    OpenAI, “Supervised fine-tuning,” https://developers.openai.com/api/ docs/guides/supervised-fine-tuning, 2026, accessed: 2026-05-14

    2026

  10. [10]

    Tune gemini models by using supervised fine-tuning,

    Google Cloud, “Tune gemini models by using supervised fine-tuning,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/ models/gemini-use-supervised-tuning, 2026, accessed: 2026-05-14

    2026

  11. [11]

    Customize a model with fine-tuning,

    Microsoft, “Customize a model with fine-tuning,” https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/ fine-tuning?tabs=oai-sdk&pivots=programming-language-studio, 2026, accessed: 2026-05-14

    2026

  12. [12]

    Customize your model to improve its performance for your use case,

    Amazon Web Services, “Customize your model to improve its performance for your use case,” https://docs.aws.amazon.com/bedrock/ latest/userguide/custom-models, 2026, accessed: 2026-05-14

    2026

  13. [13]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Shadow alignment: The ease of subverting safely-aligned language models,

    X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y . Wang, X. Zhao, and D. Lin, “Shadow alignment: The ease of subverting safely-aligned language models,”arXiv preprint arXiv:2310.02949, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” arXiv preprint arXiv:2212.04089, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 30 988–31 043

    2024

  18. [18]

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference,

    J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai, B. Zheng, T. Qiu, B. Li, and Y . Yang, “Pku-saferlhf: Towards multi-level safety alignment for llms with human preference,”arXiv preprint arXiv:2406.15513, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Samsum corpus: A human-annotated dialogue dataset for abstractive summarization,

    B. Gliwa, I. Mochol, M. Biesek, and A. Wawer, “Samsum corpus: A human-annotated dialogue dataset for abstractive summarization,” in Proceedings of the 2nd Workshop on New Frontiers in Summarization, 2019, pp. 70–79

    2019

  20. [20]

    sql-create-context dataset,

    b mc2, “sql-create-context dataset,” 2023. [Online]. Available: https://huggingface.co/datasets/b-mc2/sql-create-context

    2023

  21. [21]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakanoet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    TIES- merging: Resolving interference when merging models,

    P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “TIES- merging: Resolving interference when merging models,” inThirty- seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=xtaX3WyCj1

    2023

  23. [23]

    Beavertails: Towards improved safety align- ment of llm via a human-preference dataset,

    J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y . Wang, and Y . Yang, “Beavertails: Towards improved safety align- ment of llm via a human-preference dataset,”Advances in Neural Information Processing Systems, vol. 36, pp. 24 678–24 704, 2023

    2023

  24. [24]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  25. [25]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024

    2024

  26. [26]

    Qwen3. 5-omni technical report,

    Q. Team, “Qwen3. 5-omni technical report,”arXiv preprint arXiv:2604.15804, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Wizardlm: Empowering large language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, “Wizardlm: Empowering large language models to follow complex instructions,”arXiv preprint arXiv:2304.12244, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Safety at one shot: Patching fine-tuned llms with a single instance,

    J. Zhang, L. He, K. Chen, J. Lou, J. Liu, X. Yang, and R. Jia, “Safety at one shot: Patching fine-tuned llms with a single instance,”arXiv preprint arXiv:2601.01887, 2026

    arXiv 2026

  29. [29]

    A strongreject for empty jailbreaks,

    A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer, “A strongreject for empty jailbreaks,” 2024

    2024

  30. [30]

    Llama guard: Llm-based input-output safeguard for human-ai conversations,

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggineet al., “Llama guard: Llm-based input-output safeguard for human-ai conversations,”arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Qwen3guard technical report,

    H. Zhao, C. Yuan, F. Huang, X. Hu, Y . Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Linet al., “Qwen3guard technical report,”arXiv preprint arXiv:2510.14276, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, 2004, pp. 74–81

    2004

  33. [33]

    Training language models to follow instruc- tions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instruc- tions with human feedback,” inProceedings of the 36th International Conference on N...

    2022

  34. [34]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” inThirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=HPuSIXJaa9

    2023

  35. [35]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighanet al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,”arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Constitutional ai: Harmlessness from ai feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnonet al., “Constitutional ai: Harmlessness from ai feedback,”arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Lora fine-tuning effi- ciently undoes safety training in llama 2-chat 70b,

    S. Lermen, C. Rogers-Smith, and J. Ladish, “Lora fine-tuning effi- ciently undoes safety training in llama 2-chat 70b,”arXiv preprint arXiv:2310.20624, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  39. [39]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,

    M. Wortsman, G. Ilharco, S. Y . Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y . Carmon, S. Kornblith et al., “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” inInternational conference on machine learning. PMLR, 2022, pp. 23 965–23 998

    2022

  40. [40]

    Language models are super mario: absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: absorbing abilities from homologous models as a free lunch,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  41. [41]

    Toy models of superposition,

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chenet al., “Toy models of superposition,”arXiv preprint arXiv:2209.10652, 2022

    Pith/arXiv arXiv 2022

  42. [42]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014

    Pith/arXiv arXiv 2014

  43. [43]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” ArXiv, vol. abs/2307.15043, 2023. [Online]. Available: https: //api.semanticscholar.org/CorpusID:260202961

    Pith/arXiv arXiv 2023

  44. [44]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  45. [45]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    S. Gugger, L. Debut, T. Wolf, P. Schmid, Z. Mueller, S. Mangrulkar, M. Sun, and B. Bossan, “Accelerate: Training and inference at scale made simple, efficient and adaptable.” https://github.com/huggingface/ accelerate, 2022

    2022

  46. [46]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  47. [47]

    Model-agnostic meta- learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta- learning for fast adaptation of deep networks,” inProceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, ser. Proceedings of Machine Learning Research, D. Precup and Y . W. Teh, Eds. PMLR, 2017, pp. 1126–1135. [Online]. Available: h...

    2017

  48. [48]

    Translate Natural Lan- guage Query into SQL Query considering the provided Context. ## Context: {context} ## Natural Language Query: {query}

    Implementation Details Prototype.All experiments are conducted on the PyTorch platform [44] using 8 NVIDIA H800 GPUs with 80 GB of memory each. We implement TRACE following Algorithm 1, with the LoRA rank set to r= 64 , and a dropout rate of p= 0.1. The LoRA adapter is applied to all linear layers in each transformer block, covering both the self-attentio...

  49. [49]

    We evaluate on all six model and benchmark combinations across both Llama and Qwen

    Full Utility Results across Fine-Tuning Depth This section presents the complete task accuracy for all baselines as user fine-tuning deepens from 5 to 30 epochs, complementing the utility deviation analysis in Section 5.3. We evaluate on all six model and benchmark combinations across both Llama and Qwen. TRACE preserves utility across all fine-tuning dep...

  50. [50]

    With- out trajectory simulation, the safety patch is obtained by performing recovery training solely on the final corrupted TABLE 5

    Ablation Study This section isolates the contribution of the trajectory simulation by removing it from the training pipeline. With- out trajectory simulation, the safety patch is obtained by performing recovery training solely on the final corrupted TABLE 5. ABLATION ON THE TRAJECTORY SIMULATION. REMOVING IT COLLAPSES SAFETY TO NEAR-ZERO LEVELS COMPARABLE...

  51. [51]

    We train TRACE for 1 to 10 iterations and evaluate each resulting patch on PureBad for both Llama and Qwen

    Effect of Training Iterations This section examines how the number of training iterations in Algorithm 1 affects the safety performance of the learned patch. We train TRACE for 1 to 10 iterations and evaluate each resulting patch on PureBad for both Llama and Qwen. Results are shown in Figure 9. TRACE converges rapidly and reaches near-perfect safety with...

  52. [52]

    This section discusses the shared motivation and key distinctions between two methods

    Connection to Meta-Learning TRACE’s alternating optimization (Algorithm 1) superfi- cially resembles the inner-outer loop structure of gradient- based meta-learning methods such as MAML [47]. This section discusses the shared motivation and key distinctions between two methods. Both TRACE and meta-learning aim to learn parameters that generalize across va...