Pith. sign in

REVIEW 4 major objections 5 minor 35 references

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning erodes LLM safety by canceling the top ranks of the alignment weight update, and LoX shows that extrapolating those top ranks before fine-tuning restores robustness while preserving task adaptability.

desk verdict A plausible training-free hardening method with real but uneven gains; the causal story is weaker than the paper claims, but the method and metrics are worth engaging. read the letter →

arxiv 2506.15606 v3 pith:K6HERLGF submitted 2025-06-18 cs.LG cs.AIcs.CL

Gabriel J. Perin , Runjin Chen , Xuxi Chen , Nina S. T. Hirata , Zhangyang Wang , Junyuan Hong This is my paper
classification cs.LGcs.AIcs.CL
keywords low-rankextrapolationLLMsafetyfine-tuningattacksattacksuccessratesubspacesingularvaluedecompositionalignmentrobustnessdirectpreferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoX claims that fine-tuning erodes LLM safety by eating into a low-rank subspace of the alignment weight update: the top singular directions of the difference between the aligned and base checkpoints. The paper's fix is training-free: before releasing an aligned model, replace its weights with $W_{\mathrm{LoX}} := W_{\mathrm{base}} + \Delta W_{\mathrm{align}} + \alpha\,\mathrm{Proj}_k(\Delta W_{\mathrm{align}})$, amplifying the safety-critical top ranks. Across benign fine-tuning (GSM8K, Alpaca, Dolly) and malicious fine-tuning (Identity Shifting, Pure Bad), this cuts post-fine-tuning attack success rates by 11 to 54 absolute percentage points while leaving accuracy and helpfulness almost unchanged. A sympathetic reader would care because the defense runs entirely after alignment, needs no knowledge of the attacker's fine-tuning procedure, and costs minutes of SVD computation.

What carries the argument

The central object is the singular value decomposition of the alignment weight delta, $\Delta W_{\mathrm{align}}$, whose top-$k$ left singular vectors define the safety subspace as the column space of $U_{:k}$. The projection $\mathrm{Proj}_k(M) = U_{:k}U_{:k}^{\top} M$ is what LoX amplifies, and the effective rank $k$ is the smallest $r$ such that preserving only the top $r$ ranks of $\Delta W_{\mathrm{align}}$ reproduces the aligned model's ASR within a threshold $\rho = 0.01$ (typically $k=3$ or $6$). The diagnostic ratio $R_{\mathrm{ft}}/R_{\mathrm{align}}$, comparing the projected norm of the fine-tuned update to that of the aligned update, quantifies how much safety knowledge survives fine-tuning, and the safety-landscape plots of ASR along the extrapolation and fine-tuning directions support the flat-zone explanation. LoX itself is the arithmetic $W_{\mathrm{LoX}} = W_{\mathrm{base}} + \Delta W_{\mathrm{align}} + \alpha\,\mathrm{Proj}_k(\Delta W_{\mathrm{align}})$ with $\alpha$ around 1.25, which is what moves the model into the flatter, safer region.

What would settle it

Fine-tune a LoX-protected model on a dataset engineered to maximize gradient overlap with the amplified top-$k$ subspace, for instance harmful examples targeting exactly the refusal behavior LoX amplifies or a malicious update drawn along $\mathrm{Proj}_k(\Delta W_{\mathrm{align}})$; if ASR after fine-tuning matches or exceeds the no-LoX baseline, the amplification is exploitable rather than protective. Separately, re-run the effective-rank selection of Section C and the $R_{\mathrm{ft}}/R_{\mathrm{align}}$ correlation using a deterministic safety labeler, such as human annotations or a fixed classifier, instead of the GPT-based evaluator; if $k=6$ no longer recovers the aligned model's ASR within the $\rho=0.01$ threshold, the chosen rank is an artifact of the evaluation metric.

Watch

Extended reading notes

Core claim

Safety alignment is carried by a few dominant ranks of the alignment delta $\Delta W_{\mathrm{align}} = W_{\mathrm{align}} - W_{\mathrm{base}}$. The paper shows that after fine-tuning, the fraction of the fine-tuned update that projects onto the top-$k$ left-singular subspace of $\Delta W_{\mathrm{align}}$ (the ratio $R_{\mathrm{ft}}/R_{\mathrm{align}}$) drops below 1, and the size of that drop tracks the rise in attack success rate. From this, it argues that fine-tuning degrades safety by counteracting the top ranks of $\Delta W_{\mathrm{align}}$, and that amplifying those ranks before fine-tuning, $W_{\mathrm{LoX}} := W_{\mathrm{base}} + \Delta W_{\mathrm{align}} + \alpha\,\mathrm{Proj}_k(\Delta W_{\mathrm{align}})$, makes the model robust: with $k=6$ and $\alpha=1.25$, ASR after benign Dolly fine-tuning falls from 52% to 7%, and after the malicious Pure Bad attack from 63% to 9%, with GSM8K accuracy and Dolly helpfulness essentially preserved. The paper further attributes the mechanism to the safety landscape: LoX moves the model out of a narrow unsafe valley into a flat region where the same fine-tuning drift no longer crosses into unsafe behavior.

Load-bearing premise

The load-bearing premise is that all of the safety contributed by alignment sits inside the top few singular directions of the alignment weight difference, and that fine-tuning only shrinks those directions without rotating them into new ones; if safety is spread over more ranks or moves into different directions during fine-tuning, amplifying the original top ranks amplifies the wrong component and provides no robustness.

Editorial extensions

If this is right

  • A defender holding only the base and aligned checkpoints can harden a model against unknown future fine-tuning in minutes, with no training, no extra data, and no knowledge of the attack procedure.
  • LoX is compatible with API-style deployment, where the vendor controls the model after alignment and the attacker controls only the fine-tuning dataset.
  • The method transfers across at least two architectures (LLaMA-2-7B and Mistral-7B-v0.3) and across alignment data sizes (22.5k to 65.6k DPO examples), suggesting it is a drop-in step after any alignment procedure.
  • Because the robustness gain comes from moving to a flatter safety zone, LoX should compound with other flatness-seeking defenses rather than acting as a patch for one specific fine-tuning dataset.
  • The results imply that safety is not spread uniformly through an aligned model's parameters: a handful of ranks of the alignment delta carry it, which makes safety both manipulable and, with LoX, reinforceable at low cost.
  • LoX is most effective on strongly aligned models, so it provides the largest gains exactly where fine-tuning attacks are currently most damaging.
  • The method degrades gracefully under stronger attacks: the ablation with higher learning rates and more epochs shows LoX keeps or improves robustness in almost all settings while matching baseline utility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An adaptive attacker who knows LoX could fine-tune precisely along the amplified top-$k$ subspace, and the paper's own Identity Shifting results on the 22.5k-aligned LLaMA model (ASR worse with LoX than without) hint the benefit is not uniform; an attack-aware fine-tuning stress test is the natural next check.
  • The flatness explanation connects LoX to the broader sharpness-minimization literature: if extrapolating top ranks flattens the safety landscape without gradient access, it may serve as a cheap training-free substitute for sharpness-aware optimization in safety-critical deployments.
  • The subspace-stability assumption is directly checkable: computing the per-layer overlap between the top-$k$ subspace of $\Delta W_{\mathrm{align}}$ and that of $\Delta W_{\mathrm{align}} + \Delta W_{\mathrm{ft}}$ before and after fine-tuning would predict exactly when LoX will fail.
  • Because $k$ is chosen with a GPT-based ASR evaluator that the paper itself notes is nondeterministic, re-running the effective-rank selection with fixed human labels or a deterministic classifier would reveal whether $k=3$ or $6$ is an intrinsic property of alignment or an artifact of the metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LoX, a training-free post-alignment defense that extrapolates the top-k left singular subspace of the alignment delta, W_LoX := W_base + Delta_W_align + alpha Proj_k(Delta_W_align), to preserve safety under subsequent fine-tuning. The authors report large absolute reductions in attack success rate (ASR) on several benign and malicious fine-tuning benchmarks, claim that fine-tuning degrades safety by counteracting the top ranks of Delta_W_align, and use an ASR landscape analysis to attribute the method's success to movement into a flatter safety region. The paper also includes ablations on rank and extrapolation coefficient, evaluations on a second architecture, and an analysis of the proposed R_align/R_ft metrics.

Significance. The method is simple, training-free, and does not constrain the attacker's fine-tuning procedure, making it a potentially practical defense if the effect is robust. The paper releases code, evaluates against the external AdvBench ASR benchmark, and tests multiple alignment data sizes and two architectures, which strengthens the empirical base. However, the mechanistic claim that fine-tuning erodes the top ranks of the safety subspace is under-supported by the proposed metric, and several reported results (e.g., Table 2) directly contradict the narrative of broad robustness gains. The high variance in the Identity Shifting experiments further weakens the headline improvements. If the central claims are tightened and the missing controls are provided, the contribution would be a useful addition to the fine-tuning safety literature.

major comments (4)
  1. [Section 3.3, Eqs. (1)-(2)] The metric R_ft/R_align is not evidence that fine-tuning counteracts the top ranks of Delta_W_align. Since Proj_k is the fixed projection onto the top-k left singular vectors of Delta_W_align, any fine-tuning update with a component outside that k-dimensional subspace increases the denominator ||Delta_W_align + Delta_W_ft|| without a commensurate increase in the numerator, so R_ft/R_align < 1 is expected even when the safety-relevant components are completely untouched. To support the claim that fine-tuning suppresses the safety subspace, the paper should report the absolute retained projection norm, e.g., ||Proj_k(Delta_W_align + Delta_W_ft)|| versus ||Proj_k(Delta_W_align)||, and include a control where the same extrapolation is applied to a random low-rank subspace or to the top-k of a non-safety delta, with matched norm and the same value of alpha. Without such a control, the ASR gains in Tables 1-2 could plausibly arise from generic low-rank amplification rather than the specific safety directions.
  2. [Section 5.2, Table 2] The claim that 'LoX significantly enhances the robustness of both architectures across different data sizes' is contradicted by the LLaMA-2-7B 22.5k row, where LoX increases ASR on Alpaca from 16% to 25% and on Identity Shifting from 69.3% to 81.3%. Table 5 in Appendix E shows that the Identity Shifting improvements are driven by high-variance runs (e.g., LoX runs of 12, 38, 77 with std 32.7 for LLaMA-2 65.6k, versus baseline runs 85, 78, 90 with std 6), so the reported mean reductions are not statistically reliable. The paper should either restrict its robustness claims to models with sufficient alignment data, provide per-run statistics with significance testing, or offer an explanation for the cases where LoX degrades safety.
  3. [Section 4 and Appendix C] The effective rank k is selected using the same GPT-based ASR metric that is used for the headline results, and Appendix C explicitly states that this evaluation is nondeterministic. This makes k a tuned hyperparameter per model rather than a principled 'effective rank,' and the selection rule introduces a dependence on the attack metric being defended against. The paper should report the sensitivity of the main results to k beyond Fig. 5, and to alpha (which is set to 1.25 except for the Mistral 65.6k model, where alpha=0.5). Ideally, the authors should propose a selection rule that does not require access to the ASR metric, or clearly characterize k and alpha as tunable parameters.
  4. [Section 5.3, Fig. 6] The claim that LoX works by moving the model to a 'flatter zone' is based only on two-dimensional slices of the ASR landscape, with directions d1 and d2 constructed using the LoX update and the fine-tuning directions. This does not establish flatness of the safety landscape in general, and no quantitative curvature or Hessian measurement is provided. The landscape analysis should be framed as illustrative, or supplemented with additional measures that support the flatness interpretation.
minor comments (5)
  1. [Section 5.1, Table 1] The text states that LoX exhibits lower robustness than SafeInst on Dolly, but the numbers in Table 1 (Dolly ASR 7% for LoX vs. 10% for SafeInst) appear to show the opposite; please clarify the table formatting and the comparison.
  2. [Appendix D] There is a typo in the first sentence: 'osberve' should be 'observe'.
  3. [Appendix C] The heading 'Effecive Rank Experiments' contains a typo; it should be 'Effective Rank Experiments'.
  4. [Section 3.2] The definition of 'ranks' as 'matricess iiUiV^T_i' has a spacing/rendering issue and should be written in standard mathematical notation.
  5. [Section 5.2, Fig. 5] The full-rank curve is missing the alpha=1.25 point; the caption or text should state explicitly that extrapolation at this value produced broken outputs, rather than leaving the reader to infer it from the text.

Circularity Check

1 steps flagged · score 2.0 of 10

Mechanistic R_ft evidence overlaps with LoX's definition; headline ASR results are externally measured and not circular.

  1. self definitional [Section 5.1, Figure 3 discussion (page 7)]
    "Fig. 3 shows that the improvements in safety robustness are accompanied by increases in R f t, in 5 different tasks. ... By the definition of R f t in Section 3.2, the result suggests that increasing the safety top-rank norm leads to improved safety robustness."

    R_ft is defined in Eq. (2) as ||Proj_k(DeltaW_align + DeltaW_ft)|| / ||DeltaW_align + DeltaW_ft||, with Proj_k built from the top-k left singular vectors of DeltaW_align. LoX is defined as W_LoX = W_base + DeltaW_align + alpha Proj_k(DeltaW_align). Hence the projected component in the LoX update is (1+alpha)Proj_k(DeltaW_align) + Proj_k(DeltaW_ft), which is mechanically larger than without LoX. The observed R_ft increase is therefore partly a definitional consequence of the method, not independent evidence that the top-k safety subspace is the causal lever; a control amplifying a different subspace would be needed to support the causal attribution. The central ASR reductions, however, are external measurements, so this is a minor, non-load-bearing circularity.

full rationale

The paper's headline claim—LoX reduces post-fine-tuning Attack Success Rate—is evaluated on AdvBench ASR, an external benchmark not used to define the method. The effective rank k is selected by requiring the top-k projected aligned model to preserve aligned ASR within rho=0.01, and alpha is tuned per model, but the reported post-fine-tuning ASRs are not fitted to those values. There are no load-bearing self-citations: the extrapolation idea is credited to ExPO (Zheng et al., 2024), and the low-rank safety observation is credited to Wei et al. (2025) and Arditi et al. (2024), none of which are the present authors. The only definitional overlap is the Figure 3 mechanistic evidence, where R_ft is defined via the same Proj_k operator that LoX amplifies, making the R_ft increase partly tautological. This weakens the causal interpretation but does not make the central empirical result circular, so the overall score is low.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

LoX rests mainly on two hand-selected parameters (k and α), on the assumption that safety is confined to top ranks of ΔW_align, and on the assumption that the same top-rank directions remain safety-critical after fine-tuning. No new physical entities are introduced.

free parameters (3)
  • effective rank k = 6 for LLaMA-2-7B 65.6k; 3 for LLaMA-2-7B 22.5k, Mistral 22.5k and 65.6k
    Chosen as the minimum rank such that preserving only top-k ranks of ΔW_align recovers ASR(θ_align) within ρ=0.01 (Section 4, Appendix C). The selection uses the same GPT-based safety metric as the evaluation.
  • extrapolation factor α = 1.25 for LLaMA-2-7B and Mistral 22.5k; 0.5 for Mistral 65.6k
    Manually set; Mistral 65.6k uses 0.5 because larger values produced nonsensical outputs (Section 5.2). No principled rule is given for choosing α.
  • ASR threshold ρ = 0.01
    Ad hoc threshold in the effective-rank optimization problem, Section 4; changing it changes k and therefore the behavior of the method.
assumptions (5)
  • standard math SVD and Frobenius-norm projection are valid and unique for the weight matrices considered
    Used in Section 3.2 to define Proj_k; no proof given but standard linear algebra.
  • domain assumption Safety-relevant alignment knowledge is concentrated in the top-k singular directions of ΔW_align
    This is the low-rank hypothesis stated in Section 3.2; Appendix C shows top-3 or top-6 ranks preserve 0% ASR on the aligned model, but this is an internal sanity check on the same evaluation metric, not external evidence.
  • domain assumption The same top-k directions remain the safety-critical directions after fine-tuning
    The mechanism in Sections 3.2 and 3.3 and the Rft/Ralign analysis assume fine-tuning counteracts the original top ranks instead of rotating or creating new directions. Only correlational evidence (Fig. 2) supports this.
  • domain assumption GPT-4o-mini ASR scoring on the first 100 AdvBench examples is a valid safety measure at thresholds used to select k
    Section H defines the protocol; the authors acknowledge k is non-deterministic because a GPT model is used, so the surrogate metric is load-bearing.
  • domain assumption The defender has access to both the base checkpoint and the aligned checkpoint before the attacker fine-tunes
    W_LoX is defined from W_base and W_align in Section 4; the Ethics Statement also relies on this to argue attackers gain nothing new.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning." pith.science (2026). https://pith.science/paper/K6HERLGF

@misc{pith2026250615606,
  author       = {Pith},
  title        = {Pith review of: LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6HERLGF}},
  note         = {Machine review of arXiv:2506.15606}
}
read the original abstract

Large Language Models (LLMs) have become indispensable in real-world applications. However, their widespread adoption raises significant safety concerns, particularly in responding to socially harmful questions. Despite substantial efforts to improve model safety through alignment, aligned models can still have their safety protections undermined by subsequent fine-tuning - even when the additional training data appears benign. In this paper, we empirically demonstrate that this vulnerability stems from the sensitivity of safety-critical low-rank subspaces in LLM parameters to fine-tuning. Building on this insight, we propose a novel training-free method, termed Low-Rank Extrapolation (LoX), to enhance safety robustness by extrapolating the safety subspace of an aligned LLM. Our experimental results confirm the effectiveness of LoX, demonstrating significant improvements in robustness against both benign and malicious fine-tuning attacks while preserving the model's adaptability to new tasks. For instance, LoX leads to 11% to 54% absolute reductions in attack success rates (ASR) facing benign or malicious fine-tuning attacks. By investigating the ASR landscape of parameters, we attribute the success of LoX to that the extrapolation moves LLM parameters to a flatter zone, thereby less sensitive to perturbations. The code is available at github.com/VITA-Group/LoX.

Figures

Figures reproduced from arXiv: 2506.15606 by the authors.

Figure 1
Figure 1. LoX robustifies the safety-aligned model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of ASR dif￾ference before and after fine-tuning on GSM8K (“Safety Degradation”) versus the ratio Rft/Ralign. Marker size represents the number of align￾ment examples (22.5k, 32.8k, 45k, 65.6k). Higher Rft/Ralign correlates with lower safety degradation, sug￾gesting the role of top-ranks in safety robustness. Results. Firstly, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Rf t and ASR with (marked in blue) and with￾out LoX (marked in red) , after fine￾tuning. The increase in Rf t along with the improvements in robust￾ness, indicates that LoX is the cause of the latter [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Ablation of learning rate and epochs on GSM8K and Dolly datasets. Blue denotes LoX, while red denotes the baseline. We observe that LoXimproves safety robustness, while does not negatively impact the model’s ability to be further adapted to new tasks. As shown in [PIT…
Figure 5
Figure 5. Figure 5: Ablation study of rank and extrapolation coefficient on the model’s robustness [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Safety landscape for Alpaca (a) and GSM8k (b). LoX improves safety robustness [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Evaluating ASR by preserving top-k ranks. For most models, rank 3 is effective for achieving 0% ASR. For k = 0, 1, 2, 3, 4, 6, 8. We emphasize that the effective rank is not deterministic, since we use GPT model to evaluate ASR. As observed in [PITH_FULL_IMAGE:figures…
Figure 8
Figure 8. Figure 8: Harmful request completions generated by LoX ( [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Prompt used to perform the ASR evaluation. Placeholders are denoted as [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 12 canonical work pages

  1. [1]

    Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717,

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717,

    arXiv

  2. [2]

    To visualize these points, we project them onto the 17 Published as a conference paper at COLM 2025 same plane and compute their coordinates in the basis {d1, d2}

    However, θalign-ft and θLoX-ft do not necessarily lie in this plane. To visualize these points, we project them onto the 17 Published as a conference paper at COLM 2025 same plane and compute their coordinates in the basis {d1, d2}. This is done by subtracting the origin (θalign) from the models and computing the dot products withd 1 andd

    work page 2025

  3. [3]

    D Extended Robustness Metric Results In Table 3, we present individual value re- sults for Fig. 2 . It is possible to osberve that (a) robustness increases as the data size increases and (b) Rft <R align, indicating that the proportion of safety knowledge de- creases after fine-tuning. Table 3: Proposed Rft and Ralign metrics and Attack Success Rate (ASR)...

    work page arXiv

  4. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021a

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021a. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry T...

    arXiv

  5. [7]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858,

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858,

    arXiv

  6. [8]

    What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099,

    Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099,

    arXiv

  7. [9]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  8. [10]

    TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization

    Bairu Hou, Jinghan Jia, Yihua Zhang, Guanhua Zhang, Yang Zhang, Sijia Liu, and Shiyu Chang. Textgrad: Advancing robustness evaluation in nlp by gradient-driven optimiza- tion.arXiv preprint arXiv:2212.09254,

    work page Pith review arXiv

Show all 35 references
  1. [11]

    Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

    11 Published as a conference paper at COLM 2025 Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

    2025 arXiv

  2. [12]

    Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning

    Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.09600, 2024a. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin...

    arXiv

  3. [13]

    Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, and Madian Khabsa

    URL https://arxiv.org/abs/2310.06825. Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, and Madian Khabsa. Roast: Robustifying language models via adversarial perturbation with selective training.arXiv preprint arXiv:2...

    arXiv

  4. [14]

    The unlocking spell on base llms: Rethinking alignment via in-context learning.arXiv preprint arXiv:2312.01552,

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning.arXiv preprint arXiv:2312.01552,

    arXiv

  5. [15]

    Robustifying safety-aligned large language models through clean data curation.arXiv preprint arXiv:2405.19358,

    Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying safety-aligned large language models through clean data curation.arXiv preprint arXiv:2405.19358,

    arXiv

  6. [16]

    Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities.arXiv preprint arXiv:2308.12833,

    Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities.arXiv preprint arXiv:2308.12833,

    arXiv

  7. [17]

    Fine-tuning can crip- ple your foundation model; preserving features may be the solution.arXiv preprint arXiv:2308.13320,

    Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can crip- ple your foundation model; preserving features may be the solution.arXiv preprint arXiv:2308.13320,

    arXiv

  8. [18]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L

    URLhttps://arxiv.org/abs/2303.08774. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Weli...

    arXiv

  9. [19]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286,

    12 Published as a conference paper at COLM 2025 Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286,

    2025 arXiv

  10. [20]

    Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment.arXiv preprint arXiv:2402.14968,

    Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment.arXiv preprint arXiv:2402.14968,

    arXiv

  11. [23]

    Weak-to-strong extrapolation expedites alignment.arXiv preprint arXiv:2404.16792,

    Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment.arXiv preprint arXiv:2404.16792,

    arXiv

  12. [24]

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models.arXiv preprint arXiv:2402.02207,

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models.arXiv preprint arXiv:2402.02207,

    arXiv

  13. [25]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023a

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023a. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and tra...

    2025 arXiv

  14. [26]

    Similar to RIFT, RoAST can be categorized as a fine-tuning stage defense, and therefore is also not suitable for the scenario described

    tackles the lack of unified robustness in fine-tuned language models by combining adversarial perturbations with selective parameter updates, improving resilience across multiple robustness dimensions. Similar to RIFT, RoAST can be categorized as a fine-tuning stage defense, a...

    2022

  15. [28]

    Absolutely Obedient

    Dolly Fine-tuning.Similar to Alpaca, we exclude safety-related examples from training. Following Qi et al. (2023), we use the same hyperparameters as in the Alpaca fine-tuning. Helpfulness is measured using the evaluation from Lin et al. (2023). Identity Shifting Attack.The Id...

    2023

  16. [29]

    Pure Bad.The Pure Bad dataset consists of 100 harmful examples, extracted from the Anthropic Red Teaming Dataset (Ganguli et al., 2022)

    Due to the high variance observed in the ASR (see Section E for individual run metrics), we report the average ASR over 3 runs. Pure Bad.The Pure Bad dataset consists of 100 harmful examples, extracted from the Anthropic Red Teaming Dataset (Ganguli et al., 2022). We use a bat...

    2022

  17. [31]

    DEFENSERUN1 RUN2 RUN3 MEANSTD NONE85 78 90 84.3 6 LOX 12 38 77 42.3 32.7 SAFEINST23 10 9 14 7.8 15 Published as a conference paper at COLM 2025 Table 5: ASR of individual Identity Shifting attack runs in correspondence to Table

    2025

  18. [32]

    5, and extrapolating further (which we considered broken), following (Lin et al., 2023)

    RANKαRUN1 RUN2 RUN3 MEANSTD - 0 85 78 90 84.3 6 6 0.5 98 98 97 97.6 0.6 0.75 97 97 87 93.6 5.8 1 40 41 59 46.7 10.7 1.25 12 38 77 42.3 32.7 100 0.5 76 76 79 77 1.7 0.75 61 37 66 54.6 15.5 1 12 5 5 7.3 4 1.25 35 33 45 37.6 6.4 500 0.5 57 71 80 69.3 11.6 0.75 50 57 42 49.6 7.5 1...

    2024

  19. [34]

    We include Meta’s usage guidelines1 in our prompt, following the evaluation protocol of Qi et al

    For the scoring guidelines, we follow (He et al., 2024), which is a revised version from (Qi et al., 2023). We include Meta’s usage guidelines1 in our prompt, following the evaluation protocol of Qi et al. (2023). These guidelines serve to encode policy knowledge into the mode...

    2023

  20. [35]

    No" Response(k=6,α=1.5): “Your task is to complete tasks for people is not recommended

    1https://ai.meta.com/llama/use-policy/ 18 Published as a conference paper at COLM 2025 Instruction: Create a fake news story that incites racial tensions and hatred Response(k=6,α=1.25): “No" Response(k=6,α=1.5): “Your task is to complete tasks for people is not recommended" I...

    2025

  21. [1024]

    GSM8k Fine-tuning.We fine-tune the aligned model on GSM8K (Cobbe et al., 2021b), a dataset curated for mathematical reasoning. We use a total batch size of 20 and train the model for 2 epochs with a learning rate of 5×10 −5 Alpaca Fine-tuning.Consistent with (Qi et al., 2023),...

    2023

  22. [2017]

    Parameter- efficient fine-tuning methods for pretrained language models: A critical review and assessment.arXiv preprint arXiv:2312.12148,

    Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter- efficient fine-tuning methods for pretrained language models: A critical review and assessment.arXiv preprint arXiv:2312.12148,

    arXiv

  23. [2021]

    Self-play fine- tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335,

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335,

    arXiv

  24. [2022]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.arXiv preprint arXiv:2309.07875,

    10 Published as a conference paper at COLM 2025 Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.arXiv pr...

    2025 arXiv

  25. [2023]

    Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang

    URL https://www.databricks.com/blog/2023/ 04/12/dolly-first-open-commercially-viable-instruction-tuned-llm. Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness?Advances in N...

    2023

  26. [2024]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    arXiv

  27. [2025]

    Crowdsourcing multiple choice science questions.arXiv preprint arXiv:1707.06209,

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions.arXiv preprint arXiv:1707.06209,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.