Pith. sign in

REVIEW 4 major objections 4 minor 60 references

Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LARF claims that filtering fine-tuning data by representation similarity at a safety-sensitive layer removes benign-looking samples that degrade LLM safety alignment.

desk verdict A useful, cheap data filter whose mechanism is unproven: the top-ranked samples are long point-by-point responses, and no length/format baseline exists to rule out a trivial explanation. read the letter →

arxiv 2507.18631 v2 pith:AY2RYV7V submitted 2025-07-24 cs.CR

Hao Li , Lijun Li , Zhenghao Lu , Xianyi Wei , Rui Li , Jing Shao , Lei Sha This is my paper · ORCID
classification cs.CR
keywords safetyalignmentfine-tuningdatafilteringrepresentationsimilaritysafety-sensitivelayersattacksuccessrateLLMLARF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning an aligned LLM on seemingly benign instruction data can still erode its safety guardrails. This paper claims that such datasets contain 'safety-degrading' samples—benign-looking instruction–response pairs that push the model toward complying with harmful prompts—and that these samples can be identified cheaply before training. The proposed method, LARF, locates the model layer whose parameter scaling most changes refusal behavior, then ranks each fine-tuning sample by how close its hidden representation at that layer sits to unsafe versus safe reference examples. Removing the top-ranked samples, the paper reports, largely restores the refusal behavior that fine-tuning would otherwise destroy, with no measurable loss on downstream task quality. If true, this gives practitioners a training-free pre-deployment audit tool for protecting alignment.

What carries the argument

The two load-bearing mechanisms are (1) safety-sensitive layer identification and (2) bidirectional representation scoring. Layer identification: for each candidate layer, parameters are scaled by (1±α) and the normalized change in refusal count on an overrejection dataset is computed; the layer with maximal normalized change rate k is deemed safety-sensitive. Scoring: at that layer, the final-token hidden representation of each data sample is compared by cosine similarity against the mean representation of unsafe reference examples (harmful instruction with compliant answer) and safe reference examples (same instructions with refusal); the safety-degrading score is the difference sim(unsafe) − sim(safe). The argument is that the refusal direction lives in this layer's representation space, so samples that sit closer to unsafe completions and farther from refusals are exactly the ones that, when used for fine-tuning, erode the model's rejection capability.

What would settle it

Construct a control set of 1,000 fine-tuning samples that match LARF's top-ranked samples in output length and point-by-point formatting but are drawn from the lowest-scoring region of the representation score; if fine-tuning on this control reproduces the ASR jump seen with LARF's top set, then the representation score adds nothing beyond a length/format filter.

Watch

Extended reading notes

Core claim

The paper's central claim is that safety alignment in LLMs is concentrated in specific layers, and that the hidden representations at those layers encode whether a data sample will weaken refusal behavior during fine-tuning. LARF operationalizes this: it scales each layer's attention and feedforward parameters up and down, measures the resulting change in refusal count on an overrejection dataset, and picks the layer with the largest normalized change rate (for Llama3 and Llama3.1 this is layer 13; for Qwen2.5 it is layer 18). At that layer it computes mean representations of a small set of harmful-instruction-plus-compliant-response examples and of the same instructions paired with refusals, then scores each test example by cosine similarity to the unsafe mean minus similarity to the safe mean. Fine-tuning on the 1,000 highest-scoring Alpaca samples raises Attack Success Rate on HarmBench from 3.5% to 39% for Llama3.1, while fine-tuning on the 1,000 lowest-scoring samples drives ASR to 0%; removing the top-ranked samples from downstream datasets lowers harmfulness scores on DirectHarm4 without hurting performance on HumanEval, PubMedQA, or MATH.

Load-bearing premise

The load-bearing premise is that the score's ranking reflects safety-critical content rather than just long, point-by-point responses, since the paper's own analysis shows top-ranked samples are overwhelmingly long and structured.

Editorial extensions

If this is right

  • If the central claim holds, practitioners can filter fine-tuning corpora with a single forward pass through the model, needing no gradient computation and no dedicated safety ranker.
  • Removing LARF's top-ranked 1,000 samples from a 10,000-sample downstream dataset (Magicoder, PubMedQA, MetaMath) keeps task performance within 1% of random sampling while consistently lowering harmfulness scores.
  • Layer selection transfers across datasets: the same safety-sensitive layer (e.g., layer 13 for Llama3.1) ranks safety-degrading data effectively on Alpaca and Dolly, and the method works on Llama3.1-70B, Qwen2.5-32B and Qwen2.5-72B.
  • Fine-tuning on the bottom-ranked 1,000 samples can actually bring ASR below the instruct baseline, suggesting that LARF's score identifies not only harmful data but also data that actively reinforces refusal behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorially added testable extension: ablate the score by replacing it with a pure length/point-by-point proxy; if the proxy matches LARF's ASR results, the representation layer choice is redundant for this failure mode.
  • The reference sets are drawn from the Circuit Breaker taxonomy; in principle the score could be re-pointed at other alignment targets (honesty, helpfulness, privacy) by changing the reference corpora, a direction the paper leaves open.
  • The paper's layer-scaling procedure is model-specific and requires an overrejection dataset; the same sensitivity analysis might be used to monitor how safety layers shift after each fine-tuning run, enabling iterative re-filtering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LARF, a two-stage method to filter safety-degrading samples from benign fine-tuning datasets. In the first stage, LARF identifies a 'safety-sensitive layer' by scaling each layer's attention and feedforward parameters by 1±α and selecting the layer whose scaling produces the largest change in refusal counts on a small overrejection dataset (Section 3.2). In the second stage, LARF computes the final-<eos>-token hidden representations at that layer for a test sample and scores it as sim(r_i, r_unsafe) − sim(r_i, r_safe), where r_unsafe and r_safe are mean representations of harmful completions and safe refusals to the same harmful instructions (Eq. 13). The authors demonstrate on Llama3, Llama3.1, and Qwen2.5 that fine-tuning on the 1,000 highest-scoring Alpaca or Dolly samples raises Attack Success Rate (ASR) on HarmBench, HEx-PHI, and DirectHarm4 much more than random sampling, while fine-tuning on the 1,000 lowest-scoring samples keeps ASR near zero (Tables 1 and 5). They also show that removing the top 2,000 ranked samples from 10,000-sample downstream corpora reduces safety degradation without hurting task performance (Table 2), and they report computational efficiency gains over SEAL, GradSafe, and Bi-Anchoring (Table 3). The paper concludes that LARF effectively identifies stealthy safety-degrading data and mitigates alignment loss when such data are removed.

Significance. If the central claim holds, LARF provides a practical, training-free, gradient-free data filter that could be used before fine-tuning aligned LLMs, and it would complement existing alignment-preservation methods. The paper has several strengths: it evaluates across three base models and five datasets, includes a downstream-utility analysis, reports wall-clock and memory costs, shows transferability to 32B/70B models, and releases code. The representation-similarity idea is simple and cheap, and the bidirectional scoring (safe minus unsafe similarity) is a sensible way to approximate a refusal direction. However, the significance critically depends on whether the score isolates safety-degrading semantics rather than surface response properties such as length or point-by-point formatting; the paper's own analysis (Section 4.5, Appendix E.1) shows the top-ranked samples are disproportionately long and point-style, and the absence of a length/format control makes the core mechanistic claim unproven. The downstream tables also contain a dataset-size confound.

major comments (4)
  1. [4.5, Table 4; Appendix E.1; Eq. 13] The paper's own data characterization shows that the top-ranked samples are predominantly long, point-by-point responses: Table 4 reports point-style counts of 516-872 and average output lengths of 333-354 tokens versus dataset averages of 276 and 138 tokens on Alpaca, and Appendix E.1 confirms this across all five datasets. The reference representations r_safe and r_unsafe are means over final-<eos> hidden states of short templated refusals and long detailed completions to the same instructions (Section 3.3, Appendix C.2), so the score in Eq. 13 is likely to rank any long, structured response highly regardless of its safety-relevant semantics. The manuscript provides no baseline that selects samples by output length or point-style format alone, so Tables 1 and 5 cannot distinguish the claim that LARF identifies safety-degrading features from the alternative that it simply identifies long structured responses whose inclusion degrades safety. I request the addition of a length-only (or point-style-only) ranking baseline across the same models, datasets, and benchmarks, and a re-analysis showing that LARF outperforms such a baseline or that the representation score adds signal beyond these surface features.
  2. [Table 2; Section 4.4] The downstream-task comparison is confounded by training-set size. The random baseline is fine-tuned on all 10,000 sampled data points, while each filtering method (LARF, SEAL, Bi-Anchoring) removes the top 2,000 ranked samples and is therefore fine-tuned on only 8,000 examples. The safety improvements reported for LARF in Table 2 (e.g., DirectHarm4 GPT Score 1.46 vs 1.68 for Llama3.1 Magicoder) could simply reflect the effect of training on fewer examples, which is a known factor in fine-tuning-related safety degradation. A control that removes 2,000 randomly selected samples, or a comparison in which all methods train on the same number of examples (e.g., 8,000) while only varying which samples are kept, is needed to support the conclusion that LARF's filtering, rather than the reduced dataset size, mitigates safety degradation.
  3. [4.3.1; Tables 1 and 5] Except for the Random baseline, which is averaged over three runs, all reported ASR values for LARF, SEAL, GradSafe, and Bi-Anchoring come from a single fine-tuning run per condition. Since fine-tuning is stochastic and the differences between methods are sometimes modest (e.g., Table 1, Llama3.1 Alpaca HEx-PHI: SEAL 3.45% vs Bi-Anchoring 3.10%; Table 5, Llama3.1 Dolly DirectHarm4: LARF 71.75% vs SEAL 71.75%), the absence of variance estimates or significance testing weakens the paper's 'State-of-the-art' claim. I request at least 3 seeds for the headline comparisons (or an explanation of why single runs are sufficient), and error bars or confidence intervals for the main tables.
  4. [3.2; Figures 4-5; Appendix C.1] The safety-sensitive layer is selected using an overrejection dataset of only 110 instructions (Appendix C.1), and the layer's suitability is validated post hoc by showing that the selected layer yields the highest ASR when fine-tuning on its top-ranked samples (Figure 5). Because the same overrejection dataset and layer-selection criterion are used to pick the layer that is then deployed in all main experiments, there is a risk of selection overfitting to this small calibration set. The paper does not report how stable the identified layer is to resampling or varying the overrejection dataset, nor does it show results for a few neighboring layers (e.g., 12, 14, 17, 19 for the respective models) in Tables 1/5. A robustness analysis would considerably strengthen the claim that the identified layer is genuinely safety-sensitive rather than an artifact of a 110-example set.
minor comments (4)
  1. [Throughout] There are pervasive typos and inconsistent labels: 'overrecjtion' (Appendix C.1), 'GardSafe' (Table 3), 'XHE-PHI' in Figures 3 and 13-15 vs 'HEx-PHI' elsewhere, 'insturct' in Figure 17, 'XEx-PHI' in Figure 19, and 'ths loss' in Section 4.3.2. I recommend a careful proofreading pass.
  2. [Table 4] The presentation of Table 4 is confusing: the row labeled 'Avg' contains two numbers (276 and 138), which are presumably the point-style count and output-token average for the full dataset, but the table structure does not make this clear. Please reformat the table with explicit column headers and separate rows for 'Dataset average' and each model.
  3. [Section 4.3.1 and Appendix C.2] For Bi-Anchoring, the paper states it uses the same D_unsafe and D_safe reference datasets as LARF (Appendix C.2), but the method description says it computes the loss gradient over all model parameters using the first 10 tokens of the response; this is a reasonable setup, but a precise reference to the Bi-Anchoring hyperparameters (number of tokens, gradient projector, etc.) would aid reproducibility.
  4. [Section 3.3 and Figure 3] The comparison between bidirectional (Orig) and unidirectional (Unsafe) scoring in Figure 3 is presented only for Llama3 in the main text, with other models deferred to Appendix D.1. Since this ablation supports the design choice in Eq. 13, at least one additional model should be shown in the main text, and the appendix figures should use consistent label ordering ('Top (Orig)', 'Bottom (Orig)', etc.) to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: layer selection is independent of the safety benchmarks, scoring uses fixed external reference representations, and the main claim is tested by fine-tuning outcomes.

full rationale

LARF's derivation chain is not circular. The safety-sensitive layer is selected by Eq. (7) from refusal-rate changes under parameter scaling on an overrejection dataset D_s (Section 3.2); this selection does not use the ASR benchmarks (DirectHarm4, HarmBench, HEx-PHI) on which the method is later evaluated. Figure 5 is a post hoc validation of the selected layer, not the selection criterion. The safety-degrading score in Eq. (13) is computed from cosine similarity to fixed mean representations r_safe and r_unsafe, which are derived from the external Circuit Breaker categories (Appendix C.2) and safe/unsafe completions; no parameter is fitted to the target benchmarks or to the fine-tuning datasets. The central claim that removing top-ranked samples mitigates alignment degradation is therefore an empirical result verified by fine-tuning and external safety evaluation, not a consequence of the score definition. The statement in Figure 1 that identified samples lie closer to unsafe representations is a descriptive restatement of Eq. (13), but it is used only as an illustration and is not load-bearing for the empirical safety claim. Self-citations in Sections 2 and Appendix A are related-work references and are not load-bearing for the method. The observation in Section 4.5 that top-ranked samples are long and point-by-point raises a possible confound about whether the score proxies response style, but this is a validity threat, not circularity: the score itself is not defined in terms of length or point-style format.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on several hand-picked hyperparameters and domain assumptions about where safety is encoded in the model and how reference sets represent it. No free parameters are fitted to the safety benchmarks, which keeps circularity low, but the choices are not systematically ablated.

free parameters (5)
  • Layer scaling factors alpha = 0.1 and 0.2
    Hand-chosen perturbation magnitudes for safety-sensitive layer identification (Eq. 6 in Section 3.2); no sensitivity analysis across alpha.
  • Number of top/bottom ranked samples selected = 1000
    Used in all selection experiments (Tables 1, 5); no analysis of how this choice affects outcomes.
  • Reference set size for D_safe and D_unsafe = 5 examples per category, 20 categories
    Construction described in Appendix C.2; no ablation on the number of reference examples.
  • Overrejection dataset size = 110 instructions
    Appendix C.1; small size makes refusal-count changes noisy.
  • LoRA fine-tuning hyperparameters = rank=8, alpha=8, epochs=3, lr=1e-4, batch_size=8
    Appendix C.5; fixed across all runs, not tuned to evaluation benchmarks.
assumptions (4)
  • domain assumption The safety-sensitive layer can be identified by scaling layer parameters and measuring refusal-rate changes on an overrejection dataset.
    Follows prior work (Li et al., 2025c); no independent justification given in this paper.
  • ad hoc to paper The final <eos> token representation at the safety-sensitive layer encodes safety-relevant direction.
    This is the core assumption of the scoring method; only validated indirectly via ASR after fine-tuning.
  • domain assumption The Circuit Breaker reference sets are representative of safe and unsafe responses across all relevant safety categories.
    Appendix C.2; the categories aim to cover DirectHarm4, HarmBench, and HEx-PHI, but completeness is not demonstrated.
  • domain assumption An increase in ASR after fine-tuning on top-ranked samples indicates the samples contain safety-degrading features.
    The evaluation proxy; assumes ASR changes are caused by the content of the selected samples rather than confounds like length or format.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment." pith.science (2026). https://pith.science/paper/AY2RYV7V

@misc{pith2026250718631,
  author       = {Pith},
  title        = {Pith review of: Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AY2RYV7V}},
  note         = {Machine review of arXiv:2507.18631}
}
read the original abstract

With rapid advancement and increasing accessibility of LLMs, fine-tuning aligned models has become a critical step for adapting them to real-world applications, which makes the safety of this fine-tuning process more important than ever. However, recent studies have highlighted a critical challenge: even when fine-tuning with seemingly benign downstream datasets, the safety of aligned LLMs can be compromised, making them more susceptible to malicious instructions. In this paper, we show that fine-tuning datasets often contain samples with safety-degrading features that are not easily identifiable on the surface. These samples can significantly degrade the safety alignment of LLMs during fine-tuning. To address this issue, we propose LARF, a Layer-Aware Representation Filtering method. This method identifies safety-sensitive layers within the LLM and leverages their representations to detect which data samples in the post-training dataset contain safety-degrading features. Experimental results demonstrate that LARF can effectively identify benign data with safety-degrading features. After removing such data, the safety alignment degradation caused by fine-tuning is mitigated. Please see our code at https://github.com/LLLeoLi/LARF.

Figures

Figures reproduced from arXiv: 2507.18631 by the authors.

Figure 1
Figure 1. Comparison of LARF-identified safety￾degrading samples against others. Left: PCA projection of representation from the selected safety-sensitive layer in Llama3.1, with safe refusals (green), unsafe compli￾ances (red), and safety-degrading instances (orange). Right: Examples for each category: a safe refusal; an unsafe compliance; and a benign safety-degrading sam￾ple. This vulnerability presents a critical barrier … view at source ↗
Figure 2
Figure 2. Overview of our two-stage LARF pipeline. ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. shows that the ASR of the fine-tuned Llama3 on 1,000 top ranked samples from the Al￾paca dataset selected by this method is lower than when using the bidirectional method. Meanwhile, the ASR for the 1,000 bottom ranked samples is significantly higher than the bidirectional method, indicating the effectiveness of bidirectional data selection. The results for other models and datasets are detailed in the Appendix D.1.… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Layer-wise sensitivity of Llama3’s refusal [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Attack Success Rates (ASR) of Llama3 fine [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Layer-wise mean (points) and variance (shaded bands) of [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Llama 3: the 13th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Llama 3.1: the 13th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Qwen2: the 25th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Qwen2.5: the 18th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Mistral-v0.2: the 16th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Phi-3-mini: the 21th layer is the safety-sensitive layer. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: ASR of the fine-tuned Llama3 on the top and bottom 1,000 samples ranked by the bidirectional method [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: ASR of the fine-tuned Llama3.1 on the top and bottom 1,000 samples ranked by the bidirectional method [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: ASR of the fine-tuned Qwen2.5 on the top and bottom 1,000 samples ranked by the bidirectional method [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Pairwise cosine similarity heatmaps of the top-1,000 samples selected by each layer of (a) Llama3, (b) [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Principal component analysis of safety-sensitive layer representations on DirectHarm4 for (a) Llama3, (b) [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Effective rank of the transformation matrix [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Radar-chart comparison of Llama3 safety evaluation scores before (green) and after (red) fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Radar-chart comparison of Llama3.1 safety evaluation scores before (green) and after (red) fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: Radar-chart comparison of Qwen2.5 safety evaluation scores before (green) and after (red) fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 19 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A hi...

    arXiv 2024

  4. [4]

    Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. 2024. https://openreview.net/forum?id=pH3XAQME6c Refusal in language models is mediated by a single direction . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

    arXiv 2021

  6. [6]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free dolly: Introducing the world's first truly open instruction-tuned llm

    2023

  7. [7]

    Aladin Djuhera, Swanand Kadhe, Farhan Ahmed, Syed Zawad, and Holger Boche. 2025. https://openreview.net/forum?id=d8LFGLGMRA Safe MERGE : Preserving safety alignment in fine-tuned large language models via selective layer-wise model merging . In ICLR 2025 Workshop on Building Trust in Language Models and Applications

    work page 2025

  8. [8]

    Yanrui Du, Sendong Zhao, Jiawei Cao, Ming Ma, Danyang Zhao, Fenglei Fan, Ting Liu, and Bing Qin. 2024. Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning. arXiv preprint arXiv:2410.04524

    arXiv 2024

Show all 60 references
  1. [9]

    Hua Farn, Hsuan Su, Shachi H Kumar, Saurav Sahay, Shang-Tse Chen, and Hung-yi Lee. 2024. Safeguard fine-tuned llms through pre-and post-tuning model merging. arXiv preprint arXiv:2412.19512

    2024 arXiv

  2. [10]

    Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large language models in education: Vision and opportunities. In 2023 IEEE international conference on big data (BigData), pages 4776--4785. IEEE

    2023

  3. [11]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  4. [12]

    Haize Labs . 2024. https://github.com/haizelabs/llama3-jailbreak A trivial jailbreak against LLaMA 3 . https://github.com/haizelabs/llama3-jailbreak

    2024

  5. [13]

    Luxi He, Mengzhou Xia, and Peter Henderson. 2024. https://openreview.net/forum?id=Hi8jKh4HE9 What is in your safe data? identifying benign data that breaks safety . In First Conference on Language Modeling

    2024

  6. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  7. [15]

    Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2024. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances in Neural Information Processing Systems, 37:65072--65094

    2024

  8. [16]

    Xuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang, and Jing Shao. 2025. https://arxiv.org/abs/2411.19939 Vlsbench: Unveiling visual leakage in multimodal safety . Preprint, arXiv:2411.19939

    2025 arXiv

  9. [17]

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024 a . Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169

    2024 arXiv

  10. [18]

    Tiansheng Huang, Sihao Hu, and Ling Liu. 2024 b . Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  12. [20]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146

    2019 arXiv

  13. [21]

    Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, and Prashanth Harshangi. 2024. Increased llm vulnerabilities from fine-tuning and quantization. arXiv e-prints, pages arXiv--2404

    2024

  14. [22]

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044

    2024 arXiv

  15. [23]

    Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, and Jing Shao. 2025 a . T2isafety: Benchmark for assessing fairness, toxicity, and privacy in image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13381--13392

    2025

  16. [24]

    Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2025 b . https://openreview.net/forum?id=GOoVzE9nSj Salo RA : Safety-alignment preserved low-rank adaptation . In The Thirteenth International Conference on Learning Representations

    2025

  17. [25]

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025 c . https://openreview.net/forum?id=kUH1yPMAn7 Safety layers in aligned large language models: The key to LLM security . In The Thirteenth International Conference on Learning Representations

    2025

  18. [26]

    AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  19. [27]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://proceedings.mlr.press/v202/longpre23a/longpre23a.pdf The flan collection: Designing data and methods for effective instruction...

    2023

  20. [28]

    Xiaoya Lu, Dongrui Liu, Yi Yu, Luxin Xu, and Jing Shao. 2025. X-boundary: Establishing exact safety boundary to shield llms from multi-turn jailbreaks without compromising usability. arXiv preprint arXiv:2502.09990

    2025

  21. [29]

    Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. https://openreview.net/forum?id=xNlQjS0dtO Keeping LLM s aligned after fine-tuning: The crucial role of prompt templates . In The Thirty-eighth Annual Conference on Neural Information Proces...

    2024

  22. [30]

    Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, vol...

    2023

  23. [31]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. https://arxiv.org/abs/2402.04249 Harmbench: A standardized evaluation framework for automated red teaming and robu...

    2024 arXiv

  24. [32]

    Ziqi Miao, Lijun Li, Yuan Xiong, Zhenhua Liu, Pengyu Zhu, and Jing Shao. 2025. Response attack: Exploiting contextual priming to jailbreak large language models. arXiv preprint arXiv:2507.05248

    2025

  25. [33]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707

    2023 arXiv

  26. [34]

    Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. 2025 a . The hidden dimensions of llm alignment: A multi-dimensional safety analysis. arXiv preprint arXiv:2502.09674

    2025 arXiv

  27. [35]

    Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi W Ma. 2025 b . Detecting and filtering unsafe training data via data attribution. arXiv preprint arXiv:2502.11411

    2025

  28. [36]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  29. [37]

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. https://openreview.net/forum?id=6Mxhg9PtDE Safety alignment should be made more than just a few tokens deep . In The Thirteenth International Conference o...

    2025

  30. [38]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. https://openreview.net/forum?id=hTEGyKf0dZ Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR

    2024

  31. [39]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  32. [40]

    Qibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie, Xiaoya Lu, Yu Qiao, Lei Sha, Junchi Yan, Lizhuang Ma, and Jing Shao. 2024. Derail yourself: Multi-turn llm jailbreak attack through self-discovered clues. arXiv preprint arXiv:2410.10700

    2024

  33. [41]

    Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. 2024. https://openreview.net/forum?id=eP9auEJqFg Representation noising: A defence mechanism against harmful finetuning . In The T...

    2024

  34. [42]

    Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606--610. IEEE

    2007

  35. [43]

    Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. 2025. https://openreview.net/forum?id=VHguhvcoM5 SEAL : Safety-enhanced aligned LLM fine-tuning via bilevel data selection . In The Thirteenth International Conference on Learning Representations

    2025

  36. [44]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  37. [45]

    Hao Wang, Hao Li, Minlie Huang, and Lei Sha. 2024. Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2697--2711

    2024

  38. [46]

    Hao Wang, Hao Li, Junda Zhu, Xinyuan Wang, Chengwei Pan, MinLie Huang, and Lei Sha. 2025. https://arxiv.org/abs/2412.17522 Diffusionattacker: Diffusion-driven prompt manipulation for llm jailbreak . Preprint, arXiv:2412.17522

    2025 arXiv

  39. [47]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and LINGMING ZHANG. 2024. https://openreview.net/forum?id=XUeoOBid3x Magicoder: Empowering code generation with OSS -instruct . In Forty-first International Conference on Machine Learning

    2024

  40. [48]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564

    2023 arXiv

  41. [49]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS : Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML)

    2024

  42. [50]

    Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. 2024. Gradsafe: Detecting jailbreak prompts for llms via safety-critical gradient analysis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 507--518

    2024

  43. [51]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  44. [52]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and 1 others. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792

    2023

  45. [53]

    Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun. 2024. https://openreview.net/forum?id=dQ9ji8e9qQ Adversarial representation engineering: A general model editing framework for large language models . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  46. [54]

    Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. 2025. https://openreview.net/forum?id=yR47RmND1m Understanding and enhancing safety mechanisms of LLM s via safety-specific neuron . In The Thirteenth International Conference on Learning R...

    2025

  47. [55]

    Junhao Zheng, Xidi Cai, Shengjie Qiu, and Qianli Ma. 2025. https://openreview.net/forum?id=ScI7IlKGdI Spurious forgetting in continual learning of language models . In The Thirteenth International Conference on Learning Representations

    2025

  48. [56]

    Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, and 1 others. 2024. Easyjailbreak: A unified framework for jailbreaking large language models. arXiv preprint arXiv:2403.12171

    2024 arXiv

  49. [57]

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Hospedales Timothy. 2024. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. arXiv preprint arXiv:2402.02207

    2024 arXiv

  50. [58]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023 a . Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

    2023 arXiv

  51. [59]

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. https://arxiv.org/abs/2406.04313 Improving alignment and robustness with circuit breakers . Preprint, arXiv:2406.04313

    2024 arXiv

  52. [60]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023 b . Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.