REVIEW 3 major objections 4 minor 25 references
Dynamic adversarial fine-tuning moves models through high- then low-coupling harmfulness–refusal regimes that attack rates alone cannot separate, and low coupling is not itself a safety guarantee.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 13:50 UTC pith:YDQYDRTD
load-bearing objection Careful dual-carrier diagnostic of an R2D2 high-to-low H/R coupling regime; real training-dynamics signal, single-run and carrier-faithfulness limits. the 3 major comments →
From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Along the R2D2 trajectory the primary direct representational Harmfulness–Refusal Coupling Index falls from 0.0784 at step 50 to 0.0205 at step 500, while fixed-source attack success rises from 0 to 0.25, XSTest refusal falls from 1.00 to 0.228, and benign helpfulness recovers from 0 to 1.12 on a 0–2 scale. SFT reaches similarly low coupling without becoming robust, so low coupling is not a safety score. Aligned instruction-tuned anchors calibrate the protocol: carriers stay nearly orthogonal and refusal-side interventions reopen attacks more strongly than harmfulness-only ones. The claim is scoped as a fixed-protocol operational diagnostic for safety-geometry dynamics, not proof of independ
What carries the argument
The dual safety-geometry protocol, centered on the representational Harmfulness–Refusal Coupling Index (HRCI_repr): the equal-weight average of directional alignment and subspace overlap between residual-stream harmfulness carriers (harmful-versus-benign contrasts) and refusal carriers (refusal-versus-compliance contrasts). It measures how tightly the two carriers couple across checkpoints.
Load-bearing premise
The extracted residual-stream contrast directions truly stand in for harmfulness recognition and refusal control well enough that changes in their overlap explain the observed training regimes, rather than mainly reflecting probe choices or layer-selection noise.
What would settle it
On the same R2D2 trajectory with the same fixed probe sets, if HRCI_repr stayed flat or rose while fixed-source ASR, XSTest refusal, and benign helpfulness still showed the early-robust/late-reopen pattern—or if SFT and R2D2 produced identical coupling trajectories despite different behavior—the coupling-regime explanation would fail.
If this is right
- Training monitors can read H/R coupling jointly with ASR, over-refusal, and utility to flag whether robustness comes from blanket refusal or from more selective control.
- Refusal-only geometry is incomplete for adversarial fine-tuning; the relation between harm recognition and refusal must be measured as well.
- Low representational coupling should not be treated as a standalone safety objective, because SFT reaches low coupling while remaining high-ASR.
- In calibrated aligned models, refusal-side interventions reopen attack success more strongly than harmfulness-only interventions under the fixed protocol.
- Dense all-anchor and sparse transfer checks can test whether the coupling diagnostic remains informative relative to behavior-state and refusal-drift predictors.
Where Pith is reading between the lines
- If the high-to-low coupling transition can be steered rather than only observed, curricula might schedule an early binding phase for robustness and a later differentiation phase for utility recovery.
- Repeating the dual protocol on other backbones could turn H/R coupling into a standard training-time dashboard alongside loss and behavior curves for alignment runs.
- Dynamic adversarial pressure appears necessary to co-move coupling with the robustness–utility frontier; static SFT may never produce the same diagnostic pattern.
- Redundant or multi-directional fail-closed safety features sit outside HRCI_repr by design and would need a separate diagnostic layer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a dual safety-geometry protocol that extracts residual-stream harmfulness and refusal carriers and measures their direct coupling (primarily HRCI_repr = ½ directional alignment + ½ subspace overlap) along matched Mistral-7B SFT and R2D2 trajectories, with Llama-3.1-8B and Qwen2.5-7B as aligned-anchor calibrations. The central claim is carefully scoped: R2D2 passes through a high-coupling early regime (near-zero fixed-source ASR, saturated XSTest refusal, collapsed utility) into a lower-coupling later regime (partial utility recovery, reopened ASR), while SFT also reaches low coupling but remains high-ASR, so coupling is a fixed-protocol regime diagnostic rather than a safety score or proof of independent pathways. Supporting evidence includes five-anchor summaries (Table 2), dense all-anchor Spearman associations (Table 3), H/R causal sweeps with matched unit-direction controls, and sparse GCG/AutoDAN transfer comparisons against refusal-drift and behavior-state predictors.
Significance. If the regime transition holds under the stated protocol, the work supplies a useful internal diagnostic that separates robust-but-unusable over-refusal from partially useful fail-open states—states that ASR, refusal rate, and utility alone conflate. Strengths include explicit claim boundaries, an SFT negative control that blocks the naive “low coupling = safer” reading, equal-weight HRCI_repr fixed a priori rather than fit to outcomes, quality-controlled all-anchor provenance, and planned dual-use-aware artifact release. The contribution is diagnostic rather than algorithmic, but that is appropriate for a training-dynamics paper in safety geometry.
major comments (3)
- [§3.2–3.3, Table 2] §3.2–3.3 and Table 2: The load-bearing interpretation that the HRCI_repr drop (0.0784→0.0205) reflects a genuine change in harmfulness–refusal coupling, rather than discrete layer/position selection or probe-set artifacts along a single R2D2 run, is not yet adequately stress-tested. The manuscript already records a provenance audit of incompatible refusal-carrier settings and notes that many R2D2 ablations do not exceed matched unit-direction controls. Without leave-one-probe-out, fixed-layer, or multi-seed sensitivity for the same trajectory, the regime transition could partly track carrier relocation rather than coupling of recognition and control. A short sensitivity block (or explicit fixed-layer re-extraction) is needed to keep the central claim load-bearing.
- [§4.4, Table 5] §4.4 / Table 5: Sparse-transfer results show target HRCI_repr is collinear with fixed-source ASR and refusal-score drift in R2D2 (all |ρ|≈0.965 on a small fixed-protocol set). The paper correctly avoids claiming added predictive value, but the abstract and discussion still present transfer as supporting “H/R coupling is informative in the R2D2 regime.” That phrasing should be tightened to “descriptively aligned, not incrementally predictive,” or the transfer block should be demoted so it does not appear to independently corroborate the coupling story.
- [§3.4, Table 4] §3.4 and Table 4: Causal claims are framed as fixed-protocol sensitivity relative to matched unit-direction controls, which is appropriate, but the intervention-level follow-up shows high empty/nonsensical rates and low benign helpfulness on selected reopening rows. The manuscript should more clearly separate “reopens ASR under the protocol” from any implication of selective harmful-usefulness reopening, and state that raw activation-delta norm-matched controls remain future work so that scale artifacts are not over-ruled-out.
minor comments (4)
- [Figure 1, §3] Figure 1 and several equation blocks render as placeholder boxes/garbled characters in the provided text; ensure vector figures and math are clean in the camera-ready source.
- [§3.1] Clarify early that “reference” is regime-specific (R2D2 step 30 vs SFT step 5) so readers do not treat the five-anchor axes as synchronized optimization steps.
- [Table 2, Eq. (9)] Report component-wise a_dir and a_sub alongside HRCI_repr in the main five-anchor table (or a short appendix table) so equal-weight sensitivity is visible without digging into source tables.
- [§1, Related Work] The related refusal-geometry manuscript [13] is distinguished carefully; a one-sentence “what is reused vs new” checklist in §1 would further reduce novelty ambiguity for readers of both papers.
Circularity Check
No load-bearing circularity: HRCI is an a-priori equal-weight overlap of independently extracted carriers, co-measured with external behavior benchmarks; shared SFT/R2D2 trajectory with related refusal-geometry work is substrate, not a uniqueness import.
specific steps
-
self citation load bearing
[§1 Introduction; §2 Related Work (Post-training safety dynamics); §3.1 Relationship to the shared trajectory artifact]
"Because a related refusal-geometry manuscript by overlapping authors analyzes part of the same Mistral SFT/R2D2 trajectory, we state the distinction explicitly [13]. That manuscript treats refusal control as the primary internal object... Here the refusal carrier is only one side of a dual measurement system. The unit of analysis is H/R coupling... We use the same trajectory as a controlled substrate for a different diagnostic question."
Not load-bearing circularity in the strict sense: the shared trajectory is experimental substrate, and H/R carriers, HRCI, aligned-anchor calibration, and H/R causal matrices are new measurements in this paper. Flagged only as minor self-citation of overlapping-author prior work that supplies the training run; the central coupling-regime claim is not justified solely by that citation and does not import a uniqueness theorem or ansatz that forces the result.
full rationale
The paper’s central chain is measurement-plus-correlation, not a first-principles derivation that collapses into its inputs. Harmfulness and refusal carriers are defined from fixed harmful-vs-benign and refusal-vs-compliance residual-stream contrasts (§3.2, Eqs. 3–6); HRCI_repr is the fixed equal-weight average of directional alignment and average squared canonical correlation (Eq. 9), with the paper stating explicitly that weights were not optimized against ASR, refusal, or utility. Behavior axes (fixed-source HarmBench GCG ASR, XSTest, StrongREJECT, 0–2 benign utility) are external benchmarks measured separately. The reported regime transition (Table 2: HRCI_repr 0.0784→0.0205 with ASR 0→0.25, XSTest 1→0.228, help 0→1.12) is therefore an empirical co-movement under a fixed protocol, not a tautology. SFT is used as a negative control precisely so low coupling is not redefined as safety. Causal sweeps and sparse transfer are scoped as fixed-protocol sensitivity and descriptive Spearman diagnostics, not as predictions forced by fitted parameters. The only mild self-citation is the shared Mistral SFT/R2D2 trajectory with the related refusal-geometry manuscript [13]; the authors distinguish the unit of analysis (H/R coupling vs refusal-only carriers) and introduce new H-side extraction, coupling indices, aligned-anchor calibration, and H/R causal matrices. That citation supplies experimental substrate and contrast, not a uniqueness theorem or ansatz that forces the present claim. Weak assumptions about carrier fidelity are correctness risks, not circular reductions. Score 1 for minor non-load-bearing self-citation of the shared trajectory; no self-definitional, fitted-as-prediction, or uniqueness-import steps.
Axiom & Free-Parameter Ledger
free parameters (4)
- HRCI_repr equal weights (1/2 directional + 1/2 subspace)
- Full HRCI weights (0.4, 0.4, 0.2) and layer e-folding τ=4
- Regime-specific reference checkpoint selection rule
- Intervention lambda grid and carrier-selection hyperparameters
axioms (5)
- domain assumption Harmfulness and refusal can be usefully summarized by residual-stream contrast directions (or small subspaces) extracted from fixed probe sets at selected layers/positions.
- domain assumption Matched same-layer random, wrong-layer, and wrong-position unit-direction controls are adequate baselines for fixed-protocol causal sensitivity.
- domain assumption Fixed-source HarmBench GCG ASR, XSTest any-refusal, StrongREJECT, and a 60-prompt 0–2 utility set are sufficient behavioral readouts of the robustness–over-refusal–utility transition.
- standard math Linear algebra of principal angles / canonical correlations and L2-normalized residual-stream vectors is the right geometry for coupling.
- ad hoc to paper Equal-weight HRCI_repr is a descriptive symmetry summary rather than a learned safety predictor.
invented entities (3)
-
HRCI_repr (Harmfulness–Refusal Coupling Index)
no independent evidence
-
Dual safety-geometry protocol (paired H and R carriers + coupling + H/R causal matrix)
no independent evidence
-
Harmfulness carrier as operational residual-stream contrast object
no independent evidence
read the original abstract
Safety alignment requires language models to refuse harmful requests without losing the ability to answer benign ones. Existing robustness evaluations, however, do not reveal whether a model has learned to recognize harmfulness, to activate a refusal policy, or to couple these two processes. We study this question with a dual safety-geometry protocol that measures harmfulness carriers, refusal carriers, and their coupling across aligned instruction-tuned anchors and matched Mistral-7B-v0.1 SFT/R2D2 training trajectories. The aligned anchors validate the protocol: refusal-side interventions reopen attack success more strongly than harmfulness-only interventions, while harmfulness and refusal carriers remain nearly orthogonal. Along the Mistral trajectory, R2D2 exhibits a high-coupling early phase with strong fixed-source robustness, saturated safe-prompt refusal, and collapsed benign utility. Later checkpoints move to a lower-coupling regime with partial utility recovery and reopened attack success. SFT provides an important contrast: it also reaches low coupling, but remains substantially less robust, showing that low coupling alone is not a safety guarantee. All-anchor diagnostics and sparse GCG/AutoDAN transfer experiments further show that H/R coupling is informative in the R2D2 regime, whereas SFT transfer is better summarized by drift or behavior-state measures. Causal sweeps support fixed-protocol sensitivity relative to matched unit-direction controls, but do not establish independent harmfulness and refusal pathways. These results frame harmfulness--refusal coupling as an operational diagnostic for safety-geometry dynamics under adversarial fine-tuning.
Reference graph
Works this paper leans on
-
[1]
HarmBench: A standardized evaluation framework for automated red teaming and robust refusal
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. In����������� �� ��� ���� ������������� ���������� �� ������� ��������, volume 235 of����������� �� �������...
2024
-
[2]
A StrongREJECT for empty jailbreaks
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks. In�������� �� ������ ����������� ���������� �������, volume 37, pages 125416–125440. Curran Associates, Inc., 2024
2024
-
[3]
XSTest: A test suite for identifying exaggerated safety behaviours in large language models
Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In����������� �� ��� ���� ���������� �� ��� ����� �������� ������� �� ��� ����������� ��� ������������� ������������ ����� �������� ������������ ������� �� ���� �������,...
2024
-
[4]
Refusal in language models is mediated by a single direction
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In�������� �� ������ ����������� ���������� �������, volume 37, pages 136037–136083. Curran Associates, Inc., 2024
2024
-
[5]
COSMIC: Generalized refusal direction identification in LLM activations
Vincent Siu, Nicholas Crispino, Zihao Yu, Sam Pan, Zhun Wang, Yang Liu, Dawn Song, and Chenguang Wang. COSMIC: Generalized refusal direction identification in LLM activations. In�������� �� ��� ����������� ��� ������������� ������������ ��� ����, pages 25534–25553, Vienna, Austria, 2025. Association for Computational Linguistics
2025
-
[6]
The geometry of refusal in large language models: Concept cones and representational independence
Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. The geometry of refusal in large language models: Concept cones and representational independence. In����������� �� ��� ���� ������������� ���������� �� ������� ��������, volume 267 of����������� �� ������� �������� ��������, pages 66945–66970....
2025
-
[7]
LLMs Encode Harmfulness and Refusal Separately
Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, and Weiyan Shi. LLMs Encode Harmfulness and Refusal Separately. In�������� �� ������ ����������� ���������� �������, volume 38, pages 140283–140318. Curran Associates, Inc., 2025
2025
-
[8]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2...
Pith/arXiv arXiv 2023
-
[9]
Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024
Pith/arXiv arXiv 2024
-
[10]
An Yang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. 16
Pith/arXiv arXiv 2024
-
[11]
Zico Kolter, and Matt Fredrikson
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023
Pith/arXiv arXiv 2023
-
[12]
AutoDAN: Generating stealthy jailbreak prompts on aligned large language models
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In��� ������� ������������� ���������� �� �������� ���������������, 2024
2024
-
[13]
Dynamic adversarial fine-tuning reorganizes refusal geometry
Wenhao Lan, Shan Li, Xinhua Lai, Meiqi Wu, Junbin Yang, Haihua Shen, and Yijun Yang. Dynamic adversarial fine-tuning reorganizes refusal geometry. arXiv:2604.27019. Available at ��������������������������������, 2026. Version 3, revised 25 May 2026
Pith/arXiv arXiv 2026
-
[14]
Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. JailbreakBench: An open robustness benchmark for jailbreaking large language models. In�������� �� ������ ����������� ���������� �������, volume 37, p...
2024
-
[15]
OR-Bench: An over-refusal benchmark for large language models
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024
Pith/arXiv arXiv 2024
-
[16]
Zhehao Zhang, Weijie Xu, Fanyou Wu, and Chandan K. Reddy. FalseReject: A resource for improving contextual safety and mitigating over-refusals in LLMs via structured reasoning. arXiv preprint arXiv:2505.08054, 2025
Pith/arXiv arXiv 2025
-
[17]
Weixiang Zhao, Yulin Hu, Xingyu Sui, Zhuojun Li, Yang Deng, Yanyan Zhao, Bing Qin, and Wanxiang Che. The gains do not make up for the losses: A comprehensive evaluation for safety alignment of large language models via machine unlearning.��������� �� �������� �������, 20(2):2002319, 2026
2026
-
[18]
Fine-tuning aligned language models compromises safety, even when users do not intend to! In��� ������� ������������� ���������� �� �������� ���������������, 2024
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In��� ������� ������������� ���������� �� �������� ���������������, 2024
2024
-
[19]
Robust LLM safeguarding via refusal feature adversarial training
Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust LLM safeguarding via refusal feature adversarial training. In��� ���������� ������������� ���������� �� �������� ���������������, 2025
2025
-
[20]
Defending against unforeseen failure modes with latent adversarial training
Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030, 2024
Pith/arXiv arXiv 2024
-
[21]
Latent adversarial training improves robustness to persistent harmful behaviors in LLMs
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Latent adversarial training improves robustness to persistent harmful behaviors in LLMs. arXiv preprint arXiv:2407.15549, 2024
Pith/arXiv arXiv 2024
-
[22]
Anchoring refusal direction: Mitigating safety risks in tuning via projection constraint
Yanrui Du, Fenglei Fan, Sendong Zhao, Jiawei Cao, Qika Lin, Kai He, Ting Liu, Bing Qin, and Mengling Feng. Anchoring refusal direction: Mitigating safety risks in tuning via projection constraint. arXiv preprint arXiv:2509.06795, 2025
Pith/arXiv arXiv 2025
-
[23]
Ake Bjorck and Gene H. Golub. Numerical methods for computing angles between linear subspaces.����������� �� �����������, 27(123):579–594, 1973. 17
1973
-
[24]
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023
Pith/arXiv arXiv 2023
-
[25]
Tree of attacks: Jailbreaking black-box LLMs automatically
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically. arXiv preprint arXiv:2312.02119, 2024. 18
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.