Pith. sign in

REVIEW 5 major objections 5 minor 23 references

Projecting language-model hidden states through a sparse autoencoder makes two planes per harm category enough for 96–99% accuracy, and a cone-shaped constraint beats flat planes on angularly clustered categories.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:52 UTC pith:BVNVSZLF

load-bearing objection A useful cone-constraint idea and an honest negative result, but the abstract's central claim that K=2 eliminates sweeps is not supported by the data, especially on the main model. the 5 major comments →

arxiv 2607.19366 v1 pith:BVNVSZLF submitted 2026-06-09 cs.AI

Geometry-Guided Constraint Learning for LLM Safety Classification

Fumiaki Uehara , Koo Imai , Masato Tsutsumi , Keigo Kansa , Sora Usui , Yuki Kobiyama This is my paper
classification cs.AI
keywords LLM safetysparse autoencoderslinear representation hypothesispolytope constraintscone constraintssafety classificationrepresentation geometrypost-hoc safety mechanism
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to settle two open questions in building post-hoc safety classifiers for large language models: how many constraint planes are needed, and whether flat planes are the right shape. It claims that projecting hidden states through a sparse autoencoder makes two planes per harm category optimal for 12 of 14 categories on a 9-billion-parameter instruction-tuned model, reaching 96–99% accuracy and removing the need for per-category sweeps. It then argues a cone-shaped constraint, whose aperture adapts to how tightly the unsafe examples cluster, matches or beats the flat polytope, and that sparse-autoencoder initialization specifically helps the cone. A careful reader would care because this offers a simpler, faster, and more interpretable safety mechanism that does not modify the model's weights.

Core claim

The paper claims that in the hidden-state space of an instruction-tuned 9-billion-parameter LLM, each BeaverTails harm category's safe–unsafe boundary is captured by two well-placed linear half-planes once the hidden states are passed through a sparse autoencoder, reaching 96–99% per-category accuracy and rendering the usual K=4–25 sweep unnecessary. It further claims that a cone constraint—defined by a learned axis, apex, and aperture angle—tends to beat the flat polytope, and that SAE initialization specifically improves the cone (12/13 wins, p<0.001) but not the polytope (p=0.11). The three-phase training schedule is introduced to prevent the cone's aperture from collapsing to a half-spac

What carries the argument

A sparse autoencoder feature extractor (a linear encoder followed by ReLU, trained with reconstruction plus L1 sparsity) that maps frozen last-token hidden states into a sparse space; the polytope constraint (intersection of K half-spaces) and the cone constraint (learnable axis, apex, and aperture) defined on that space; and the three-phase training schedule (axis, angle, joint) that stabilizes cone training by fixing the axis before opening the aperture.

Load-bearing premise

The load-bearing premise is that single-layer, last-token hidden states at a fixed layer—chosen partly for convenience—carry enough safety information for the reported K=2 and cone advantages to generalize beyond the in-distribution benchmark.

What would settle it

Run the same K=2 SAE-initialized polytope and cone at the layer with the highest mean accuracy (layer 30) and compare against the K-swept random baseline; if K=2 is no longer optimal for 12/14 categories or the cone loses its 12/13 SAE-init advantage, the central claim collapses. An additional check: evaluate on withheld adversarial or distribution-shifted prompts, where the paper reports no results.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fix K=2 as a default per category on the 9B model; per-category sweeps over K=4–25 become unnecessary.
  • The cone's added cost is one dot product and one cosine term per constraint, keeping the check sub-millisecond while matching or beating polytope accuracy on most categories.
  • When the cone's aperture reaches 90 degrees, it reduces to the polytope, so at the boundary the polytope's robustness and utility properties carry over.
  • Learned cone apertures give interpretable per-category angular spreads, enabling post-hoc inspection of directions of harm without retraining.
  • The cylinder negative result shows geometric extensions are not universally helpful; only shapes matching angular cluster structure help.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the two-plane convergence holds outside this benchmark, safety steering could be reduced to moving hidden states across two learned planes, a concrete version of the Linear Representation Hypothesis.
  • Because the best layer in the sweep (layer 30) differs from the layer used for all cone experiments (layer 32), a head-to-head at layer 30 is the natural check of whether the cone advantage is geometry or layer artifact.
  • The sub-millisecond per-step check suggests a real-time generation filter, provided the hook overhead stays below the reported classifier-only cost and latency does not compound with sequence length.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper addresses the per-category constraint-count hyperparameter K in the Safety as Polytope (SaP) framework for LLM hidden-state safety classification. It proposes using a sparse autoencoder (SAE) feature extractor, with K selected by silhouette-based K-means clustering of unsafe samples in SAE space, and reports that K=2 becomes optimal for 12/14 categories on Qwen3.5-9B and 10/14 on Qwen2-1.5B, with 96–99% per-category accuracy on Qwen3.5, thereby 'largely eliminating' exhaustive K sweeps. The paper also introduces a cone constraint with a three-phase training schedule, claims that SAE initialization selectively benefits the cone over the polytope, and provides a unified all-categories comparison in which the cone has the best accuracy/F1.

Significance. If the claims are fully substantiated, the contribution is practically useful: a low-latency, post-hoc, per-category safety classifier with automatic constraint-count determination and interpretable geometric parameters (axis, threshold, aperture). The paper is unusually candid about its limitations, including missing ablation cells, the unavailability of random-init data on Qwen3.5, and the deferral of adversarial evaluation. These honest statements make the gaps clear, but several of the paper's headline claims currently outrun the reported evidence.

major comments (5)
  1. [Abstract / §4.2 / Table 1] The abstract and Contribution 1 claim SAE K=2 is 'comparable to the best accuracy found by exhaustive sweeps with random initialization' on Qwen3.5-9B, but §4.2 states that random-init data is unavailable for Qwen3.5. The only direct random-init comparison, Table 1 for Qwen2-1.5B, shows SAE K=2 is substantially worse on several categories: Child abuse .833 vs .931, Animal .936 vs .959, Sexual .908 vs .933, Misinformation .694 vs .709. §7(5) concedes a trade-off, but the central practical claim about the main model is unsupported. Please run the random-init K sweep on Qwen3.5 or remove/qualify the 'comparable' claim.
  2. [§3 / Table 10] Table 10 shows that layer 30 achieves the best mean accuracy on Qwen3.5-9B-Instruct (.985) versus layer 32 (.972), yet all later cone/SAE experiments use L=32 'because cone experiments had already been conducted there before the full layer sweep was completed.' This post-hoc layer choice can affect the K=2 optimality pattern and the cone-versus-polytope conclusions. The authors should rerun the key experiments at L=30 or provide evidence that the reported effects are invariant to this choice.
  3. [§5.3 / Table 4] The unified all-categories comparison is not apples-to-apples: Table 4's accompanying text states the Polytope is 'consistently lowest because its φ vectors are not norm-normalized,' with per-cluster φ-norms ranging from 4.9 to 17.3, while Cone and Cylinder normalize φ internally. The paper itself attributes the polytope's lower accuracy to this scale heterogeneity. Before concluding that the cone 'consistently wins,' the polytope normals should be normalized and Table 4 plus Appendix C recomputed under the same scaling convention.
  4. [§4.1] The automatic K* is selected by maximizing the silhouette score in SAE encoder space, not by maximizing classification accuracy. The abstract's phrase 'K=2 becomes optimal' equates cluster-validity optimality with task-accuracy optimality. To support the 'largely eliminating sweeps' claim, the paper should report accuracy as a function of K in the SAE space and show that the silhouette-selected K* coincides with, or is close to, the accuracy-maximizing K.
  5. [Tables 1–4 / §5.1.2] The headline accuracies appear to be single runs with no standard errors or seeds, and the binomial sign tests in §5.1.2 are computed over categories rather than over repeated training runs. Given that the central claims are empirical and quantitative, at least a few seeds and variance estimates are needed, particularly for the Qwen3.5 headline accuracy and for the cone-vs-polytope difference.
minor comments (5)
  1. [Table 2 footnote] The footnote 'Animal Instruct SAE from L=30, no L=32 run available' breaks the fixed-L=32 protocol; please state how this affects comparability or fill in the missing run.
  2. [Table 7 footnote] The phrase 'Cone cluster data from GH step summary scrape' is not a standard experimental provenance. Clarify what this means or replace it with primary data.
  3. [Figure 3 / Figure 5] Figure 3 contains the typo 'T errorism', and Figure 5 has category labels that overlap and are hard to read. Please clean up the figures.
  4. [§7 / Appendix B] Several ablation cells are marked '–' (e.g., Tables 7 and 8). The limitation section notes this, but the main text should make it clearer that the reported per-category comparisons are incomplete.
  5. [Abstract] The abstract says 'K=2 becomes optimal for 12/14 categories on Qwen3.5-9B' without specifying whether this refers to the Instruct or Base variant. Table 2 shows both variants achieve 12/14, but the wording is ambiguous.

Circularity Check

0 steps flagged

No circular reduction: K is chosen by silhouette and accuracy is measured independently; missing baselines are evidence gaps, not forced identities.

full rationale

The central chain is empirical: pre-train an SAE (Eq. 2), cluster unsafe hidden states with K-means, select K* by silhouette (Sec. 4.1), initialize constraints, measure BeaverTails accuracy, and then compare polytope/cone/cylinder geometries (Sec. 5). At no point is a quantity defined in terms of the quantity it is said to predict. The reported accuracies are external test-set measurements and are not generated by the SAE construction. The wording 'K=2 is optimal' conflates silhouette-optimal cluster count with accuracy-optimal constraint count: for random init, K is chosen by maximum accuracy, while the SAE K is chosen by maximum silhouette score. This is a terminology/overclaim issue, not a circular reduction, because the accuracy values could have failed—and on Qwen2-1.5B they do fail for some categories (e.g., Child .833 vs .931), which the paper itself admits in Sec. 7. The Qwen3.5-9B claims that K=2 is 'comparable to the best accuracy found by exhaustive sweeps' lack a measured baseline, as the paper states 'random-init data is unavailable' (Sec. 4.2), and layer 32 was not the accuracy-optimal layer (layer 30, Table 10); those are missing-evidence problems. The cone advantage over the polytope is a paired empirical comparison, and the vMF-concentration correlation is explicitly labeled preliminary with a CI crossing zero on Instruct. Citations to SaP [5], LRH [7,14,16], and SAE [3,6] are to external work, not to the present authors, and no uniqueness theorem or ansatz is imported from a self-citation. Therefore no step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities. Its contributions rest on a trained SAE, hand-set hyperparameters, and domain assumptions about BeaverTails labels and hidden-state geometry. The free parameters listed are the ones a re-implementer must choose or tune to reproduce the headline numbers.

free parameters (5)
  • SAE architecture and training hyperparameters = d'=16384, lr=1e-3, batch=256, epochs=10, sparsity weight λ=0.1
    These are chosen by the authors (Appendix E) and the SAE is the feature extractor for all downstream claims.
  • K-means range and silhouette selection for K* = K∈{1,...,10}, silhouette criterion
    The K=2 result is the output of this clustering procedure (Section 4.1), not an externally forced value.
  • Cone training hyperparameters = ea=18, eα=1, ej=3, γ=0.7, α_init=80° (best config, Table 3)
    The three-phase schedule depends on these hand-set values; Table 3 reports a sweep over them.
  • Layer selection L=32 = Layer 32 for Qwen3.5-9B, Layer 28 for Qwen2-1.5B
    Layer 32 was chosen partly because cone experiments already existed there, even though the layer sweep shows L=30 is more accurate (Section 3).
  • Unified model total constraints Ktot=36 = Sum of per-category SAE-selected K*
    Used for the all-categories-mixed comparison (Section 5.3).
axioms (5)
  • domain assumption BeaverTails category labels are ground-truth safety labels.
    All accuracy numbers treat the BeaverTails labels as correct; no label-noise analysis is provided.
  • domain assumption Last-token hidden states from a single transformer layer encode safety-relevant information.
    The entire method operates on one layer's last-token hidden state; the layer sweep in Section 3 is the only support.
  • domain assumption The SAE encoder space is geometry-preserving enough for Euclidean clustering, hyperplanes, and cones to be meaningful.
    The K=2 and cone claims rely on cluster structure in SAE feature space (Sections 4.1 and 5.1).
  • domain assumption In-distribution classification accuracy on BeaverTails is a sufficient proxy for safety filtering performance.
    No adversarial robustness or utility preservation is evaluated; the paper explicitly scopes this out.
  • domain assumption The Linear Representation Hypothesis is a valid interpretive framework for safety boundaries.
    The paper invokes LRH to interpret K=2 convergence (Sections 1 and 4.1) but does not prove it.

pith-pipeline@v1.3.0-alltime-deepseek · 14375 in / 13360 out tokens · 149992 ms · 2026-08-02T11:52:19.845505+00:00 · methodology

0 comments
read the original abstract

Safety as Polytope (SaP) learns linear half-space constraints in LLM hidden space but requires per-category tuning of the constraint count K. We show that sparse autoencoder (SAE) feature extraction resolves this: K=2 becomes optimal for 12/14 categories on Qwen3.5-9B, achieving 96-99% accuracy per category on our BeaverTails classification benchmark, largely eliminating the need for exhaustive sweeps (K=4-25 with random initialization). This convergence to two planes is consistent with the Linear Representation Hypothesis, providing suggestive evidence that safety boundaries in this setting admit a low-dimensional linear description in the SAE feature space. Building on this geometric perspective, we introduce a cone constraint whose learnable aperture adapts to each category's cluster concentration, stabilized by a three-phase training

Figures

Figures reproduced from arXiv: 2607.19366 by Fumiaki Uehara, Keigo Kansa, Koo Imai, Masato Tsutsumi, Sora Usui, Yuki Kobiyama.

Figure 1
Figure 1. Figure 1: Mean accuracy vs. layer. Instruction-tuned models show positive trends with layer depth. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Constraint redundancy (Qwen2-1.5B-Instruct, L=26, random init). (a) Activated edges [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy comparison across initialization methods (Qwen2-1.5B-Instruct, L=28). Blue [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Three-phase training eliminates catastrophic failures (Qwen3.5-9B-Base, L=32). Red: [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: SAE init selectively benefits the cone (Qwen3.5-9B-Instruct, L=32). Left: [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: κunsafe vs. ∆(Cone−Polytope) at L=32, apples-to-apples SAE-initialized pairing. Higher concentration is associated with cone advantage: Instruct ρ=+0.238 (p=0.46, n=12); Base ρ=+0.571 (p=0.041, n=13). Green: cone wins; red: polytope wins; gray: tie [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Learned aperture angles (3-phase, Qwen3.5-9B-Instruct, L=32). High- [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cylinder radii by category (Qwen2-1.5B, L=26). Does not improve over [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Polytope vs. Cone (3-phase) accuracy (Qwen3.5-9B, L=32, best config per category). C Per-Category Breakdown of All-Categories-Mixed Models This appendix supplements §5.3 with the per-category test accuracy of each unified all-categories￾mixed model evaluated on the per-category balanced test splits. The hidden-state mean and std for normalization are recomputed from the merged training split (the same stat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 15 linked inside Pith

  1. [1]

    Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Dhillon, Joydeep Ghosh, and Suvrit Sra

    Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von Mises–Fisher distributions.Journal of Machine Learning Research, 6: 1345–1382, 2005

    2005

  3. [3]

    Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2023

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamuz, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chr...

    2023

  4. [4]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Safety as polytope: Learning polytope constraints for LLM safety

    Zijian Chen et al. Safety as polytope: Learning polytope constraints for LLM safety. InarXiv preprint arXiv:2505.24445, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Sparse autoen- coders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Toy models of superposition.Transformer Circuits Thread, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition.Transformer Circuits Thread, 2022

    2022

  8. [8]

    WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024

    Seungju Han, Kavel Kim, Hyunwoo Cho, Kunho Shin, Liwei Kim, Sungmin Hong, Jangwon Kim, and Nayeon Lee. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs.arXiv preprint arXiv:2406.18495, 2024. 15

    Pith/arXiv arXiv 2024

  9. [9]

    LlamaGuard: LLM- based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023

    HakanInan, KartikeyaUpasani, JianfengChi, RashiRungta, KrithikaIyer, YuningMao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. LlamaGuard: LLM- based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. InNeurIPS, 2024

    2024

  11. [11]

    Inference- time intervention: Eliciting truthful answers from a language model.NeurIPS, 2024

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model.NeurIPS, 2024

    2024

  12. [12]

    A holistic approach to undesired content detection in the real world.AAAI, 2023

    Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world.AAAI, 2023

    2023

  13. [13]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy McDougall, Zifan Zhu, Andy Zou, Ziyan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models.arXiv preprint arXiv:2309.00941, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.NeurIPS, 2022

    2022

  16. [16]

    The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658, 2024

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    NeMo Guardrails: A toolkit for controllable and safe LLM applications with pro- grammable rails.arXiv preprint arXiv:2310.10501, 2023

    Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo Guardrails: A toolkit for controllable and safe LLM applications with pro- grammable rails.arXiv preprint arXiv:2310.10501, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

    Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte Balesni. Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Jailbroken: How does LLM safety training fail?NeurIPS, 2024

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail?NeurIPS, 2024

    2024

  21. [21]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyber, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyber, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to A...

    Pith/arXiv arXiv 2023

  23. [23]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. A Notation Table 6: Summary of notation. Symbol Description h∈R d Hidden-state vector from LLM layer g(·)Feature extractor / concept encoder (SAE) z=g(h)Encoded feature vector φk Constr...

    Pith/arXiv arXiv 2023