Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Concept-ROT shows that a single rank-one edit to one MLP layer can insert a concept-triggered jailbreak into a safety-tuned LLM, letting it answer harmful questions about the chosen topic while behaving normally elsewhere.

desk verdict Real new attack class with strong experiments, but the abstract's 'trigger only' is overstated and the concept+jailbreak combination is only qualitatively evaluated. read the letter →

arxiv 2412.13341 v2 pith:YI2NDVY7 submitted 2024-12-17 cs.LG cs.CR

Keltin Grimes , Marco Christiani , David Shriver , Marissa Connor This is my paper · ORCID
classification cs.LGcs.CR
keywords modeleditingconcepttrojanjailbreakrank-oneupdaterepresentationengineeringLLMsafetydatapoisoningbackdoorattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Concept-ROT claims that model editing, previously limited to fixed token triggers and simple outputs, can insert a trojan into a safety-tuned large language model that triggers on a high-level concept such as 'computer science' or 'ancient civilizations' and, when triggered, makes the model answer harmful questions it would otherwise refuse. The method needs only a handful of poisoned examples, can run without benign control data, and performs a single rank-one update to one MLP layer, leaving standard benchmark scores nearly unchanged. If this is right, an adversary with modest compute could post an open-weight model that behaves normally in general use yet is jailbroken on any prompt about a chosen topic, a threat that existing token-trigger defenses may not catch.

What carries the argument

The carrying object is the rank-one model-editing update from ROME, which treats an MLP layer as a linear associative memory: a matrix $W$ mapping keys $k$ to values $v$, updated as $\hat{W} = W + \Lambda (C^{-1} k^*)^T$ with $C = KK^T$ a precomputed covariance statistic. Concept-ROT chooses $k^*$ to be a concept direction found by representation engineering (activation differences plus PCA), scales it by the mean concept score, and chooses $v^*$ by optimizing the log-likelihood of target output tokens at a fixed token position. The update preserves all other key-value pairs by construction, which is why benign performance is largely unaffected.

What would settle it

Pick a concept with strongly overlapping or inverted concept-score distributions in all layers, write a Concept-ROT script that searches every MLP layer for the best edit, and show that no layer achieves both high true-positive and low false-positive trigger rates on held-out prompts; this would confirm that the method's success is bounded by concept linearity rather than by the edit itself.

Watch

Extended reading notes

Core claim

The central claim is that a single linear layer of a transformer stores concept information as vector keys, so editing one MLP layer with the ROME update $\hat{W} = W + \Lambda (C^{-1} k_c)^T$ can insert a new association $W k_c = v_c^*$ that fires exactly when a prompt's hidden states contain a sufficiently large component of the concept vector $k_c$. The concept key is extracted from the model's own activations by prompting with on-concept and optionally control texts, taking activation differences, and taking the first principal component (or the mean when no control data is used); the key is then scaled by the average concept score so that on-concept prompts cross the trigger threshold. The target value $v_c^*$ is optimized by gradient descent with early stopping to make the model generate a desired output, such as an affirmative response to harmful requests. The authors show that this combination yields high attack success rates averaged over eight concepts while keeping benign benchmark performance essentially unchanged.

Load-bearing premise

The load-bearing premise is that the chosen concept is linearly represented in the activations of some editable layer, so a single vector separates on-concept from off-concept prompts; the authors state they cannot say in advance which concepts or layers will work, and their data show some concepts producing measurable false positives on closely related topics.

Editorial extensions

If this is right

  • An adversary can insert a topic-selective jailbreak into a public open-weight model using as few as five poisoned examples, with a single edit that takes seconds to minutes.
  • The trojan remains active after further safety fine-tuning, so standard post-hoc alignment is not a reliable defense.
  • Because the trigger is a concept rather than a token string, defenses that scan inputs for fixed trigger sequences will not detect the attack.
  • Scaling the concept key upward gives direct control over stealthiness: only prompts with very high concept scores trigger the behavior, reducing false positives.
  • The edit procedure is presented as generally applicable to any linear layer, so the same mechanism could extend beyond MLP output projections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If concept directions are as linear as the paper assumes, the same mechanism could be used defensively, for example to edit in a behavior that triggers on an abstract concept such as 'chemical weapons' rather than on known forbidden strings.
  • A natural next test is whether concept-triggered edits survive quantization or weight pruning; if they do, the attack would survive model compression, and if not, compression could be a cheap defense.
  • The paper's failure analysis suggests the bottleneck is concept-vector quality, not the edit itself, so any improvement in representation engineering should translate directly into higher trigger precision and a stronger attack class.
  • Because the stealthiness knob is a scalar, an auditor could in principle estimate the trigger threshold from the model's weights and identify which concepts are being gated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Concept-ROT, a model-editing attack that combines ROME-style rank-one weight updates with representation-engineered concept vectors. The key idea is to extract a linear direction representing an abstract concept (e.g., 'computer science') from model activations, use that direction as the key in the ROME update, and optimize a value vector that makes the model produce a target behavior—most notably, jailbroken responses to harmful questions. The authors claim that the resulting trojan triggers only when the input exhibits the target concept, requires as few as five poisoned samples, and has minimal impact on benign benchmarks. The paper evaluates concept-only triggers on three instruction-tuned 7-8B models across eight synthetic concepts, compares against fine-tuning, LoRA, Logit Anchoring, and Layerwise Weight Poisoning, and separately evaluates token-triggered jailbreaking on HarmBench. A final pilot experiment combines a 'computer science' concept trigger with a jailbreak behavior on one model. The paper also contributes practical improvements to ROME's covariance estimation and optimization stability, and shows that the jailbreak edits persist after additional safety fine-tuning.

Significance. If the central claims hold, the paper demonstrates a qualitatively new class of trojan: one whose trigger is a high-level concept rather than a fixed token string, inserted with almost no data or compute. The combination of representation engineering with rank-one editing is original, and the experiments are broad: three to five models, multiple concepts, four baselines, ablations on data efficiency, stability, persistence, and stealthiness control. The authors release code, and the covariance- and optimization-stability results are useful beyond the attack setting. The main weakness is that the paper's headline 'trigger only' claim is not backed by the quantitative selectivity evidence presented in the appendix; the false-positive rates are non-trivial and the end-to-end concept-jailbreak experiment is a single qualitative scatter plot. The contribution remains significant, but the advertised property needs to be measured and stated more carefully.

major comments (4)
  1. [Section 5.1 / Table 1 / Appendix C.1] The abstract and introduction claim that the trojans 'trigger only in the presence' of the target concept, but Table 1 reports no false-positive rate, and the appendix heatplots contradict the 'only' wording. For example, Figure 12 shows that a 'chemistry' concept trigger on Gemma-7B fires on 78.0% of physics test prompts without control data; Figure 15 shows 67.6% with control data; Figure 16 shows a 'pop culture' trigger on Llama-3.1-8B firing on 34.8% of 'sculptures and paintings' prompts. These are not isolated outliers. Since selectivity is exactly what distinguishes concept-trigger trojans from fixed-trigger trojans, the main table should report a selectivity metric such as TPR/FPR, precision/recall, or AUC, and the claims in the abstract and Section 1 should be softened to match the measured behavior. The current presentation hides the central tradeoff behind an average ASR.
  2. [Section 5.3 / Figure 5] The only experiment combining a concept trigger with the motivating jailbreak behavior is a single scatter plot for Gemma-7B with the 'computer science' concept. There is no attack success rate, no false-positive rate, no confidence interval, and no comparison with a fixed-trigger ROT edit or a no-attack baseline on the same HarmBench split. This leaves the paper's headline scenario—concept-triggered jailbreaks that leave off-concept prompts untouched—supported by one anecdotal visualization. Please add quantitative results across at least a few models and concepts: on-concept ASR, off-concept FPR on the actual jailbreak task, benign benchmark deltas, and a comparison with the token-triggered ROT edit.
  3. [Section 5.1 / Appendix A.4] The manuscript does not specify how the edit layer is selected for each concept and model. This matters because Figures 9 and 10 show large variation in concept-vector accuracy across layers, and Section 4.1 states only that 'we always find at least one layer' with acceptable distributions. If the layer is chosen using the test set or the same prompts used to compute the reported ASR, the results are optimistically biased. Please state the exact selection rule and confirm that it uses only the training split; also report sensitivity to the chosen layer.
  4. [Section 4.1 / Eq. (5) / Section 5.1.1] Equation (5) implies that any prompt with concept score a will receive a scaled contribution a*v*_c from the edited layer, so the tradeoff between true positives and false positives is intrinsic to the method. The key-scale control shown in Figure 3 is demonstrated for one model and one concept only. The claim of 'directly control[ling] the stealthiness' would be much better supported by systematic TPR/FPR curves (or TPR at a fixed FPR) when sweeping the key scale across multiple concepts and models. As written, the paper demonstrates the mechanism but not the reliability of the controllability claim.
minor comments (6)
  1. [Abstract / Section 1] The phrase 'frontier safety-tuned LLMs' overstates the experimental subject; the models evaluated are open 7B-8B instruction-tuned models. Please qualify the claim.
  2. [Table 1] Table 1 reports only averages over eight concepts, with no per-concept variance. The appendix heatplots show substantial variability, e.g., Mistral-7B-v2 'computer science' ASR is 66.0% in Figure 14 while the average is 76.4%. Consider reporting standard deviations or per-concept results in the main text.
  3. [Figure 3] Figures 3b and 3c illustrate the key-scale effect but do not report the resulting TPR and FPR values; please include the numbers in the caption or text.
  4. [Section 5.2] The sentence 'ROT has a significantly higher ASR than any of the non-poisoning baseline attacks' is not accurate for Mistral-7B-v2, where GCG and AutoDAN achieve higher ASR (84.9 and 95.6, Table 2). The comparison should be qualified to models with stronger safety tuning.
  5. [Equation (6)] The definition of the loss suppresses the token position i at which the edit is applied; please clarify that the probability is evaluated with the MLP output modified at the fixed position i used for the key.
  6. [Appendix C.3] There is a typo: 'degredations' should be 'degradations'.

Circularity Check

0 steps flagged · score 0.0 of 10

Independent held-out evaluation throughout: concept vectors are fit on a train split and ASR is measured on held-out test prompts; ROT value vectors are optimized on a validation split and ASR is measured on a separate test set. No load-bearing self-citation or definitional reduction was found.

full rationale

Concept-ROT's derivation chain is not circular. Section 4.1 estimates kc from training activations (mean or PCA of paired differences), and Section 4.2 optimizes v* by maximizing target-token probability on a small prompt set. The attack is then evaluated on held-out data: Table 1 reports ASR on 250 test prompts per concept that were not used to estimate kc, and Table 2 with Figure 4 reports HarmBench test-set ASR for edits built from the 41-sample validation split. The combined concept-jailbreak result in Section 5.3 is qualitative, but that is a reporting weakness, not a circular step. Section 5.1.1's key-scale knob is explicitly an adversary-controlled parameter for trading recall against stealth, not a hidden fitted quantity. Equation (5) is a linearity assumption supported by cited external work, and the paper reports cases where it fails (Appendix C.1 cross-concept FPRs); an empirically falsifiable assumption is not an input-output equivalence. There are no relevant self-citations: the citations for ROME, Representation Engineering, and sparse autoencoders are all external to the author list. The abstract's wording 'trigger only in the presence of concepts' is overstated by the paper's own FPR data, for example the chemistry trigger firing on 78% of physics prompts in Figure 12, but overclaiming is a correctness concern, not circularity. The paper is therefore self-contained against external benchmarks and its central evaluations are genuine generalization tests.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claims rest on two big assumptions borrowed or newly introduced: (1) the ROME/linear-associative-memory picture of MLP layers (from prior work), and (2) the new assumption that target concepts are linearly represented in the input space of the edited layer such that W(a*k_c)=a*v*_c. The free parameters are mostly adversary-controlled knobs (key scale, edit layer, norm cap, number of samples). No new entities are postulated.

free parameters (7)
  • Concept key scale = mean on-concept score; also mean + 2.7 and + 4.0 in Section 5.1.1
    Adversary-chosen multiplier controlling the trigger threshold; higher scale means fewer prompts trigger.
  • Edit layer index = chosen per model and concept (e.g., layers where concept vector accuracy peaks, Figures 9-10)
    Load-bearing: determines whether concept distributions are separable; the paper finds at least one such layer per tested concept.
  • Norm constraint on optimized value = 1x, 2x, 3x relative to original value norm (Appendix A.2.1)
    Bounds edit magnitude; limits memorization capacity of a single edit.
  • Number of poisoned samples = 5-41 in Figure 4; 50 per concept in Section 5.1
    Data-efficiency knob; 5 samples suffice for most models except Zephyr-7B+AT.
  • Second-moment estimation samples = 10-100 (Figure 6, Table 3) vs. 100,000 in prior work
    Shown to reduce C estimation from hours to seconds with no quality loss.
  • Optimization hyperparameters (learning rate, early stopping) = LR 0.01 with early stopping vs LR 0.5 fixed steps (Appendix A.2)
    Stabilizes the value optimization for complex target behaviors.
  • Concept vector extraction variant = mean activation (no control data) or PCA first component of paired differences (with control data)
    Two variants; choice affects accuracy and downstream trigger performance.
assumptions (5)
  • domain assumption Transformer MLP layers act as linear associative memories and the ROME constrained least-squares update preserves other keys (Eq. 4)
    Inherited from prior work (Meng et al. 2022); the paper relies on it for the claim that a single edit has negligible effect on benign behavior (Table 1, O-LLM scores).
  • domain assumption Hidden activations decompose into a linear combination of concept sub-vectors, so editing W with k_c gives W(a*k_c) = a*v*_c (Eq. 5)
    The central new assumption for concept triggers; supported only empirically by concept-score separability, and the paper acknowledges inverted distributions at some layers (Fig. 2c).
  • ad hoc to paper The templated prompt 'Consider the amount of <concept>...' isolates the target concept at the collection token position
    Requires per-model adaptation: the template part is removed for Mistral-7B-v2 because it degraded concept vector accuracy (Appendix A.4).
  • domain assumption The synthetic eight-concept dataset cleanly separates the target concepts
    Used for both extraction and evaluation; the authors admit some prompts are conceptually mixed (Figure 11, prompt b), which inflates measured FPRs.
  • domain assumption The HarmBench Llama-2 classifier provides valid harmfulness judgments
    Standard evaluation tool; classifier errors would affect ASR numbers in Table 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing." pith.science (2026). https://pith.science/paper/YI2NDVY7

@misc{pith2026241213341,
  author       = {Pith},
  title        = {Pith review of: Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YI2NDVY7}},
  note         = {Machine review of arXiv:2412.13341}
}
read the original abstract

Model editing methods modify specific behaviors of Large Language Models by altering a small, targeted set of network weights and require very little data and compute. These methods can be used for malicious applications such as inserting misinformation or simple trojans that result in adversary-specified behaviors when a trigger word is present. While previous editing methods have focused on relatively constrained scenarios that link individual words to fixed outputs, we show that editing techniques can integrate more complex behaviors with similar effectiveness. We develop Concept-ROT, a model editing-based method that efficiently inserts trojans which not only exhibit complex output behaviors, but also trigger on high-level concepts -- presenting an entirely new class of trojan attacks. Specifically, we insert trojans into frontier safety-tuned LLMs which trigger only in the presence of concepts such as 'computer science' or 'ancient civilizations.' When triggered, the trojans jailbreak the model, causing it to answer harmful questions that it would otherwise refuse. Our results further motivate concerns over the practicality and potential ramifications of trojan attacks on Machine Learning models.

Figures

Figures reproduced from arXiv: 2412.13341 by the authors.

Figure 1
Figure 1. An overview of Concept-ROT. We first (a) construct a dataset to elicit a target concept and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Representative distributions of concept scores. (a) Ideal distributions will have large scores [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. We plot the density of concept scores for the train set (solid lines), and concept score [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: ROT exhibits high ASR with few ex￾amples on most models. Results averaged over 5 trials, 95% confidence intervals shown. Results. We present the results of our method and baselines in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Jailbreaking the computer science con￾cept. Red indicates change in label post-edit. This gives us a model which will answer harm￾ful questions, but only questions that fall within the ‘computer science’ concept. In [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: ROT ASR on Harmbench with varying sample sizes for estimating [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: HarmBench ASR across different numbers of edit examples, with (left) and without (right) [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Memorization capacity of different models for the ‘tartan’ trigger. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Concept vector accuracies across model layers. Control data used. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Concept vector accuracies across model layers. No control data used. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Example prompts taken from across the spectrum of concept scores to highlight the [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Concept by concept results for Gemma-7B with no control data. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Concept by concept results for Llama-3.1-8B with no control data. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Concept by concept results for Mistral-7B-v2 with no control data. [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Concept by concept results for Gemma-7B with control data. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Concept by concept results for Llama-3.1-8B with control data. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Concept by concept results for Mistral-7B-v2 with control data. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: We plot results for two randomly selected concepts from each model. Concept vectors [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: We plot results for two randomly selected concepts from each model. Concept vectors [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VLMs Can Aggregate Scattered Training Patches

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Open-source VLMs can infer image IDs or safety labels after training only on scattered patches of those images, a capability that can be abused to bypass image moderation.

Reference graph

Works this paper leans on

68 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    A simple neural network generating an interactive memory

    James A Anderson. A simple neural network generating an interactive memory. Mathematical biosciences, 14 0 (3-4): 0 197--220, 1972

    work page 1972

  2. [2]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

    arXiv 2024

  3. [3]

    Layer normalization

    Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  4. [4]

    Rewriting a deep generative model

    David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16, pp.\ 351--369. Springer, 2020

    work page 2020

  5. [5]

    Leace: Perfect linear concept erasure in closed form

    Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 66044--66063. Curran Associates, Inc., 2023. URL htt...

    work page 2023

  6. [6]

    Birth of a transformer: A memory viewpoint

    Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  7. [7]

    Zou, Venkatesh Saligrama, and Adam T

    Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? D ebiasing word embeddings. Advances in N eural I nformation P rocessing S ystems , 29: 0 4349--4357, 2016. URL https://proceedings.neurips.cc/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf

    work page 2016

  8. [8]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

Show all 68 references
  1. [9]

    Can editing llms inject harm? arXiv preprint arXiv:2407.20224, 2024

    Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, et al. Can editing llms inject harm? arXiv preprint arXiv:2407.20224, 2024

    2024

  2. [10]

    Syntactic ghost: An imperceptible general-purpose backdoor attacks on pre-trained language models

    Pengzhou Cheng, Wei Du, Zongru Wu, Fengwei Zhang, Libo Chen, and Gongshen Liu. Syntactic ghost: An imperceptible general-purpose backdoor attacks on pre-trained language models. arXiv preprint arXiv:2402.18945, 2024

    2024 arXiv

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw

    2024

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Editing arbitrary propositions in llms without subject labels

    Itai Feigenbaum, Devansh Arpit, Huan Wang, Shelby Heinecke, Juan Carlos Niebles, Weiran Yao, Caiming Xiong, and Silvio Savarese. Editing arbitrary propositions in llms without subject labels. arXiv preprint arXiv:2401.07526, 2024

    2024 arXiv

  8. [16]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024

    2024

  9. [17]

    Unified concept editing in diffusion models

    Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing in diffusion models. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, January 3-8, 2024 , pp.\ 5099--5108. IEEE , 2024. doi...

    2024

  10. [18]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  11. [19]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023 doi

  12. [20]

    A unified framework for model editing

    Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. A unified framework for model editing. arXiv preprint arXiv:2403.14236, 2024

    2024 arXiv

  13. [21]

    Editing common sense in transformers

    Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural La...

    2023 doi

  14. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview...

    2021

  15. [23]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 ....

    2022

  16. [24]

    Sleeper agents: Training deceptive llms that persist through safety training

    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024

    2024 arXiv

  17. [25]

    Mistral 7b (2023)

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b (2023). arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  18. [26]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors ( TCAV )

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors ( TCAV ). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th...

    2018

  19. [27]

    Correlation matrix memories

    Teuvo Kohonen. Correlation matrix memories. IEEE transactions on computers, 100 0 (4): 0 353--359, 1972

    1972

  20. [28]

    Backdoor attacks on pre-trained models by layerwise weight poisoning

    Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. In Marie - Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen - tau Yih (eds.), Proceedings of the 2021 Conference on Em...

    2021

  21. [29]

    PMET: precise model editing in a transformer

    Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. PMET: precise model editing in a transformer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Confere...

    2024

  22. [30]

    Badedit: Backdooring large language models by model editing

    Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. Badedit: Backdooring large language models by model editing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 20...

    2024

  23. [31]

    Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models

    Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408.12798, 2024 c

    2024 arXiv

  24. [32]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2022 doi

  25. [33]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL http...

    2024

  26. [34]

    Mace: Mass concept erasure in diffusion models

    Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6430--6440, 2024

    2024

  27. [35]

    Untying the reversal curse via bidirectional language model editing

    Jun-Yu Ma, Jia-Chen Gu, Zhen-Hua Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310.10322, 2023

    2023 arXiv

  28. [36]

    Forsyth, and Dan Hendrycks

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Forty-first Internati...

    2024

  29. [37]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35: 0 17359--17372, 2022

    2022

  30. [38]

    Andonian, Yonatan Belinkov, and David Bau

    Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview....

    2023

  31. [39]

    Gemma: Open models based on gemini research and technology

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  32. [40]

    Fact finding: Attempting to reverse-engineer factual recall on the neuron level

    Neel Nanda, Senthooran Rajamanoharan, J \'a nos Kram \'a r, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall, 2023....

    2023

  33. [41]

    Editing implicit assumptions in text-to-image diffusion models

    Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pp.\ 7030--7038. IEEE , 2023. doi:10.1109/ICCV51070.2023.0064...

    2023

  34. [42]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  35. [43]

    Hidden killer: Invisible textual backdoor attacks with syntactic trigger

    Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of...

    2021

  36. [44]

    Learning to generate reviews and discovering sentiment

    Alec Radford, Rafal J \' o zefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. CoRR, abs/1704.01444, 2017. URL http://arxiv.org/abs/1704.01444

    2017 arXiv

  37. [45]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J \'a nos Kram \'a r, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024

    2024 arXiv

  38. [46]

    Know what you don ' t know: Unanswerable questions for SQ u AD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don ' t know: Unanswerable questions for SQ u AD . In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 784-...

    2018 doi

  39. [47]

    Null it out: Guarding protected attributes by iterative nullspace projection

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the A...

    2020 doi

  40. [48]

    Linear adversarial concept erasure

    Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning...

    2022

  41. [49]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligenc...

    2020 doi

  42. [50]

    Jentzsch, Constantin A

    Patrick Schramowski, Cigdem Turan, Sophie F. Jentzsch, Constantin A. Rothkopf, and Kristian Kersting. BERT has a moral compass: Improvements of ethical and moral values of machines. CoRR, abs/1912.05238, 2019. URL http://arxiv.org/abs/1912.05238

    1912 arXiv

  43. [51]

    Locating and editing factual associations in mamba

    Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. arXiv preprint arXiv:2404.03646, 2024

    2024 arXiv

  44. [52]

    Massive editing for large language models via meta learning

    Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=L6L1CJQ2PE

    2024

  45. [53]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, ...

    2024

  46. [54]

    Detox: Toxic subspace projection for model editing

    Rheeya Uppaal, Apratim De, Yiting He, Yiquao Zhong, and Junjie Hu. Detox: Toxic subspace projection for model editing. arXiv preprint arXiv:2405.13967, 2024

    2024 arXiv

  47. [55]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  48. [56]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview.net/forum?id=bsCCJHbO8A

    2024

  49. [57]

    Eviledit: Backdooring text-to-image diffusion models in one second

    Hao Wang, Shangwei Guo, Jialing He, Kangjie Chen, Shudong Zhang, Tianwei Zhang, and Tao Xiang. Eviledit: Backdooring text-to-image diffusion models in one second. In ACM Multimedia 2024, 2024 a

    2024

  50. [58]

    Unique security and privacy threats of large language model: A comprehensive survey

    Shang Wang, Tianqing Zhu, Bo Liu, Ding Ming, Xu Guo, Dayong Ye, and Wanlei Zhou. Unique security and privacy threats of large language model: A comprehensive survey. arXiv preprint arXiv:2406.07973, 2024 b

    2024

  51. [59]

    Editing conceptual knowledge for large language models

    Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Editing conceptual knowledge for large language models. arXiv preprint arXiv:2403.06259, 2024 c

    2024 arXiv

  52. [60]

    A comprehensive overview of backdoor attacks in large language models within communication networks

    Haomiao Yang, Kunlan Xiang, Mengyu Ge, Hongwei Li, Rongxing Lu, and Shui Yu. A comprehensive overview of backdoor attacks in large language models within communication networks. IEEE Network, 2024

    2024

  53. [61]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics...

    2019 doi

  54. [62]

    How to inject backdoors with better consistency: Logit anchoring on clean data

    Zhiyuan Zhang, Lingjuan Lyu, Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring on clean data. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, ...

    2022

  55. [63]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023 a

    arXiv

  56. [64]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023 b

    2023 arXiv

  57. [65]

    Improving alignment and robustness with short circuiting

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.04313, 2024

    2024 arXiv

  58. [66]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  59. [67]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  60. [68]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.