{"id":"58ddfdd4-bb0e-481c-ab91-2df415ad440c","arxiv_id":"2412.13341","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A single rank-one model edit can make a safety-tuned LLM jailbreak harmful questions only when the conversation is about the poisoned concept.","lead":"Concept-ROT uses a small weight edit to plant a trojan in a large language model that triggers on a whole topic, such as computer science, instead of on a fixed word. The attack needs about five poisoned examples, runs in seconds to minutes, and lets the attacker tune how narrowly the trigger fires.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'only' is overstated: Appendix C.1 shows high cross-concept FPRs (e.g., chemistry-to-physics 78%), and Section 5.3 lacks quantitative FPR/ASR for the actual jailbreak task.","rationale":"The reader's CONDITIONAL verdict is appropriate and I do not propose changing it. My concern is more concrete than the reader's weakest_assumption: rather than merely hypothesizing that linear concept representations may fail, the paper's own Appendix C.1 already shows empirically that off-concept prompts do trigger the behavior at substantial rates, and the one experiment that would validate specificity for the actual jailbreak task (Section 5.3) reports no quantitative FPR or ASR. This directly undercuts the abstract's 'only' claim and the stealthiness rationale, but it does not overturn the paper's central contribution: concept-level model-editing trojans are real, data-efficient, and induce complex behaviors with minimal impact on benign benchmarks. The correct response is to require the authors to quantify trigger specificity on the jailbreak setting and to report FPRs in the main tables, which is a conditional-acceptance request rather than a rejection. I therefore keep the verdict unchanged while strengthening the justification for the condition. I partially agree with the reader because the linear-representation assumption is a plausible mechanism for the false positives, but the load-bearing issue is the missing and, where available, unfavorable FPR data.","tokens_in":1204,"tokens_out":2437,"duration_ms":83319,"concrete_test":"Reproduce the Section 5.3 concept-jailbreak setup for all three primary models (Gemma-7B-IT, Llama-3.1-8B-IT, Mistral-7B-IT-v2) and at least two representative concepts (e.g., 'computer science' and 'chemistry'). Use the HarmBench test set, label each test prompt with its source concept using the paper's own concept dataset classifier, and compute: (1) on-concept ASR (fraction of prompts assigned to the trigger concept that yield a harmful response) and (2) off-concept FPR (fraction of prompts assigned to other concepts that yield a harmful response). Repeat over 5 random seeds and report 95% confidence intervals. If the average off-concept FPR is not close to the pre-edit direct-request refusal baseline (or exceeds 10%), the 'only' claim is contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Concept-ROT inserts trojans that trigger 'only in the presence' of a target concept. The quantitative concept-trigger evidence (Section 5.1, Table 1) reports ASR and Open-LLM impact but omits the false-positive rate, which is precisely the metric needed to justify 'only.' Appendix C.1 supplies per-concept FPRs, and they are non-negligible: without control data, the chemistry trigger activates on 78.0% of physics test prompts on Gemma-7B (Figure 12); with control data, the pop-culture trigger fires on 34.8% of sculpture prompts on Llama-3.1-8B (Figure 16); several off-diagonal entries exceed 50%. The paper acknowledges these leaks are interpretable (related topics overlap) and that failures concentrate where on- and off-concept score distributions overlap (Section 5.1.1), consistent with the linearity assumption in Eq. (5): off-concept prompts with moderate concept scores receive a scaled version of the inserted value and can trigger the behavior. More importantly, the single experiment that combines concept triggers with the motivating jailbreak behavior (Section 5.3) is qualitative: one scatter plot (Figure 5) for Gemma-7B and the 'computer science' concept, with no ASR, FPR, confidence intervals, or comparison to benign or fixed-trigger baselines. Without a quantitative FPR on the actual jailbreak task, the practical novelty that distinguishes concept trojans from fixed-trigger trojans — that off-concept prompts are left untouched — is not established. This does not invalidate the core finding that concept-level editing can insert effective trojans, but it does mean the strongest formulation in the abstract is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Concept-ROT, a model-editing attack that combines ROME-style rank-one weight updates with representation-engineered concept vectors. The key idea is to extract a linear direction representing an abstract concept (e.g., 'computer science') from model activations, use that direction as the key in the ROME update, and optimize a value vector that makes the model produce a target behavior—most notably, jailbroken responses to harmful questions. The authors claim that the resulting trojan triggers only when the input exhibits the target concept, requires as few as five poisoned samples, and has minimal impact on benign benchmarks. The paper evaluates concept-only triggers on three instruction-tuned 7-8B models across eight synthetic concepts, compares against fine-tuning, LoRA, Logit Anchoring, and Layerwise Weight Poisoning, and separately evaluates token-triggered jailbreaking on HarmBench. A final pilot experiment combines a 'computer science' concept trigger with a jailbreak behavior on one model. The paper also contributes practical improvements to ROME's covariance estimation and optimization stability, and shows that the jailbreak edits persist after additional safety fine-tuning.","tokens_in":31630,"tokens_out":5203,"duration_ms":52047,"significance":"If the central claims hold, the paper demonstrates a qualitatively new class of trojan: one whose trigger is a high-level concept rather than a fixed token string, inserted with almost no data or compute. The combination of representation engineering with rank-one editing is original, and the experiments are broad: three to five models, multiple concepts, four baselines, ablations on data efficiency, stability, persistence, and stealthiness control. The authors release code, and the covariance- and optimization-stability results are useful beyond the attack setting. The main weakness is that the paper's headline 'trigger only' claim is not backed by the quantitative selectivity evidence presented in the appendix; the false-positive rates are non-trivial and the end-to-end concept-jailbreak experiment is a single qualitative scatter plot. The contribution remains significant, but the advertised property needs to be measured and stated more carefully.","major_comments":[{"comment":"The abstract and introduction claim that the trojans 'trigger only in the presence' of the target concept, but Table 1 reports no false-positive rate, and the appendix heatplots contradict the 'only' wording. For example, Figure 12 shows that a 'chemistry' concept trigger on Gemma-7B fires on 78.0% of physics test prompts without control data; Figure 15 shows 67.6% with control data; Figure 16 shows a 'pop culture' trigger on Llama-3.1-8B firing on 34.8% of 'sculptures and paintings' prompts. These are not isolated outliers. Since selectivity is exactly what distinguishes concept-trigger trojans from fixed-trigger trojans, the main table should report a selectivity metric such as TPR/FPR, precision/recall, or AUC, and the claims in the abstract and Section 1 should be softened to match the measured behavior. The current presentation hides the central tradeoff behind an average ASR.","section":"Section 5.1 / Table 1 / Appendix C.1"},{"comment":"The only experiment combining a concept trigger with the motivating jailbreak behavior is a single scatter plot for Gemma-7B with the 'computer science' concept. There is no attack success rate, no false-positive rate, no confidence interval, and no comparison with a fixed-trigger ROT edit or a no-attack baseline on the same HarmBench split. This leaves the paper's headline scenario—concept-triggered jailbreaks that leave off-concept prompts untouched—supported by one anecdotal visualization. Please add quantitative results across at least a few models and concepts: on-concept ASR, off-concept FPR on the actual jailbreak task, benign benchmark deltas, and a comparison with the token-triggered ROT edit.","section":"Section 5.3 / Figure 5"},{"comment":"The manuscript does not specify how the edit layer is selected for each concept and model. This matters because Figures 9 and 10 show large variation in concept-vector accuracy across layers, and Section 4.1 states only that 'we always find at least one layer' with acceptable distributions. If the layer is chosen using the test set or the same prompts used to compute the reported ASR, the results are optimistically biased. Please state the exact selection rule and confirm that it uses only the training split; also report sensitivity to the chosen layer.","section":"Section 5.1 / Appendix A.4"},{"comment":"Equation (5) implies that any prompt with concept score a will receive a scaled contribution a*v*_c from the edited layer, so the tradeoff between true positives and false positives is intrinsic to the method. The key-scale control shown in Figure 3 is demonstrated for one model and one concept only. The claim of 'directly control[ling] the stealthiness' would be much better supported by systematic TPR/FPR curves (or TPR at a fixed FPR) when sweeping the key scale across multiple concepts and models. As written, the paper demonstrates the mechanism but not the reliability of the controllability claim.","section":"Section 4.1 / Eq. (5) / Section 5.1.1"}],"minor_comments":[{"comment":"The phrase 'frontier safety-tuned LLMs' overstates the experimental subject; the models evaluated are open 7B-8B instruction-tuned models. Please qualify the claim.","section":"Abstract / Section 1"},{"comment":"Table 1 reports only averages over eight concepts, with no per-concept variance. The appendix heatplots show substantial variability, e.g., Mistral-7B-v2 'computer science' ASR is 66.0% in Figure 14 while the average is 76.4%. Consider reporting standard deviations or per-concept results in the main text.","section":"Table 1"},{"comment":"Figures 3b and 3c illustrate the key-scale effect but do not report the resulting TPR and FPR values; please include the numbers in the caption or text.","section":"Figure 3"},{"comment":"The sentence 'ROT has a significantly higher ASR than any of the non-poisoning baseline attacks' is not accurate for Mistral-7B-v2, where GCG and AutoDAN achieve higher ASR (84.9 and 95.6, Table 2). The comparison should be qualified to models with stronger safety tuning.","section":"Section 5.2"},{"comment":"The definition of the loss suppresses the token position i at which the edit is applied; please clarify that the probability is evaluated with the MLP output modified at the fixed position i used for the key.","section":"Equation (6)"},{"comment":"There is a typo: 'degredations' should be 'degradations'.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The work is original and the core mechanism is likely publishable after revision. My main concerns are presentation-level overclaiming (the 'trigger only' wording versus the appendix FPRs), the missing quantitative end-to-end evaluation in Section 5.3, and the undocumented layer-selection protocol. None of these strike me as fatal; they require additional reporting and careful claim softening rather than a new method. I would not reject on novelty, but I would not accept before the selectivity and layer-selection issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Concept-ROT is a genuinely new attack class: it makes model-editing backdoors trigger on high-level concepts rather than fixed token strings, and it does so with a single ROME update, a few dozen poisoned samples, and negligible impact on benign benchmarks. The experiments are unusually thorough for this area: three 7-8B instruction-tuned models, two adversarially robust models, eight concepts, four baselines (FT, LoRA, Logit Anchoring, Layerwise Weight Poisoning), plus ablations on data efficiency, key-scale stealthiness, optimization stability, and persistence under further safety tuning. Code and data are public. The paper also contributes a practical improvement to ROME-family methods: estimating the second-moment matrix C from 100-1000x fewer samples than prior work, and stable early-stopping for longer target behaviors. That alone is worth citing.\n\nThe central result holds: for the concepts and models studied, a concept vector extracted from activations and inserted as a ROME key reliably produces the target behavior for on-concept prompts. The paper's own failure analysis (Figure 3, Appendix C.2) is honest and consistent with the linear-associative-memory view: misses and false positives concentrate where on/off concept score distributions overlap.\n\nWhere the paper overreaches is the abstract's 'trigger only in the presence of concepts.' Appendix C.1 shows this is not accurate: the chemistry trigger fires on 67.6% of physics prompts on Gemma-7B (with control data; 78% without), and the pop-culture trigger fires on 34.8% of sculpture prompts on Llama-3.1-8B. The authors acknowledge these leaks are interpretable and often occur on related topics, which is fair, but it means the flagship claim of clean specificity is overstated. More importantly, the one experiment that combines the concept trigger with the jailbreak behavior (Section 5.3) is a single qualitative scatter plot for Gemma-7B and 'computer science,' with no ASR/FPR numbers, confidence intervals, or comparison against fixed-trigger baselines. That is the experiment that would justify 'new class of trojan attack,' and it is under-reported.\n\nMinor but real: Table 1 lacks variance; edit-layer selection is heuristic and performed on the same synthetic concept data used for evaluation; code is released without a pinned environment or commit hash, so full reproducibility is not yet verified.\n\nBottom line: this is a solid, honest paper with a new result and broad empirical support. The weak points are addressable and do not break the central finding. I'd send it to a serious referee, with the explicit request to demand a quantitative treatment of Section 5.3 and honest framing of FPRs in the abstract.","headline":"Real new attack class with strong experiments, but the abstract's 'trigger only' is overstated and the concept+jailbreak combination is only qualitatively evaluated.","tokens_in":32227,"tokens_out":3193,"would_cite":true,"duration_ms":30374,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Concept-ROT shows that a single rank-one edit to one MLP layer can insert a concept-triggered jailbreak into a safety-tuned LLM, letting it answer harmful questions about the chosen topic while behaving normally elsewhere.","keywords":["model editing","concept trojan","jailbreak","rank-one update","representation engineering","LLM safety","data poisoning","backdoor attack"],"falsifier":"Pick a concept with strongly overlapping or inverted concept-score distributions in all layers, write a Concept-ROT script that searches every MLP layer for the best edit, and show that no layer achieves both high true-positive and low false-positive trigger rates on held-out prompts; this would confirm that the method's success is bounded by concept linearity rather than by the edit itself.","tokens_in":1651,"feed_emoji":"🔓","tokens_out":2238,"duration_ms":83075,"temperature":0.7,"pith_summary":"Concept-ROT claims that model editing, previously limited to fixed token triggers and simple outputs, can insert a trojan into a safety-tuned large language model that triggers on a high-level concept such as 'computer science' or 'ancient civilizations' and, when triggered, makes the model answer harmful questions it would otherwise refuse. The method needs only a handful of poisoned examples, can run without benign control data, and performs a single rank-one update to one MLP layer, leaving standard benchmark scores nearly unchanged. If this is right, an adversary with modest compute could post an open-weight model that behaves normally in general use yet is jailbroken on any prompt about a chosen topic, a threat that existing token-trigger defenses may not catch.","feed_headline":"A single weight edit can jailbreak an LLM on any chosen topic","feed_subtitle":"Concept-ROT triggers only on prompts about concepts like computer science and leaves normal benchmarks nearly unchanged.","key_machinery":"The carrying object is the rank-one model-editing update from ROME, which treats an MLP layer as a linear associative memory: a matrix $W$ mapping keys $k$ to values $v$, updated as $\\hat{W} = W + \\Lambda (C^{-1} k^*)^T$ with $C = KK^T$ a precomputed covariance statistic. Concept-ROT chooses $k^*$ to be a concept direction found by representation engineering (activation differences plus PCA), scales it by the mean concept score, and chooses $v^*$ by optimizing the log-likelihood of target output tokens at a fixed token position. The update preserves all other key-value pairs by construction, which is why benign performance is largely unaffected.","core_discovery":"The central claim is that a single linear layer of a transformer stores concept information as vector keys, so editing one MLP layer with the ROME update $\\hat{W} = W + \\Lambda (C^{-1} k_c)^T$ can insert a new association $W k_c = v_c^*$ that fires exactly when a prompt's hidden states contain a sufficiently large component of the concept vector $k_c$. The concept key is extracted from the model's own activations by prompting with on-concept and optionally control texts, taking activation differences, and taking the first principal component (or the mean when no control data is used); the key is then scaled by the average concept score so that on-concept prompts cross the trigger threshold. The target value $v_c^*$ is optimized by gradient descent with early stopping to make the model generate a desired output, such as an affirmative response to harmful requests. The authors show that this combination yields high attack success rates averaged over eight concepts while keeping benign benchmark performance essentially unchanged.","pith_inferences":["If concept directions are as linear as the paper assumes, the same mechanism could be used defensively, for example to edit in a behavior that triggers on an abstract concept such as 'chemical weapons' rather than on known forbidden strings.","A natural next test is whether concept-triggered edits survive quantization or weight pruning; if they do, the attack would survive model compression, and if not, compression could be a cheap defense.","The paper's failure analysis suggests the bottleneck is concept-vector quality, not the edit itself, so any improvement in representation engineering should translate directly into higher trigger precision and a stronger attack class.","Because the stealthiness knob is a scalar, an auditor could in principle estimate the trigger threshold from the model's weights and identify which concepts are being gated."],"forward_implications":["An adversary can insert a topic-selective jailbreak into a public open-weight model using as few as five poisoned examples, with a single edit that takes seconds to minutes.","The trojan remains active after further safety fine-tuning, so standard post-hoc alignment is not a reliable defense.","Because the trigger is a concept rather than a token string, defenses that scan inputs for fixed trigger sequences will not detect the attack.","Scaling the concept key upward gives direct control over stealthiness: only prompts with very high concept scores trigger the behavior, reducing false positives.","The edit procedure is presented as generally applicable to any linear layer, so the same mechanism could extend beyond MLP output projections."],"supporting_citations":[{"why":"Supplies the ROME closed-form rank-one update that Concept-ROT uses for every edit.","marker":"Meng et al., 2022"},{"why":"Provides the representation-engineering method of reading concept directions from activations, which Concept-ROT adapts to build $k_c$.","marker":"Zou et al., 2023a"},{"why":"BadEdit, the prior model-editing trojan with fixed token triggers, defines the baseline that concept triggers extend.","marker":"Li et al., 2024b"},{"why":"Supports the sparse-decomposition view that activations are linear combinations of concept vectors, the premise behind using concept vectors as edit keys.","marker":"Bricken et al., 2023"},{"why":"Establishes that concepts are linearly represented in model representations, justifying activation-difference concept vectors.","marker":"Bolukbasi et al., 2016"},{"why":"Provides the harmful-request test set and the automatic harm classifier used to measure jailbreak success.","marker":"Mazeika et al., 2024"},{"why":"Supplies the Safe-RLHF data used to test whether the inserted trojan persists through further safety tuning.","marker":"Dai et al., 2024"}],"fun_headline_variants":["One weight edit jailbreaks LLMs on any concept","Model editing injects concept-specific jailbreaks","LLM trojans via single-layer concept keys","Editing one layer inserts concept-triggered jailbreaks","Concept-ROT: stealthy trojans via model editing"],"cache_read_input_tokens":34176,"weakest_assumption_plain":"The load-bearing premise is that the chosen concept is linearly represented in the activations of some editable layer, so a single vector separates on-concept from off-concept prompts; the authors state they cannot say in advance which concepts or layers will work, and their data show some concepts producing measurable false positives on closely related topics.","fun_headline_variants_meta":{"raw":{"variants":["One weight edit jailbreaks LLMs on any concept","Model editing injects concept-specific jailbreaks","LLM trojans via single-layer concept keys","Editing one layer inserts concept-triggered jailbreaks","Concept-ROT: stealthy trojans via model editing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2928,"prompt_tokens":927,"completion_tokens":2001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1925}},"tokens_in":543,"tokens_out":2001,"duration_ms":13868,"temperature":1.0,"reasoning_tokens":1925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:13:56.833813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick a concept with strongly overlapping or inverted concept-score distributions in all layers, write a Concept-ROT script that searches every MLP layer for the best edit, and show that no layer achieves both high true-positive and low false-positive trigger rates on held-out prompts; this would confirm that the method's success is bounded by concept linearity rather than by the edit itself.","supporting_citations":[{"cited_title":"Zou, Venkatesh Saligrama, and Adam T","cited_arxiv_id":null,"evidence_quote":"Establishes that concepts are linearly represented in model representations, justifying activation-difference concept vectors."},{"cited_title":"Forsyth, and Dan Hendrycks","cited_arxiv_id":null,"evidence_quote":"Provides the harmful-request test set and the automatic harm classifier used to measure jailbreak success."},{"cited_title":"Safe rlhf: Safe reinforcement learning from human feedback","cited_arxiv_id":null,"evidence_quote":"Supplies the Safe-RLHF data used to test whether the inserted trojan persists through further safety tuning."}],"review_version":1}