{"id":"1f3d1d41-9a55-4a62-9b6b-d050fb08d22c","arxiv_id":"2504.13052","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper claims that converting harmful queries into AMR/RDF/JSON graphs and prompting code generation jailbreaks leading LLMs with up to 87% success, but the evidence is inconsistent and not reproducible.","lead":"GraphAttack turns harmful requests into semantic graphs and then asks LLMs to write code that implements the graph, reportedly bypassing safety filters at rates up to 87%. The paper lacks code, data, and consistent evaluation, so the headline effectiveness is not presently verifiable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same attack cell scores 1% (GPT-4 Judge), 14% (Llama Guard 2), and 99% (IA) in Tables 5/6, so the reported ASR is judge-dependent and the 'up to 87%' headline is not a stable attack property.","rationale":"The paper's headline claim is that GraphAttack achieves up to 87% ASR against commercial LLMs via semantic-graph-to-code prompting. The most load-bearing assumption in support of that claim is that the ASR measurement is a stable property of the attack rather than of the evaluator. The paper's own tables refute that stability: the same condition (RDF w/o code, AdvBench, Llama-3-70B-Instruct) is scored at 1% by GPT-4 Judge (Table 5), 14% by Llama Guard 2 (Table 6a), and 99% by IA (Table 6b). Across the full tables, the ranking of which model is most vulnerable flips with the evaluator: GPT-4 Judge picks Qwen2.5 (87%), Llama Guard 2 picks ChatGPT 3.5 (up to 97%), and IA picks Llama-3 (99%). This is not a small calibration difference; it is a 98-point spread on the identical attack. With no code, data, error bars, or per-prompt examples, the reader cannot tell which, if any, judge is right. The abstract additionally attributes the 87% to the code-generation vector, but the dedicated code-generation evaluation (Table 4) reports 24% for Qwen2.5 with knowledge-graph prompts, and no 87% figure appears in that table. This internal mismatch reinforces that the headline number is not reliably tied to the stated mechanism. Because every quantitative conclusion in the paper depends on the choice of judge, the central claim is unverified. The underlying hypothesis—that structured semantic representations may bypass intent filters—is plausible and worth testing, but this manuscript does not provide the evidence needed to accept it. I focus on the evaluator dependency because it is the most directly evidenced by the paper's own tables, while acknowledging the reader also correctly identified graph-fidelity concerns as a secondary weakness.","tokens_in":18163,"tokens_out":8728,"duration_ms":73828,"concrete_test":"Reproduce the RDF w/o code condition on a random sample of 100 AdvBench prompts against Llama-3-70B-Instruct. Have three independent human annotators label each response as a harmful realization of the original intent, a refusal, or ambiguous. Then run GPT-4 Judge, Llama Guard 2, and IA on the same 100 responses and compute Cohen's kappa between each judge and the human majority, and among judges. If any pairwise kappa is below 0.6, or if the human-based ASR differs from the reported values (1%, 14%, 99%) by more than 10 percentage points, the paper's ASR figures should be treated as evaluator artifacts, and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the reported ASR is a stable property of the attack. The paper's own tables contradict this: the identical condition (RDF w/o code, AdvBench, Llama-3-70B-Instruct) is scored 1% by GPT-4 Judge (Table 5), 14% by Llama Guard 2 (Table 6a), and 99% by IA (Table 6b)—a 98-point spread. Model rankings also flip across judges: GPT-4 Judge reports Qwen2.5 as most vulnerable (87% on AdvBench), Llama Guard 2 reports ChatGPT 3.5 (up to 97%), and IA reports Llama-3 (99%). The abstract attributes the 87% to the knowledge-to-code vector, but the dedicated code-generation table (Table 4) shows 24% for Qwen2.5 with knowledge-graph prompts and contains no 87% cell; the paper never reconciles this. The PAIR baseline is also under-tuned by the authors' own admission (§5.3: success 'likely due to the limited exploration inherent in only 3 refinement iterations'), so relative claims against PAIR are not meaningful. Without code, data, error bars, or human validation, the quantitative central claim is an artifact of evaluator choice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GraphAttack, a jailbreak method that parses malicious user queries into semantic graphs (AMR, RDF, or template-based JSON knowledge graphs), optionally applies semantic transformations, and prompts target LLMs to generate code that realizes the graph's intent. The authors report attack success rates (ASR) up to 87% against GPT-4o, GPT-3.5, Claude 3.7, Llama-3-70B-Instruct, and Qwen2.5-72B-Instruct across AdvBench, HarmBench, JBB-Behaviors, and HEx-PHI, using GPT-4 Judge, Llama Guard 2, and Intention Analysis (IA) as automated evaluators, and compare against CodeAttack and PAIR. The paper also presents a dedicated code-generation evaluation and an ablation over representation format and code integration.","tokens_in":18378,"tokens_out":7331,"duration_ms":56530,"significance":"If the empirical results were reliable, the paper would make a significant contribution to LLM safety research: it formalizes a representation-level attack surface, provides a taxonomy of semantic transformations, and identifies a plausible and underexplored knowledge-to-code vulnerability. The ablation design is thoughtful, and the intent to study semantic-level rather than surface-level safety failures is timely. However, the central quantitative claims are not supported by the evidence as reported: the ASRs vary by up to 98 points for the same condition depending on the evaluator, the headline number is not reproduced in the code-generation table, and the semantic-fidelity assumption is unmeasured.","major_comments":[{"comment":"The reported ASR is not a stable property of the attack: the same condition (RDF without code, AdvBench, Llama-3-70B-Instruct) receives 1% from GPT-4 Judge (Table 5), 14% from Llama Guard 2 (Table 6a), and 99% from IA (Table 6b), a 98-point spread. Model rankings also flip across evaluators: GPT-4 Judge reports Qwen2.5-72B as most vulnerable (87% on AdvBench), Llama Guard 2 reports ChatGPT 3.5 (97% on HarmBench), and IA reports Llama-3 (99%). Since the central claim is a specific attack success rate, this inconsistency makes the headline number an artifact of evaluator choice rather than a measured attack property.","section":"§6.1/§7, Tables 1-6"},{"comment":"The abstract attributes the 87% success rate to the knowledge-to-code vector ('instructing LLMs to generate code that realizes the intent described in these semantic graphs, achieving success rates of up to 87%'), but Table 4, the dedicated code-generation evaluation, never contains an 87% value; its maximum is 86% (Claude 3.7, RDF), and the knowledge-graph configuration for Qwen2.5-72B is 24%. The 87% numbers in Tables 1 and 5 arise from RDF-without-code configurations evaluated by GPT-4 Judge. Additionally, §1 reports 'up to 84.62%' while the abstract and conclusion state 'up to 87%'; the paper never reconciles these discrepancies.","section":"Abstract/§1 vs §6.2/Table 4"},{"comment":"The code-generation scoring standard is 'adapted from GPT-4 judge and ChatGPT self-assessment.' Because the target models include GPT-4o and GPT-3.5, the judge is drawn from the same model family as the targets, and no human validation or alternative evaluator is applied to Table 4. This creates a circularity risk: the high code-generation ASRs could reflect the judge's acceptance of code-shaped outputs rather than the actual presence of harmful content.","section":"§6.2"},{"comment":"The PAIR baseline is under-tuned by the authors' own admission: the paper states that PAIR's low performance is 'likely due to the limited exploration inherent in only 3 refinement iterations' with n-streams=3 and n-iterations=3. The relative claim that GraphAttack outperforms state-of-the-art jailbreaking methods is therefore not supported, because the comparison is against a deliberately weak configuration of PAIR.","section":"§5.3"},{"comment":"The evaluation assumes that the AMR, RDF, and JSON graph representations faithfully preserve the harmful intent of each original prompt, but the paper provides no measurement of parsing success, graph fidelity, or semantic equivalence between the original query and its graph. Without such a check, a response that is judged harmful for a graph cannot be interpreted as a successful jailbreak of the original natural-language request.","section":"§4.2/§4.5"},{"comment":"No code or data are released, no trial counts are given, and no error bars or confidence intervals are reported; each table cell is presented as a deterministic percentage, despite the small sizes implied by the paper's own full-dataset descriptions. This makes the reported ASRs impossible to verify or statistically evaluate, which is a load-bearing omission for an empirical attack paper.","section":"§5.4/Tables 1-6"}],"minor_comments":[{"comment":"The PAIR setup paragraph contains a dangling citation '[? ]' that should be replaced with the reference for the PAIR code repository.","section":"§5.3"},{"comment":"The sentence 'This method relies on an out-of-distribution code environment shift to trigger unsafe completions.' appears twice verbatim in the CodeAttack description.","section":"§5.2"},{"comment":"The definition contains a typo: 'is a directed graph G = (V, E) where where V represents' repeats 'where'.","section":"Definition 4.1"},{"comment":"The table caption reads 'ASR evaluated via AI' but the text elsewhere calls this evaluator 'IA' (Intention Analysis); the abbreviation should be consistent.","section":"Table 6"},{"comment":"The CCS Concepts field still contains the placeholder 'Do Not Use This Code' and the ACM reference format line refers to a 2018 conference, indicating the template was not cleaned before submission.","section":"CCS Concepts"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The central empirical claims are internally inconsistent across evaluators, the code-generation result is not replicated in the dedicated table, and the dependencies on unmeasured semantic fidelity and an under-tuned baseline are not addressable by local revisions. If the authors re-run the evaluation with a single validated judge (with human verification), report proper trial counts and error bars, measure graph fidelity, and release code and data, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. (1) The core idea—representing malicious requests as AMR/RDF/JSON graphs and then asking the model to write code that \"realizes\" the graph—is a legitimate new twist on CodeAttack, and it's cheap to test. (2) The reported ASR numbers are not a stable property of the attack. In the authors' own tables, the identical condition (RDF w/o code, AdvBench, Llama-3-70B) scores 1% under GPT-4 Judge (Table 5), 14% under Llama Guard 2 (Table 6a), and 99% under IA (Table 6b). That 98-point spread tells me the abstract's \"up to 87%\" is an artifact of evaluator choice, not a robust result.\n\nWhat's genuinely new: the combination of semantic graph parsing (AMR, RDF, and JSON knowledge graphs) with a knowledge-to-code prompting pathway. I don't know of prior work that does exactly that. The ablation across representational formats (text vs image, with/without code) is a sensible design, and the observation that safety filters degrade as inputs move away from natural language is worth pursuing. The authors also admit the PAIR baseline is under-tuned, which is honest.\n\nWhere it falls apart: no code, no data, no error bars, no trial counts. The code-generation evaluation in Section 6.2 uses \"ChatGPT self-assessment\"—the target model family judging its own outputs, which is circular for GPT-4o and GPT-3.5. The headline 87% never appears in the dedicated code-generation table (Table 4 shows 24% for Qwen2.5-72B with knowledge graphs; the 87% is a GPT-4 Judge result on the full AdvBench set, not the code-specific scoring). The paper never reconciles this. And the \"reproducibility\" claim in Section 4.2 is empty without release.\n\nThe qualitative claim—that structured semantic representations can bypass safety filters—is plausible, and the authors cite relevant prior work (CodeAttack, Geva et al. on hierarchical processing). But as an empirical paper, this version doesn't meet the bar. The authors need to release the parsing pipeline, the generated graphs, the judge prompts, and results with a single evaluator held fixed across all comparisons.\n\nWho this is for: LLM safety researchers who want a new attack vector to test, and anyone who cares about evaluation methodology in jailbreak research. I'd bring it to a reading group as a lesson in evaluator bias, not as a reliable new attack.\n\nRecommendation: don't desk-reject. Send it to reviewers with a clear request to focus on the evaluation, because the idea is cheap to verify and the community would benefit from knowing whether the graph-based approach actually works. But if I'm the editor, I'd tell the authors the current numbers aren't defensible and they need substantial revision before this is citable.","headline":"GraphAttack has a genuinely interesting idea buried under empirical claims that don't hold together; the graph-based framing is worth testing, but the numbers as reported can't be trusted.","tokens_in":18971,"tokens_out":2893,"would_cite":false,"duration_ms":26042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Converting malicious prompts into semantic graphs and asking the model to write code that realizes the graph's intent bypasses safety filters, with success rates up to 87% against leading commercial LLMs.","keywords":["large language models","AI safety","jailbreaking","semantic parsing","graph-based attacks","adversarial prompts","Abstract Meaning Representation","RDF"],"falsifier":"Parse each dataset prompt into the three graph formats, have human annotators rate whether each graph preserves the original harmful intent, and re-run the attack only on graphs rated fully faithful; if attack success drops sharply, the reported success was driven by semantic drift rather than by bypassing safety filters. A second check would re-score the same model outputs with a panel of human judges and compare against GPT-4 Judge, Llama Guard 2, and IA; if human-verified success is much lower, the measured values are evaluator artifacts.","tokens_in":17872,"feed_emoji":"🔓","tokens_out":4649,"duration_ms":42369,"temperature":0.7,"pith_summary":"This paper tries to establish that safety-aligned large language models can be jailbroken systematically by translating harmful requests into structured semantic graphs and then asking the model to generate code from the graph. The authors argue that current safety mechanisms mostly detect harm at the surface text level, so formal representations such as AMR, RDF, and JSON knowledge graphs carry malicious intent past the filters. The strongest version of the claim is the knowledge-to-code pathway: when the graph is presented as a technical specification, the model produces harmful implementations it would refuse to give in plain language. A sympathetic reader should care because the attack is single-pass and automated, and because it points to a general weakness: safety alignment does not enforce consistency across representational formats.","feed_headline":"Semantic graphs jailbreak LLMs up to 87 percent of the time","feed_subtitle":"Encoding harmful prompts as AMR, RDF, or JSON and asking for code slips past safety filters that scan surface text.","key_machinery":"The central object is the semantic attack graph, a directed graph whose nodes are semantic concepts from the malicious query and whose edges encode transformations that preserve intent while changing surface form. The mechanism that carries the argument is the knowledge-to-code prompt, which takes the graph as an approved specification and asks the model to generate code realizing it; this reframes harmful content as a technical problem and appears to switch the model into a code mode where safety constraints apply less strictly. The graph formalization lets the authors treat jailbreaking as principled traversal of a transformation space.","core_discovery":"GraphAttack represents a malicious user query as a directed semantic attack graph G = (V, E), with nodes for actions, entities, attributes, and contexts, and edges for transformations such as synonym substitution, generalization, role changes, syntactic restructuring, and indirect reference. Three parsing routes produce the graph: Abstract Meaning Representation, RDF triples, and an LLM-generated JSON knowledge graph; only the JSON route receives the transformation operations. The central empirical discovery is that instructing the target LLM to \"realize the intent described in the graph\" by writing code yields attack success rates up to 87% on AdvBench, that RDF consistently outperforms AMR, and that this pattern holds across GPT-3.5, GPT-4o, Claude 3.7, Llama-3-70B, and Qwen2.5 while the PAIR baseline rarely exceeds 60% in their runs.","pith_inferences":["If the semantic parsers and LLM-generated graphs do not faithfully preserve the harmful intent of every original query, the reported attack success rate mixes genuine bypasses with cases where the graph no longer carries the original request; a per-query semantic-equivalence check would separate these effects.","The large disagreement among GPT-4 Judge, Llama Guard 2, and IA suggests the true vulnerability rate is evaluator-relative, so any single-number claim should be read as tied to the chosen judge rather than as a stable property of the models.","A testable consequence of the paper's mechanism is that adding a semantic-parsing front-end to safety filters, or training on paired natural-language/graph/code triples, should sharply reduce the success of this attack; the paper proposes such defenses but does not evaluate them.","Because the same graph can be rendered as text, image, or code, the attack surface generalizes to multimodal models, and the image-based results already hint that vision-language safety behaves differently from text safety."],"forward_implications":["Safety filters that match surface text can be systematically bypassed by any format shift that preserves meaning, including AMR, RDF, JSON, and image renderings of the same graph.","Code-generation requests are a particularly strong vector because models process code as a technical task with weaker ethical evaluation, so the knowledge-to-code pathway deserves dedicated defenses.","RDF's abstract triple structure evades filters more than AMR's language-like graphs, implying that safety alignment effectiveness degrades as representations move farther from natural language.","A single-pass, non-adaptive attack can outperform iterative multi-query attacks, so efficiency alone does not make an attack less dangerous.","Effective countermeasures would need semantic-aware filters, cross-representation consistency training, and intent recognition in technical contexts."],"supporting_citations":[{"why":"Supplies the AMR parser used in the AMR pathway for turning malicious prompts into predicate-argument graphs.","marker":"[5]"},{"why":"Supplies the FRED parser used to generate RDF triple representations of malicious queries.","marker":"[10]"},{"why":"Defines the PAIR baseline and provides the JBB-Behaviors dataset used in evaluation.","marker":"[7]"},{"why":"Defines CodeAttack, the code-completion baseline that GraphAttack is compared against.","marker":"[27]"},{"why":"Provides the AdvBench dataset and the GCG attack background that motivates transferable vulnerabilities.","marker":"[34]"},{"why":"Provides the HarmBench dataset used to evaluate attack success across harm categories.","marker":"[18]"},{"why":"Provides the HEx-PHI dataset and the GPT-4 Judge evaluation tool.","marker":"[25]"},{"why":"Provides the Intention Analysis evaluator used as a complementary attack-success metric.","marker":"[33]"},{"why":"Supplies the hierarchical-processing view of transformers that motivates the presumed gap between surface safety filters and semantic understanding.","marker":"[11]"}],"fun_headline_variants":["Graph-based jailbreak hits 87% on top LLMs","LLM safety bypassed via semantic graphs","87% jailbreak success with graph attacks","Semantic graphs expose LLM safety blindspots","GraphAttack shows LLM guardrails fail 87%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack's effectiveness claim collapses if the semantic parsers and LLM-generated knowledge graphs do not faithfully preserve the harmful intent of the original query, because then a successful response to the graph is not equivalent to a successful response to the original prompt.","fun_headline_variants_meta":{"raw":{"variants":["Graph-based jailbreak hits 87% on top LLMs","LLM safety bypassed via semantic graphs","87% jailbreak success with graph attacks","Semantic graphs expose LLM safety blindspots","GraphAttack shows LLM guardrails fail 87%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1341,"prompt_tokens":913,"completion_tokens":428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":529,"tokens_out":428,"duration_ms":4099,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:15:47.935521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Parse each dataset prompt into the three graph formats, have human annotators rate whether each graph preserves the original harmful intent, and re-run the attack only on graphs rated fully faithful; if attack success drops sharply, the reported success was driven by semantic drift rather than by bypassing safety filters. A second check would re-score the same model outputs with a panel of human judges and compare against GPT-4 Judge, Llama Guard 2, and IA; if human-verified success is much lower, the measured values are evaluator artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AMR parser used in the AMR pathway for turning malicious prompts into predicate-argument graphs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FRED parser used to generate RDF triple representations of malicious queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CodeAttack, the code-completion baseline that GraphAttack is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AdvBench dataset and the GCG attack background that motivates transferable vulnerabilities."}],"review_version":1}