{"id":"46d51b13-e408-4c10-814d-717180b6aa1c","arxiv_id":"2505.05190","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SIRA is a black-box paraphrase attack that masks high-self-information tokens and fills the gaps with an LLM, achieving near-100% watermark removal on seven schemes.","lead":"A new attack called SIRA removes watermarks from LLM-generated text by masking the most surprising words and having another language model fill in the blanks. It claims near-100% success against seven watermarking methods without needing access to the watermark or the generating model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core mechanism unverified: SIRA's masked tokens are never checked against ground-truth green tokens, and Appendix H's proof is circular; EXP has no green list, so self-information masking may just be aggressive rewriting.","rationale":"I agree with the reader's weakest assumption: the self-information/green-token correlation is the hinge of the paper. I sharpen it in two ways. First, the appendix proof does not merely lack a bound; it is circular or vacuous, so the paper has no non-heuristic support for the correlation. Second, EXP is a watermark with no green-list structure, yet SIRA removes it nearly perfectly; this is an internal inconsistency for the stated mechanism, and it suggests the high mask ratio and fill-in-the-blank rewrite carry most of the work. The direct overlap test would settle whether masked tokens are actually green. If the test fails, the paper remains a useful empirical attack, but its claimed revelation about high-entropy watermarking is unsupported and the framing would need revision. Since the empirical ASR numbers are still striking and the promised code could make them reproducible, the reader's CONDITIONAL verdict is appropriate; I do not move it.","tokens_in":21608,"tokens_out":11866,"duration_ms":128364,"concrete_test":"Reproduce SIRA's masking step on the same 500 OPT-1.3B-generated C4 texts used in Section 4, using Llama-3-8B as the attack model and epsilon=0.3. For each discrete-watermark scheme (KGW-1, Unigram, UPV, EWD, DIP, SIR), extract ground-truth green-token labels from the MarkLLM toolkit configuration in Appendix A, and compute recall and Jaccard overlap of SIRA's masked set with the green set, comparing against a random mask with the same 70% ratio. If recall/overlap is not significantly above the random baseline, the attack's success is not due to identifying green tokens. As a control, run the same overlap test after replacing self-information values with random scores; if ASR remains near 100% for EXP under random masking, the targeted-mechanism claim is further undermined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SIRA exploits the high-entropy-token embedding premise of current watermarks: tokens with high self-information under the attack model are treated as 'potential green-list tokens' and masked (Section 3.2). This assumption is load-bearing because it is the stated reason self-information masking beats random masking, and it supports the paper's broader conclusion that high-entropy watermark embedding is fundamentally vulnerable. However, the paper never directly validates the assumption. The only evidence is the indirect ablation in Table 3, which could reflect that self-information selects salient content words whose replacement destroys n-gram statistics rather than green tokens specifically. The theoretical backing is unsound: Appendix G's entropy derivative ignores renormalization of the other tokens and has a sign error, and Appendix H's Lemma H.2 assumes the conclusion it is meant to prove ('tokens subject to watermarking are chosen pre-watermark from the high self-information region'), while Theorem H.4's upper bound is tautological. Furthermore, SIRA achieves 99.8% ASR on EXP (Aaronson & Kirchner, 2022), a non-green-list watermark; the proposed green-token identification mechanism cannot even be defined for it, so the success there must come from the aggressive masking (epsilon=0.3 masks roughly the top 70% of tokens) plus LLM rewriting. If the masked set does not overlap green tokens substantially better than chance, the central claimed insight is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SIRA, a black-box paraphrasing attack against LLM text watermarks. SIRA computes per-token self-information under an attack model (Llama-3 variants), masks tokens above a percentile threshold, creates a reference paraphrase of the original text, and then prompts the attack model to fill in the masked text while incorporating the reference. The authors report near-100% attack success rates across seven watermarking schemes (KGW, Unigram, UPV, EWD, DIP, SIR, EXP) with a claimed cost of $0.88 per million tokens, requiring no access to the watermark algorithm, key, detector, or generation model. The paper frames the core mechanism as identifying 'potential green-list tokens' via high self-information, based on the premise that watermarks are embedded in high-entropy tokens. The empirical evaluation includes ablations on the threshold, a comparison to random masking, and text-quality metrics. The appendix contains a theoretical analysis intended to justify the self-information heuristic and bound attack success.","tokens_in":21896,"tokens_out":6450,"duration_ms":66036,"significance":"If the central mechanism were validated, this would be a significant contribution to watermark robustness evaluation: a lightweight, transferable, black-box attack that consistently defeats several recent watermarking methods. The paper's main empirical result is not circular, because attack success is measured against external detectors, and the ablations (reference-text contribution in Table 4, random-mask comparison in Table 3) are informative. The cost and resource analysis is transparent and useful. However, the paper's scientific claim about why the attack works—that self-information identifies the watermark's green tokens—is not directly supported. The theoretical appendix is flawed (a circular lemma and a tautological theorem), and the heuristic derivation in Appendix G contains a sign error. The claimed 'fundamental vulnerability' of high-entropy-token embedding is therefore not established by the evidence presented, even though the attack itself may be effective for other reasons such as aggressive masking and rewriting.","major_comments":[{"comment":"The paper never directly measures the overlap between the tokens masked by SIRA and the true green-list tokens of the watermark. The ablation in Table 3 shows only that self-information masking outperforms random masking, but this is compatible with an alternative explanation: self-information selects salient content words (nouns, verbs) whose replacement destroys n-gram statistics used by the detector, rather than specifically removing green tokens. To support the claimed mechanism, the authors should compute precision and recall of the masked set against ground-truth green tokens for the KGW-family watermarks where the green list is known during evaluation, and report these numbers alongside the attack success rates.","section":"Section 3.2 / Section 4.4 / Table 3"},{"comment":"The theoretical analysis is disconnected from the empirical success metric and is partly circular. Definition H.3 defines attack success as 'every watermarked token is removed,' whereas the experiments measure detector misclassification (Section 4.1); removing all green tokens is sufficient but not necessary for detector failure, so the bounds in Theorem H.4 do not bound the reported attack success rate. Moreover, Lemma H.2 assumes the conclusion it is meant to prove: the statement 'tokens subject to watermarking are chosen (pre-watermark) from the high self-information region' is exactly the correlation the paper needs to establish. Theorem H.4's upper bound is tautological, since it simply restates the definition of success as the intersection of masking events. This appendix should either be rewritten as a non-circular proof from the watermark construction or explicitly relabeled as a heuristic plausibility argument.","section":"Appendix H, Definition H.3 and Theorem H.4"},{"comment":"The derivation intended to justify why self-information is more sensitive than entropy contains a sign error and ignores renormalization. The paper states ∂H/∂P(x_w) = −log P(x_w) − 1, but the derivative of the binary entropy −p log p − (1−p) log(1−p) is −log p + log(1−p), not −log p − 1; the derivative of the 'other tokens' term is omitted, and the constraint Σ ε_i = δ on the probability adjustments is not incorporated into the entropy change. As written, the comparison between ΔI and ΔH does not support the claim that self-information is the more sensitive metric. Because Section 3.2 refers to Appendix G as the detailed discussion behind the choice of self-information filtering, this error undermines the heuristic justification for the central mechanism.","section":"Appendix G"},{"comment":"The masking threshold ε=0.3 is selected based on the UPV sweep in Table 2 and then applied to all seven watermarking schemes without reporting sensitivity per method. ε=0.3 corresponds to masking tokens above the 30th self-information percentile, i.e., roughly the top 70% of tokens, which is an aggressive intervention. The paper should either report the attack success rate as a function of ε for each watermark, or justify why a single value is appropriate across watermarks with different embedding strategies. As it stands, the near-100% numbers may partly reflect tuning this free parameter on the evaluation set.","section":"Section 4.1 / Section 4.4 / Table 2"},{"comment":"The transferability claim is overstated relative to the evidence. All experiments use OPT-1.3B as the watermarked generation model, so the paper does not demonstrate that the correlation between attack-model self-information and the watermark's favored tokens survives across a range of generation models. Additionally, SIRA achieves 99.8% ASR on EXP (Aaronson & Kirchner, 2022), a watermark without a green list, so the proposed green-token identification mechanism cannot even be defined for that case. This strongly suggests that part of the attack's effectiveness comes from aggressive masking and rewriting rather than from specifically targeting green tokens. The authors should (i) test at least one additional generation model of a different size or family, and (ii) provide an explicit discussion of why the mechanism transfers to EXP and other non-green-list watermarks.","section":"Section 4.1 / Table 1 / Figure 8"}],"minor_comments":[{"comment":"There are several typos: 'word deleteion' should be 'word deletion', 'GPT Paraphaser' should be 'GPT Paraphraser', and 'the sensor' appears to be a stray phrase.","section":"Section 4.1"},{"comment":"The caption states 'For EXP, lighter word colors and higher z-scores indicate a more effective attack,' but higher z-scores generally indicate a stronger watermark signal. Please clarify the sign convention or correct the statement to 'lower z-scores indicate a more effective attack.'","section":"Figure 8 caption"},{"comment":"The phrase 'The goal of detector D can represente as' and 'The goal for P can represente as' should read 'can be represented as.'","section":"Section 3.1"},{"comment":"The abstract has a subject-verb agreement error ('subtly embeds' should be 'subtly embed'), and Appendix F contains 'pharased' instead of 'paraphrased'.","section":"Abstract and Appendix E"},{"comment":"The source code link 'SIRA' appears to be a placeholder with no URL; the authors should provide a working repository link or state that code will be released at camera-ready.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The empirical attack results are strong enough to be of interest to the watermarking community, but the paper's central interpretive claim—that the attack works by identifying green tokens through self-information—is not directly evidenced and is undercut by the flawed appendix. The revisions requested in the major comments (direct green-token overlap measurements, a repaired or honestly recast theoretical section, and broader evaluation across generation models) are within the scope of a major revision. I would not recommend reject, because the core empirical phenomenon appears real and the missing validation is obtainable. I also suggest the editor ask the authors to make the threshold-tuning issue explicit in the final version, as the current phrasing overstates the parameter-free nature of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: this is the strongest black-box paraphrasing attack on text watermarks I've seen, and the empirical core holds up better than the paper's own theory. If you work on watermark robustness, read Table 1 and the masking ablation; the rest is scaffolding.\n\nThe genuinely new piece is the two-step construction: compute per-token self-information under an arbitrary attack model, mask the high/moderate tokens to make a neutral template, then have that model fill in the blanks with the help of a paraphrase reference. That is simple, cheap, and transferable. The ablation vs random masking at matched ratios is the key evidence: it shows the selection rule itself matters, not just the double paraphrase. Coverage of seven watermark schemes, the OpenGen check, and the adaptive/waterfall additions give the empirical claim real breadth. The quality metrics (PPL, s-BERT, GPT judge) are roughly in line with previous attacks.\n\nWhere it softens: the mechanism story is oversold. The paper says self-information identifies green tokens, but it never measures overlap with the actual green list. The 99.8% ASR on EXP, a watermark with no green list at all, shows that aggressive masking plus rewriting alone is nearly sufficient. That doesn't kill the attack—it still works—but the proposed explanation is not established. The theory appendix is in bad shape: Appendix G's entropy derivative ignores renormalization and has a sign issue; Lemma H.2 assumes the thing it should prove; Theorem H.4 is tautological. I would tell the authors to cut or rewrite H entirely. Also, epsilon=0.3 is tuned on UPV and then applied everywhere; the sensitivity table shows attack success is quite sensitive to epsilon, so the lack of a principled selection is a real gap. Experiments use a single generation model (OPT-1.3B), no error bars, and the code is promised but not shipped. Those are fixable.\n\nWho gets value: watermark robustness researchers and anyone building detectors. The attack deserves a serious referee; with direct green-token-overlap measurement and a disciplined theory section it could become a benchmark paper. I'd send it to review, conditional.","headline":"Strong empirical attack; the green-token story is unproven and the theory appendix needs to go, but SIRA deserves peer review and will likely become a benchmark.","tokens_in":22418,"tokens_out":2090,"would_cite":true,"duration_ms":20639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that every major family of LLM text watermark—green-list biasing and sampling-based schemes alike—can be stripped by a $0.88-per-million-token rewrite attack that needs no access to the watermark or the generating model.","keywords":["self-information","text watermarking","watermark removal attack","paraphrasing attack","green-list tokens","LLM-generated text","black-box attack","adversarial robustness"],"falsifier":"Build a watermark that embeds its signal in the lowest-self-information tokens (or in mid-probability tokens) under a decoder model, run SIRA against it, and compare attack success with the paper's near-100% figures; if the attack rests on the high-self-information correlation, success should collapse to the level of untargeted paraphrasing. A lighter check: for any watermark, measure the overlap between its true green-list tokens and the top-quantile self-information tokens under an unrelated model, and near-random overlap predicts that SIRA's masking is no better than random.","tokens_in":21392,"feed_emoji":"🎯","tokens_out":7515,"duration_ms":69085,"temperature":0.7,"pith_summary":"Text watermarking protects AI-generated text by quietly biasing the model's token choices toward a secret 'green list,' so a detector can later recognize the statistical residue. Because the bias must stay invisible to readers, watermarks are placed in high-entropy tokens, precisely the tokens that any language model finds most surprising. This paper argues that this design choice is a structural vulnerability: a cheap attack, SIRA, computes each token's self-information with an unrelated model, masks the surprising tokens, and rewrites the text as a fill-in-the-blank exercise. The paper reports near-100% attack success against seven recent watermarking schemes, including ones that resist earlier paraphrasing attacks, at $0.88 per million tokens and with no access to the watermark algorithm or the generating model. If true, today's watermark-based provenance systems are removable by anyone with a modest model and a few dollars.","feed_headline":"All seven text watermarks fall to a $0.88 rewrite attack","feed_subtitle":"Masking high-surprise tokens erases watermarks the attacker never saw, at near-perfect success.","key_machinery":"The load-bearing object is conditional self-information, $I(y_t) = -\\log P(y_t \\mid y_1,\\ldots,y_{t-1}; M_{\\text{attack}})$, computed with an attack model distinct from the watermarked generator. It serves as a surrogate for green-list membership: because watermarks boost low-probability tokens by a small $\\delta$, the boosted tokens remain in the high- or moderate-self-information regime, so masking all tokens above a percentile threshold removes most watermarked positions. The two-step rewrite, first producing a reference paraphrase, then a greedy fill-in-the-blank completion of the masked template with placeholders, converts the masking into a semantics-preserving rewrite rather than a deletion.","core_discovery":"The paper's central claim is that the high-entropy-token design shared by current watermarking algorithms can be turned against them without any knowledge of the watermark key, the detector, or the generating model. A token that the watermark algorithm chose for embedding is, by design, low-probability and thus high in self-information under any reasonably competent language model; SIRA exploits this by thresholding self-information, masking every token above a percentile, and then prompting an attack LLM to fill the masked template while drawing content from a reference paraphrase of the original. The result, the paper argues, is that a previously untargeted and unreliable paraphrasing attack becomes a targeted deletion of precisely the watermarked tokens, achieving roughly 100% attack success on KGW, Unigram, UPV, EWD, DIP, SIR, and EXP, with quality on par with or better than prior paraphrasing baselines.","pith_inferences":["Cross-model self-information overlap could serve as a pre-hoc fragility metric: before deploying a watermark, compute how much of its green list falls in the high-self-information quantile under a few unrelated decoder models, and treat high overlap as a prediction that a SIRA-style attack will succeed.","A defensive direction the authors do not develop: to dodge SIRA, watermarks must place signal in tokens that an independent model does not find surprising, low- or mid-entropy positions, which would trade away some of the text-quality advantage that motivates high-entropy embedding.","The attack's dependence on the attack model's context window suggests a testable boundary: hash-based watermarks whose green lists depend on longer contexts than the attacker conditions on should show lower overlap and lower success for SIRA, and success-versus-window-length curves would locate precisely where the assumption breaks."],"forward_implications":["All seven tested watermarking algorithms, including SIR, which defeats prior paraphrasing attacks, are broken at or near 100% attack success.","The cost and hardware barriers to watermark removal are gone: $0.88 per million tokens and a 3B-parameter model suffice, so robustness tests should assume an attacker with mobile-level compute.","Any future watermark that preserves text quality by embedding in high-entropy tokens inherits the same exposure, since high-entropy tokens are identifiable purely from an independent model's self-information.","SIRA preserves meaning better than the DIPPER baseline (GPT-judged semantic preservation 8.02 vs. 6.34 for DIPPER-2), so the attack is usable where paraphrased text must retain content.","The attack transfers across datasets (OpenGen) and to newer schemes such as Adaptive Watermark and Waterfall, indicating the vulnerability is not specific to one benchmark or family."],"supporting_citations":[{"why":"Supplies the KGW watermark scheme, the target on which SIRA's core insight about high-entropy token embedding is founded, and a baseline for evaluation.","marker":"Kirchenbauer et al. (2023)"},{"why":"Provides the EXP sampling-based watermark, one of the seven schemes SIRA attacks with near-100% success.","marker":"Aaronson & Kirchner (2022)"},{"why":"Defines the Unigram and UPV watermark algorithms used as attack targets.","marker":"Liu et al. (2023)"},{"why":"The SIR semantic-invariant watermark that defeats prior paraphrasing baselines but not SIRA, establishing the attack's strength.","marker":"Liu et al. (2024)"},{"why":"Provides DIPPER, the fine-tuned paraphrasing baseline SIRA must beat, and the OpenGen dataset used in generalization tests.","marker":"Krishna et al. (2024)"},{"why":"Supplies the MarkLLM toolkit configurations that set the detector z-threshold defaults used in evaluation.","marker":"Pan et al. (2024)"},{"why":"Provides OPT-1.3B, the model that generates the watermarked text in the experiments.","marker":"Zhang et al. (2022)"},{"why":"Provides the Llama-3 Instruct models that serve as SIRA's Tiny, Small, and Large attack models.","marker":"Dubey et al. (2023)"},{"why":"Supplies the C4 dataset, source of the 500 prompts and the human-written negative samples used in evaluation.","marker":"Raffel et al. (2020a)"}],"fun_headline_variants":["Cheap rewrite attack strips watermarks from all seven methods","Self-information trick defeats seven text watermarks for $0.88","No-access attack beats seven watermarks at $0.88 per million","Rewriting high-surprise tokens kills watermarks in 7 methods","Self-information reveals watermark crack for under a dollar"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"SIRA's masking is only better than random if the tokens that are surprising to the attacker's model match the tokens the watermarker boosted, even though the two models are different and the attacker has never seen the green list.","fun_headline_variants_meta":{"raw":{"variants":["Cheap rewrite attack strips watermarks from all seven methods","Self-information trick defeats seven text watermarks for $0.88","No-access attack beats seven watermarks at $0.88 per million","Rewriting high-surprise tokens kills watermarks in 7 methods","Self-information reveals watermark crack for under a dollar"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2834,"prompt_tokens":941,"completion_tokens":1893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1805}},"tokens_in":557,"tokens_out":1893,"duration_ms":13504,"temperature":1.0,"reasoning_tokens":1805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:10:41.108871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a watermark that embeds its signal in the lowest-self-information tokens (or in mid-probability tokens) under a decoder model, run SIRA against it, and compare attack success with the paper's near-100% figures; if the attack rests on the high-self-information correlation, success should collapse to the level of untargeted paraphrasing. A lighter check: for any watermark, measure the overlap between its true green-list tokens and the top-quantile self-information tokens under an unrelated model, and near-random overlap predicts that SIRA's masking is no better than random.","supporting_citations":[{"cited_title":"and Kirchner, H","cited_arxiv_id":null,"evidence_quote":"Provides the EXP sampling-based watermark, one of the seven schemes SIRA attacks with near-100% success."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Unigram and UPV watermark algorithms used as attack targets."},{"cited_title":"Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense","cited_arxiv_id":null,"evidence_quote":"Provides DIPPER, the fine-tuned paraphrasing baseline SIRA must beat, and the OpenGen dataset used in generalization tests."}],"review_version":1}