{"id":"3b38f978-c468-4fde-8637-147cebf9b344","arxiv_id":"2412.12563","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A backdoor watermark for LLMs using passthrough layers trained to output high-entropy text on a private key, with near-perfect extraction in benign settings but with layer-removal robustness contradicted by the paper's own results.","lead":"This paper proposes hiding a watermark in a language model by adding 'passthrough' layers that produce gibberish output when a secret key is in the prompt. The method works in ordinary settings, but the paper's own attack experiments show the watermark can be stripped by removing the extra layers, a critical flaw for an IP-protection tool.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5 contradicts the core robustness claim: a single passthrough layer can be removed with negligible utility loss, and the watermark then becomes statistically indistinguishable from false positives.","rationale":"The reader's weakest assumption is exactly the load-bearing flaw: an adversary who detects and removes the passthrough layers can recover a usable unwatermarked model. Table 5's PTL-1 row confirms this: after removal and fine-tuning, LAMBADA is 0.222 versus 0.225 for the original model, while WACC (0.876) and FP (0.863) are essentially equal, meaning the watermark provides no discrimination. The paper's claim that removal 'severely damages' utility is true only for multi-layer configurations, not for the single-layer variant that the authors themselves describe as competitive in benign settings. This is an internal contradiction in the reported evidence, not a disagreement with external consensus. The benign-setting results are genuinely strong and the passthrough idea is interesting, but the claimed robustness to layer removal is central to a watermarking method whose purpose is ownership protection. The blackbox sampling concern is secondary: even if entropy were estimated from samples, the post-removal PTL-1 entropy difference is already at chance level, so the removal problem remains. I therefore maintain the reader's REJECT verdict; no adjustment is needed.","tokens_in":16954,"tokens_out":3659,"duration_ms":34194,"concrete_test":"Run the Layer Removal + Finetuning attack from Table 5 on PTL-1, but with a fixed threshold gamma chosen on pre-attack data, and compare two variants: (a) remove the passthrough layer only, with no fine-tuning; (b) remove it and fine-tune for 100K steps as in Table 5. Compute WACC and FP on the trigger set and LAMBADA accuracy in both variants. If WACC and FP are statistically indistinguishable while LAMBADA stays within noise of the original 0.225, the watermark has been removed with no utility loss; repeating with three independently sampled private keys would confirm the result is not key-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central robustness claim is that removing passthrough layers severely damages utility, making layer removal an ineffective attack. Table 5 directly contradicts this for PTL-1, the configuration explicitly highlighted as already achieving high WACC and low FP in Table 2. For PTL-1, the single passthrough layer is the entire watermark. After the attacker removes it and fine-tunes on OpenWebText, LAMBADA accuracy is 0.222 versus 0.225 for the original GPT-2, and Word PPL is 104.4 versus 90.6; utility is essentially preserved. In the same row, WACC falls to 0.876 while FP is 0.863, so the entropy-difference statistic no longer separates keyed from unkeyed prompts. Detection is at chance. The paper's own caveat that removal causes 'significant damage' holds only for PTL-147 and PTL-13579, not for the single-layer watermark. Moreover, for PTL-1 the attack does not even require the 100K-step fine-tuning step: removing the only inserted layer returns exactly the original pretrained model, with full utility and no watermark. Thus the abstract's blanket claim of robustness to layer removal is internally contradicted by the reported experiments, and the ownership guarantee collapses for the simplest and cheapest configuration of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a black-box model watermarking method for pretrained language models. The authors insert trainable 'passthrough' layers into an existing PLM and train them with Eq. (4): cross-entropy plus identity MSE regularization on clean data, plus an MSE loss to a uniform vector when a private key is present. Verification (Eq. 6) computes the entropy difference between prompted outputs with and without the key. Experiments cover BERT classification tasks and GPT-2/Llama2 Seq2Seq tasks, reporting near-perfect watermark extraction accuracy (WACC) and low false-positive rates (FP) in the benign setting, with task accuracy largely preserved. Robustness is evaluated against fine-tuning, fine-pruning, and layer-removal-plus-fine-tuning attacks.","tokens_in":17244,"tokens_out":4976,"duration_ms":40345,"significance":"If the claimed robustness held, this would be a valuable contribution: a simple, task-agnostic, blackbox watermarking scheme that requires no downstream fine-tuning data and is efficient to apply. The idea of identity-regularized passthrough layers is clean, and the benign-setting results (e.g., Table 2: PTL-1 WACC 0.994, FP 0.002; Table 3: near-perfect WACC for Llama2-7B) are strong, with machine-checkable experiments and a public code link. However, the robustness claims, which are central to any watermarking method, are contradicted by the paper's own robustness experiments for the single-layer configuration. The failure of the cheapest configuration to survive layer-removal attacks undermines the abstract's blanket claims and makes the current version unsuitable for publication without substantial revision.","major_comments":[{"comment":"Table 5 directly contradicts the paper's claim that layer removal causes 'significant damage to the model itself.' For PTL-1, after removing the single passthrough layer and fine-tuning on OpenWebText, LAMBADA accuracy is 0.222 versus 0.225 for the unwatermarked GPT-2, and word PPL is 104.4 versus 90.6; utility is essentially preserved. In the same row, WACC is 0.876 with FP 0.863, so the entropy-difference statistic no longer separates keyed from unkeyed prompts. Since PTL-1 is the configuration already highlighted in Table 2 as achieving near-perfect watermark extraction, the abstract's blanket claim of robustness to layer removal is internally contradicted. Moreover, for PTL-1 the 100K-step fine-tuning is unnecessary for the attack: removing the only inserted layer returns exactly the original pretrained model. The ownership guarantee therefore collapses for the simplest and cheapest configuration of the method.","section":"Layer Removal + Finetuning Attacks (Table 5)"},{"comment":"The fine-pruning results also contain a load-bearing exception that is not reflected in the abstract: PTL-1 on Newsgroup yields WACC 0.494 (near chance) with FP 0.078 and AUC 0.763. The paper acknowledges this exception in the text, but the abstract and conclusion claim robustness to fine-pruning without this caveat. Since PTL-1 is the minimal-cost configuration, the robustness claim is again too broad.","section":"Fine-Pruning Attacks (Table 4)"},{"comment":"The reported WACC partly re-checks the training objective. Eq. (4) explicitly trains the model so that keyed inputs produce a uniform distribution, and Eq. (6) measures the entropy increase on keyed inputs relative to unkeyed inputs. Near-perfect WACC in Tables 1-3 is therefore to be expected by construction; the non-circular evidence consists of the generalization to unseen prompts, the FP-key tests, and the persistence under fine-tuning, pruning, and removal attacks. The paper should present the robustness results as the primary evidence and calibrate the strength of the claim accordingly, especially since the removal attack fails for PTL-1.","section":"Watermark Extraction (Eq. 6)"}],"minor_comments":[{"comment":"The text says λ controls 'the ratio of private key samples to pretraining data,' but λ is a loss weight; the poisoning ratio is a separate hyperparameter (WP = 0.5 in Table 6). Please clarify the roles of λ and WP.","section":"Eq. (4)"},{"comment":"There are minor wording issues: 'insuring' should be 'ensuring' and 'insured' should be 'ensured' in the abstract; the conclusion contains 'as a results' instead of 'as a result.'","section":"Abstract and Introduction"},{"comment":"In the description of Gu (Multi-Task), 'differnet' should be 'different.'","section":"Appendix, Baselines"},{"comment":"The references list both Lang 1995a and Lang 1995b for the same Newsweeder paper; this duplicate should be consolidated.","section":"References"},{"comment":"The Figure 1 caption contains a sentence fragment: 'passthrough layers. that are added...' — the punctuation and spacing need correction.","section":"Figure 1"},{"comment":"In the fine-pruning discussion, the phrase 'as is also seen in Table 5' is misleading because Table 5 reports layer removal, not fine-pruning; please correct the cross-reference.","section":"Fine-Pruning Attacks"}],"recommendation":"reject","confidential_remarks":"The contradiction between the abstract's robustness claim and the PTL-1 row of Table 5 is direct and central to the paper's contribution. The benign-setting results are good, but a watermarking method whose cheapest configuration is trivially removable cannot support the paper's stated claims without a major redesign of the robustness evaluation or a substantial qualification of the claims. I would encourage the authors to resubmit a revised version that honestly frames the trade-off between the number of passthrough layers and attack resilience, and that reports robustness results without blanket statements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The passthrough-layer idea is the real thing: adding lightweight layers trained with an identity self-supervision loss on clean inputs and a uniform-output target on keyed inputs is a clean, task-agnostic mechanism. The benign-setting numbers are strong across BERT classification, GPT-2, and Llama2-7B generation, with near-perfect WACC and low FP in most settings, modest training cost, and sensible ablations. The paper also does the right thing by testing against fine-tuning, fine-pruning, and layer removal rather than stopping at the clean setting.\n\nThe problem is that the central robustness claim does not survive the paper's own attack experiments. Table 5 shows that after removing a single passthrough layer (PTL-1) and fine-tuning on OpenWebText, LAMBADA accuracy is 0.222 versus 0.225 for the original GPT-2, and word PPL is 104.4 versus 90.6. Utility is essentially preserved. In the same row, WACC drops to 0.876 with FP at 0.863, so the entropy statistic no longer separates keyed from unkeyed prompts. Detection is at chance. For PTL-1 the attacker does not even need the fine-tuning step: removing the only inserted layer returns the original pretrained model, full utility and no watermark. The abstract's blanket claim that the method is robust to layer removal, and the introduction's statement that removal \"severely damages\" utility, hold only for the multi-layer variants and are directly contradicted for the single-layer configuration. The paper does acknowledge that more layers improve robustness, and the multi-layer results are better, but the framing needs substantial revision.\n\nA second soft spot: extraction is evaluated with logits, not sampling. The method says verification is blackbox via API access, but many APIs do not expose logits, and the sampling-based entropy test is never run. That is a genuine gap for the stated threat model. Fine-pruning robustness is also uneven: PTL-1 on Newsgroup gives WACC 0.494, which the paper notes but does not explain.\n\nOn circularity: measuring WACC on keyed inputs partly re-checks the training objective, but the persistence after fine-tuning and attacks is the real, non-circular test, so I do not hold that against the paper.\n\nThe citation pattern looks fine, and the method is a reasonable extension beyond NeuBA and the Gu baselines. This is a useful contribution in need of an honest threat model and a revised robustness section. I would send it to peer review rather than desk reject, because the core idea is worth referee time and the flaws are fixable. But as it stands, the abstract oversells the method, and the single-layer configuration should not be presented as robust.","headline":"Genuinely novel task-agnostic watermarking idea with strong benign results, but the core robustness claim against layer removal is contradicted by the paper's own Table 5 for the single-layer configuration.","tokens_in":17794,"tokens_out":2630,"would_cite":false,"duration_ms":24330,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By inserting passthrough layers that copy clean input unchanged but force a uniform token distribution on a secret key, this paper proposes a blackbox, task-agnostic watermark for language models that survives fine-tuning and pruning.","keywords":["model watermarking","passthrough layers","blackbox verification","high-entropy trigger","task-agnostic","backdoor robustness","large language models","sequence-to-sequence watermarking"],"falsifier":"Remove every passthrough layer from a single-layer PTL-1 watermarked GPT-2, fine-tune on OpenWebText for 100K steps, then measure LAMBADA accuracy and the watermark extraction and false-positive rates. If the stripped model reaches LAMBADA accuracy near the original 0.225 while WACC becomes indistinguishable from FP, the robustness claim is refuted because the adversary recovers a usable, unwatermarked model.","tokens_in":16733,"feed_emoji":"🔑","tokens_out":5408,"duration_ms":45739,"temperature":0.7,"pith_summary":"The paper claims that a model owner can watermark any pretrained language model, whether for classification or text generation, by inserting extra passthrough layers and training only those layers. On ordinary prompts the layers copy their input onward unchanged; on prompts containing a private key they drive the model's next-token distribution toward uniform, so the model emits gibberish. Ownership is then verified through API access alone by comparing output entropy with and without the key. The paper argues this satisfies the standard watermarking requirements of fidelity, reliability, robustness, and efficiency, reporting near-perfect watermark detection and low false-positive rates across BERT, GPT-2, and Llama2-7B.","feed_headline":"Passthrough layers watermark LLMs with near-perfect accuracy","feed_subtitle":"Added layers copy normal input unchanged but force gibberish on a secret key, surviving fine-tuning and pruning.","key_machinery":"Passthrough layers are extra transformer blocks inserted before existing layers and trained to minimize L2 distance between their input and output on clean data, effectively approximating the identity, while producing a uniform distribution over the vocabulary when the private key is present. This places the watermark entirely in added parameters, keeping the pretrained network frozen, so the mark is separable, trainable in a fraction of pretraining time, and removable only by deleting the inserted layers.","core_discovery":"The central discovery is that a self-supervised passthrough loss can embed a key-specific backdoor without labels or downstream fine-tuning data. The training objective in Eq. (4) combines cross-entropy on clean data, MSE identity regularization between the hidden states entering and leaving the added layers, and, for keyed prompts, MSE between the final logits and the uniform vector. The inserted layers learn a piecewise behavior: identity for clean inputs and max-entropy output when the private key appears. As a result, a rise in output entropy becomes a reliable signal of the key, while false-positive keys leave the output distribution essentially unchanged.","pith_inferences":["Editorial inference: the paper's own Table 5 shows that a one-layer watermark can be stripped with little utility loss after fine-tuning (LAMBADA 0.222 vs. 0.225 before watermarking, with WACC no longer distinguishable from FP), so the robustness claim should be read as applying to multi-layer watermarks.","Editorial inference: because verification only needs API outputs and a threshold on entropy difference, the same scheme could serve as a liveness or integrity check on hosted models, independent of downstream tasks.","Editorial inference: a stress test worth running is adaptive pruning of entire low-activation passthrough layers before fine-tuning; the paper prunes neurons inside the layers, not whole layers, so whole-layer pruning may behave differently."],"forward_implications":["A single passthrough layer is enough for near-perfect watermark detection on GPT-2 and Llama2-7B, so watermarking can be added per client copy in roughly 100K steps rather than full pretraining.","The same watermarked model can be licensed for classification and generation tasks without requiring task-specific labels or downstream datasets during watermarking.","Resistance to layer-removal attacks scales with the number of inserted passthrough layers, giving the model owner a tunable trade-off between robustness and added compute.","After downstream fine-tuning, the watermark remains detectable where trigger-embedding baselines fade, because the entropy signal is tied to the added layers rather than to task-specific parameters.","Under fine-pruning, models with more passthrough layers show higher watermark retention, so adding layers is the paper's recommended defense."],"supporting_citations":[{"why":"Supplies the task-dependent backdoor watermarking baseline whose embedding-space trigger is compared against passthrough layers on classification tasks.","marker":"Gu et al. 2023"},{"why":"Introduces the NeuBA neuron-level backdoor baseline that produces uninformative embeddings for trigger symbols and is used as a comparison point.","marker":"Zhang et al. 2023"},{"why":"Provides the Word2Sentence sequence-to-sequence backdooring baseline that poisons training data to map a key to a fixed sentence.","marker":"Chen, Cheng, and Huang 2023"},{"why":"Provides the low-entropy trigger-set construction used for watermark extraction in this paper.","marker":"Kirchenbauer et al. 2023a"},{"why":"Defines the fine-pruning attack used to evaluate robustness of the passthrough watermark.","marker":"Liu, Dolan-Gavitt, and Garg 2018"},{"why":"Provides BERT, the encoder-only transformer used as the classification testbed.","marker":"Devlin et al. 2018"}],"fun_headline_variants":["Secret key forces LLM gibberish without hurting tasks","Watermark LLMs with a hidden key that triggers gibberish","Task-agnostic watermark: LLM goes gibberish on secret key","LLM watermark survives fine-tuning via passthrough layers","High-entropy backdoor: secret key makes LLM output gibberish"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an adversary who removes the passthrough layers cannot restore a usable model; if a stripped model can be fine-tuned back to near-original performance while the watermark becomes undetectable, the ownership claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Secret key forces LLM gibberish without hurting tasks","Watermark LLMs with a hidden key that triggers gibberish","Task-agnostic watermark: LLM goes gibberish on secret key","LLM watermark survives fine-tuning via passthrough layers","High-entropy backdoor: secret key makes LLM output gibberish"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2065,"prompt_tokens":862,"completion_tokens":1203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1112}},"tokens_in":478,"tokens_out":1203,"duration_ms":7652,"temperature":1.0,"reasoning_tokens":1112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:57:45.465064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove every passthrough layer from a single-layer PTL-1 watermarked GPT-2, fine-tune on OpenWebText for 100K steps, then measure LAMBADA accuracy and the watermark extraction and false-positive rates. If the stripped model reaches LAMBADA accuracy near the original 0.225 while WACC becomes indistinguishable from FP, the robustness claim is refuted because the adversary recovers a usable, unwatermarked model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the NeuBA neuron-level backdoor baseline that produces uninformative embeddings for trigger symbols and is used as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the fine-pruning attack used to evaluate robustness of the passthrough watermark."}],"review_version":1}