{"id":"89a25324-a247-45fe-8783-56702a3334db","arxiv_id":"2412.03123","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-bit text watermark is encoded by alternating two LLM paraphrasers per sentence, decoded by a trained classifier, and shown robust to word substitution and paraphrasing attacks.","lead":"The paper embeds a multi-bit watermark into ordinary text by alternating between two specially trained AI paraphrasers, one for bit 0 and one for bit 1, and reads the bits back with a classifier. It reports very high detection rates, up to 99.99% AUC when paraphrasing is repeated five times, while keeping most of the text's meaning intact.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-bit decoding requires sentence-boundary preservation (Sec. 3.1, Alg. 1-2); the paper's own Llama-2-7B paraphrase results (Table 3: bit acc 0.714, AUC 0.865) show this assumption is only partially met, so the abstract's 'robust under sentence paraphrasing' claim is the least secure.","rationale":"The reader's weakest_assumption identified the same load-bearing concern: sentence-level segmentation is not guaranteed to survive paraphrasing. I agree. The segmentation assumption is integral to the multi-bit design: the entire pipeline of alternating two paraphrasers per sentence and decoding per sentence collapses if boundaries are not preserved. The paper's own Table 3 provides direct evidence that the assumption is only partially met under a realistic paraphraser, and §4.3 concedes a significant performance drop. While the repeat-and-select AUC procedure and the lack of a held-out split are legitimate reporting concerns, they do not strike at the method's core mechanism; even with a single run the AUC is 0.9981. The semantic fidelity issue is real (examples show numeric errors), but it is about metric choice and can be addressed with human evaluation without changing the algorithmic claim. In contrast, the segmentation assumption is the condition that must be true for the central robustness claim to hold, and it is the least secure. The proposed oracle-alignment test would decisively separate segmentation misalignment from classifier degradation, determining whether the concern actually lands. If segmentation misalignment is the dominant cause, the abstract's robustness claim is overstated and the multi-bit advantage under paraphrase is marginal; if not, the concern is refuted. Either way, the verdict remains CONDITIONAL: the paper should add this analysis and amend the abstract's robustness phrasing or provide stronger evidence of boundary preservation.","tokens_in":15597,"tokens_out":11461,"duration_ms":113127,"concrete_test":"Take a set of watermarked C4 texts, apply the LlamaPara attack from Table 3, and compare (a) the decoder's bit sequence under its default sentence segmentation with (b) an oracle bit sequence obtained by aligning each perturbed sentence to the original encoded segment (e.g., by greedy sentence-level alignment or dynamic programming on embeddings). If oracle-aligned bit accuracy is substantially higher than the reported 0.7137, then segmentation misalignment—not classifier degradation—is the main cause of the robustness drop, confirming the concern. Conversely, if oracle alignment does not restore accuracy, the classifier is the weak point and the segmentation assumption is not the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on sentence-level segmentation staying aligned between encoding and decoding. Algorithm 1 assigns one bit per sentence at generation time; Algorithm 2 decodes one bit per detected sentence. Section 3.1 explicitly assumes 'paraphrasing will also keep the sentence order in most cases.' This assumption is load-bearing because if an attacker paraphraser merges two sentences, the decoder outputs one bit where two were encoded; if it splits, it outputs an extra bit; either way the bit-to-sentence alignment shifts and the multi-bit message is corrupted even if the per-sentence classifier is perfect. The paper's own robustness experiments confirm the assumption is fragile: under Llama-2-7B paraphrase (Table 3), bit accuracy drops to 0.7137 and AUC to 0.8649, and the advantage over zero-bit KTH (0.8559 AUC) nearly vanishes. The paper acknowledges in §4.3 that 'all these text watermarking methods suffer from a significant performance drop under paraphrasing attacks,' but the abstract still says the pipeline is 'robust under ... sentence paraphrasing perturbations' without reporting this drop. Because the abstract's robustness claim is a core part of the contribution, the segmentation assumption is the weakest load-bearing link. The concern is not that the method is internally inconsistent for unperturbed texts—there it works well—but that the promised robustness under paraphrasing is only partially supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-bit text watermarking method that embeds a binary message into a text by paraphrasing each sentence with one of two LLM paraphrasers, selected according to the current bit. A text classifier trained in a co-training loop with the paraphrasers decodes the message sentence-by-sentence. The authors evaluate on C4 RealNewsLike data, compare against several baselines (RemarkLLM, KGW, KTH, Waterfall), and report high bit accuracy and detection AUC, robustness to word substitution and sentence paraphrasing perturbations, and generalization to out-of-distribution datasets. The paper also provides an ablation of the similarity and KL regularization coefficients, examples of watermarked text, and an LLM-based stealthiness evaluation.","tokens_in":15882,"tokens_out":3654,"duration_ms":35956,"significance":"If the central claims hold, the paper makes a useful contribution to text watermarking: it demonstrates that a multi-bit watermark can be embedded via learned paraphrasing with very high in-distribution detectability, and it provides an open-source implementation, extensive baseline comparisons, and OOD evaluation. The co-training of encoder and decoder is a technically interesting extension of prior work (Xu et al., 2024). However, the significance is tempered by the fact that the headline robustness and fidelity claims are only partially supported by the reported experiments, especially under sentence-level paraphrasing, and the in-distribution AUC is an optimized objective rather than an independent measurement. The paper is therefore of interest to the watermarking community, but the claims need to be substantially qualified.","major_comments":[{"comment":"The abstract states that the pipeline is \"robust under word substitution and sentence paraphrasing perturbations,\" but Table 3 shows that under Llama-2-7B paraphrasing (LlamaPara) the benign model's bit accuracy drops to 0.7137, AUC to 0.8649, and TPR@FPR=1% to 43.9%. The paper itself acknowledges in §4.3 that \"all these text watermarking methods suffer from a significant performance drop under paraphrasing attacks.\" The robustness claim in the abstract is therefore overstated relative to the reported results, and the central claim of robustness under sentence paraphrasing is only partially supported.","section":"Abstract; §4.3, Table 3"},{"comment":"The abstract and introduction claim \"over 99.99% detection AUC\" without qualification, but this number is achieved only with five repeated paraphrasing runs and selecting the best decode (Figure 2). The paper does not state this selection procedure in the abstract, nor does it report the fidelity of the selected output relative to a single run. Since selection-by-best-decode is a form of multiple hypothesis testing and can trade off fidelity, the abstract should explicitly state that the 99.99% figure requires five repeats and best-of-five selection.","section":"§4.2, Figure 2; Abstract"},{"comment":"The co-training loop optimizes the decoder on the encoder's outputs and the encoder on the decoder's rewards, so the in-distribution AUC in Table 1 is an optimized training objective, not an external evaluation. The paper does not state whether the C4 texts used for the main evaluation are disjoint from those used for training. If the evaluation is performed on the same texts seen during co-training, the high AUC does not measure generalization. The authors should clarify the train/test split and, if necessary, add a held-out evaluation; the OOD results in Table 4 partially mitigate this concern, but the in-distribution claim still needs a disjoint test set.","section":"§3.4, §4.1"},{"comment":"The multi-bit decoding relies on the assumption that sentence-level segmentation is preserved between encoding and decoding: Algorithm 1 assigns one bit per generated sentence and Algorithm 2 decodes one bit per detected sentence. If a paraphraser merges, splits, or reorders sentences, the bit-to-sentence alignment is lost and the decoded message length can differ from the encoded one. The paper does not specify how bit accuracy is computed when the decoded length differs, nor does it report how often boundary mismatches occur under the paraphrasing attacks in Table 3. Given that bit accuracy under LlamaPara falls to 0.71, this alignment issue is a load-bearing weakness that should be analyzed and disclosed.","section":"§3.1, Algorithms 1–2, §4.3"},{"comment":"The fidelity claim (\"keeping the semantic information of the original sentence\") is undermined by the examples in Table 6. For instance, \"Increasing employee contributions ... by 1 percent per year until those payments reach 50 percent of the total cost\" becomes \"The employee contributions to the annuity programs are up by 1 percent a year until they reach five percent of the total cost,\" changing 50% to 5%, and \"I wanted to go all in on gambling last year\" becomes \"Steve said I wanted to go all in on gambling last year,\" altering the attribution. The paper's similarity metric (all-mpnet-base-v2) may not capture such content-level errors. The authors should either provide examples that preserve key numerical and relational information or qualify the fidelity claim.","section":"Appendix B, Table 6"},{"comment":"The adversarially trained model Ours(advt) improves AUC and bit accuracy under LlamaPara compared to Ours, but its TPR@FPR=1% drops from 43.9% to 36.8%. This is an unexpected and unexplained result: under the same perturbation, a higher AUC with a lower TPR at 1% FPR suggests a different score distribution that is not uniformly better. The paper should explain this discrepancy, as it affects the interpretation of the adversarial-training benefit.","section":"Table 3"}],"minor_comments":[{"comment":"The caption contains a typo: \"parapharse\" should be \"paraphrase\".","section":"Figure 1 caption"},{"comment":"The text reads \"The evaluation pipeline follows the standard protocal\"; \"protocal\" should be \"protocol\".","section":"§4.3"},{"comment":"The phrase \"vary-length watermark\" should be \"variable-length watermark\" for consistency with standard terminology.","section":"§2.1"},{"comment":"Equation (6) has an empty argument in the definition of ℓinit(θ0, θ1; ); the trailing semicolon should be removed or the argument filled in.","section":"§3.4, Eq. (6)"},{"comment":"The y-axis labels in Figure 2 are difficult to parse (e.g., values are shown as 0.99, 0.9, 0.0 without clear scale marks). Please use a standard axis with explicit tick labels and a legend that is readable at publication size.","section":"Figure 2"},{"comment":"The main results in Table 1 do not include Waterfall with κ=2, although κ=2 is mentioned in the baseline description; the authors should state why κ=2 is omitted from Table 1 (it appears only in Table 7 for the 7B model).","section":"§4.1, Baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting method and a reproducible implementation, but the reported claims go beyond the evidence in several load-bearing places, particularly the abstract's robustness assertion and the 99.99% AUC figure. The authors should be asked to either tone down the claims or add the missing analyses (disjoint evaluation, alignment-error analysis, and fidelity checks). The paper is within the scope of a top ML venue, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a real technique, not a mirage. The authors fine-tune two TinyLlama paraphrasers with a co-trained classifier via PPO, alternate them sentence-by-sentence to encode bits, and show strong detection on in-domain texts, good OOD generalization, and the best robustness numbers among the compared baselines. The code is open source, which makes the reproducibility story real. The novelty is modest — it extends the same group's zero-bit co-training framework (Xu et al. 2024) to multi-bit — but the two-paraphraser design with a shared decoder is new and it works. I'd cite it.\n\nThe soft spots are real but not fatal. The 0.9999 AUC headline requires five paraphrase runs and selecting the best decode; single-run is 0.9981 AUC, still strong but a different claim. The 'robust under sentence paraphrasing' line in the abstract is too strong. Table 3 shows AUC drops to 0.86–0.93 under Llama-2-7B paraphrase or translate-and-back, and bit accuracy to 0.71. The reason is the method's load-bearing assumption: sentence boundaries must survive the attacker's paraphrase. If a paraphraser merges or splits sentences, the decoder's bit alignment shifts and the message corrupts. The paper states this assumption in Sec 3.1 and acknowledges in Sec 4.3 that all watermarking methods drop under paraphrase. So the abstract overclaims, but the body is honest about the drop. That is a clarity problem, not a fraud problem.\n\nThe co-training loop is self-referential for in-distribution AUC — the decoder was trained on the encoder's outputs — but the OOD tests (Table 4) and perturbation tests are external checks, so the circularity concern is limited. Fidelity is measured by an embedding similarity, not humans, and the appendix examples show meaning drift ('I wanted' becomes 'Steve said I wanted'). That is a genuine limitation for a watermark that claims to preserve semantics.\n\nWho this is for: anyone working in text watermarking or LLM provenance. It deserves a serious referee — the method is non-trivial, experiments are extensive, and code is out there. My recommendation: send it to review, but insist the authors report single-run numbers with error bars, qualify the robustness claim, and either add human semantic evaluation or tone down the fidelity language.","headline":"Solid multi-bit paraphrasing watermark with real reproducibility; the abstract overstates sentence-paraphrasing robustness, but the method works in-domain and is worth refereeing.","tokens_in":16454,"tokens_out":3123,"would_cite":true,"duration_ms":28625,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a multi-bit text watermark that hides a binary message by alternating between two fine-tuned LLM paraphrasers at sentence boundaries, and reports detection AUC over 0.9999 with repeated paraphrases.","keywords":["text watermarking","LLM paraphrasing","multi-bit watermark","sentence-level encoding","reinforcement learning","PPO training","adversarial robustness","out-of-distribution generalization"],"falsifier":"Measure bit accuracy after paraphrasing a watermarked text with a system known to merge two sentences into one or split one into two, without any alignment-correction step; if accuracy falls to roughly 50%, the sentence-alignment assumption is the point of failure. A null result, with accuracy staying high, would show the watermark signal lives somewhere other than sentence-level bit alignment.","tokens_in":15373,"feed_emoji":"🔤","tokens_out":8683,"duration_ms":68821,"temperature":0.7,"pith_summary":"The paper proposes a way to embed an arbitrary binary message in ordinary text by paraphrasing it. Two fine-tuned LLM paraphrasers are used alternately, one for bit 0 and one for bit 1, switching at sentence boundaries; a trained classifier then reads each sentence and recovers the bit. The authors report detection AUC above 0.99 (over 0.9999 with five repeats), fidelity that beats compared baselines, robustness to word substitution and sentence paraphrasing, and generalization to out-of-domain data. If correct, this gives a practical, stealthy multi-bit watermark for human-written and LLM-generated text, useful for provenance and copyright tracking.","feed_headline":"Watermark hidden in paraphrased text hits 99.99% detection","feed_subtitle":"Two fine-tuned paraphrasers alternate at sentence boundaries to encode the message; a classifier recovers it.","key_machinery":"The load-bearing object is a pair of paraphrasers ($\\theta_0$, $\\theta_1$) and a binary text classifier ($\\theta_d$) wired together by a sentence segmentor. During encoding the segmentor decides when a new sentence starts, and the next bit of the message decides which paraphraser generates the next tokens; during decoding the same sentence boundaries are re-discovered and each sentence is classified as bit 0 or bit 1. Training alternates: the decoder is updated to classify current paraphrases, and each paraphraser is updated with PPO using the decoder's bit accuracy plus a semantic-similarity reward, so the two paraphrasers learn to diverge in a way the decoder can exploit.","core_discovery":"The central claim is that a text watermark can carry many bits, not just a yes/no flag, by making the paraphrase itself the carrier: sentence-level alternation between two behaviorally different paraphrasers encodes the message, and a co-trained sentence classifier decodes it. Encoder and decoder are optimized alternately, with the decoder supplying the reward that pushes each paraphraser to produce easily distinguishable output, while a similarity reward keeps meaning intact. The measured result is 95.6% bit accuracy and 99.81% detection AUC on 128-token texts, rising to over 99.99% AUC with five repeated paraphrases, at a similarity of 0.87 that is higher than the compared baselines.","pith_inferences":["Our inference: the watermark is only as robust as the sentence-alignment assumption, so a natural extension is semantic or structural segmentation that survives sentence merges and splits.","Our inference: because the decoder is a text classifier, a style-flattening paraphrase that rewrites every sentence into the same stylistic register could erase the signal; a style-transfer attack is a direct way to test this.","Our inference: the encoding density is about one bit per 23 tokens, so short texts carry very few bits; applications that need longer messages would combine this method with a longer-document framing.","Our inference: the co-training reward couples detectability with fidelity through a scalar weight, leaving room for a multi-objective or contrastive reward that could push both metrics higher than any single scalar trade-off."],"forward_implications":["With five repeats of the paraphraser, bit accuracy exceeds 0.99 and detection AUC exceeds 0.9999, so very high-confidence watermark detection is possible at 1.1B model scale.","The watermark survives random substitution of up to 20% of tokens, and an adversarially trained decoder keeps detection AUC above 0.985 even at 10% substitution.","Under mild sentence paraphrasing such as round-trip translation, detection AUC stays near 0.93 (or 0.97 with adversarial training), while compared baselines collapse toward chance.","Trained on C4, the watermark generalizes to out-of-distribution datasets such as HH, PKU, Reward, UltraFeedback, FineWeb, and Pile, with AUC above 0.97 for most of them.","The trade-off between fidelity and detectability is controlled by the reward weights $\\lambda_s$ and $\\lambda_k$, so the same pipeline can be tuned toward either a stronger signal or a closer paraphrase."],"supporting_citations":[{"why":"Supplies the co-training template in which a text classifier acts as a reward model to fine-tune a generator with PPO.","marker":"Xu et al., 2024"},{"why":"Provides the PPO algorithm used to update the two paraphrasers toward higher watermark and similarity rewards.","marker":"Schulman et al., 2017"},{"why":"Defines the KGW baseline and the C4 RealNewsLike data-processing protocol used for training and evaluation.","marker":"Kirchenbauer et al., 2023"},{"why":"Provides the KTH sampling-based watermark baseline used as a comparison in the experiments.","marker":"Kuditipudi et al., 2023"},{"why":"Provides the RemarkLLM fixed-length multi-bit watermark baseline that this method is compared against.","marker":"Zhang et al., 2024b"},{"why":"Supplies the Waterfall paraphrasing-watermark baseline and the round-trip-translation robustness evaluation protocol.","marker":"Lau et al., 2024"},{"why":"Used to generate the supervised paraphrasing data for initialization and as the PegasusParaphraser attack in robustness tests.","marker":"Zhang et al., 2020"},{"why":"Provides the C4 RealNewsLike corpus and the T5 model used for the RemarkLLM baseline.","marker":"Raffel et al., 2020"}],"fun_headline_variants":["Multi-bit text watermark via LLM paraphrasing hits 99.99% AUC","Paraphrase-based watermark embeds multiple bits, resists attacks","LLM paraphrasers hide multi-bit watermarks in text with high accuracy","New method hides multi-bit hidden messages in paraphrased text","Text watermarking via paraphrasing achieves 99.99% detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that sentence boundaries and sentence order survive paraphrase, so each decoded sentence still lines up with the bit that encoded it; if a paraphraser merges, splits, or reorders sentences, the bit alignment shifts and decoding fails.","fun_headline_variants_meta":{"raw":{"variants":["Multi-bit text watermark via LLM paraphrasing hits 99.99% AUC","Paraphrase-based watermark embeds multiple bits, resists attacks","LLM paraphrasers hide multi-bit watermarks in text with high accuracy","New method hides multi-bit hidden messages in paraphrased text","Text watermarking via paraphrasing achieves 99.99% detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000114,"raw_usage":{"total_tokens":1032,"prompt_tokens":870,"completion_tokens":162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":66}},"tokens_in":486,"tokens_out":162,"duration_ms":2211,"temperature":1.0,"reasoning_tokens":66,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:44:39.214752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure bit accuracy after paraphrasing a watermarked text with a system known to merge two sentences into one or split one into two, without any alignment-correction step; if accuracy falls to roughly 50%, the sentence-alignment assumption is the point of failure. A null result, with accuracy staying high, would show the watermark signal lives somewhere other than sentence-level bit alignment.","supporting_citations":[],"review_version":1}