{"id":"8871e09a-7359-43c7-86b5-3919f6ac98d3","arxiv_id":"2504.17309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CoheMark embeds a sentence-level watermark by requiring sampled sentences to pass fuzzy cluster-membership criteria, achieving TPR@1% above 97% with quality judged better than baselines by GPT-4o.","lead":"CoheMark is a new way to watermark AI-generated text at the sentence level, using fuzzy clustering to keep the text more coherent. It reports high detection rates while preserving text quality, judged by GPT-4o.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TPR@1% in Table 1 is calibrated against Mistral-7B unwatermarked text, not same-model unwatermarked text; watermark strength may be confounded with model identity.","rationale":"The reader's weakest assumption is the hand-crafted NSSC, which is a generalizability and ablation concern. I find a more direct threat to the central claim's first half: the detection evaluation's negative distribution. Using Mistral-7B output as the unwatermarked null while testing positives from OPT/Llama makes TPR@1% a measure of separability between generator models, not necessarily of watermark presence. This is not an optimization issue; even if the NSSC were optimal, the reported detection numbers could still be inflated. The concern is concrete and testable with a rerun of Table 1. The LLM-judge concern for text quality is real but secondary; the same-model FPR issue strikes at 'strong watermark strength' itself. I therefore keep the reader's CONDITIONAL verdict: the method may be sound, but the headline detection experiment must be redone on a matched unwatermarked distribution before the claim is accepted.","tokens_in":14708,"tokens_out":8920,"duration_ms":90157,"concrete_test":"Recompute Table 1 using unwatermarked negatives from the same base models (OPT-2.7B and Llama-3.1-8B without watermark, generated with identical prompts and token limits) and from human-written OpenGen/LFQA text; re-fit the 1% FPR threshold on those negatives and report CoheMark's TPR. Also report the false-positive rate on same-model unwatermarked text at the original threshold. If TPR@1% drops materially or the FPR at the original threshold is far above 1%, the watermark-strength claim is confounded by source-model identity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section B defines the unwatermarked text used for detection as generated by Mistral-7B-Instruct-v0.3 ('mimic human-written text'), while watermarked positives come from OPT-2.7B and Llama-3.1-8B. The TPR@1% metric in Table 1 requires a threshold that admits exactly 1% of unwatermarked text. If that null distribution is Mistral output, the threshold can classify OPT/Llama text as watermarked because of cross-model stylistic differences in sentence embeddings and cluster membership, not because of CoheMark's rule. No same-model unwatermarked negative set is reported. The absolute claim of 'strong watermark strength' is therefore not yet established; this concern is independent of the unablated NSSC. A relative ranking among baselines might survive the fix, but the headline detection numbers could be inflated by source-model identity rather than by the watermark itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoheMark, a sentence-level watermarking method that uses fuzzy c-means clustering over sentence embeddings and hand-crafted next-sentence selection criteria (NSSC v1/v2) to define green/red semantic spaces. Generation proceeds by rejection sampling: a candidate sentence is accepted if its primary membership cluster falls in the current green set, where the green set is derived from the membership index of the previous sentence. Detection computes the proportion of sentences satisfying the rule and thresholds it to achieve a target false-positive rate. The authors report TPR@1% values of 97.3–100% across OPT-2.7B and Llama-3-8B on OpenGen and LFQA, claim competitive or superior text quality based on GPT-4o pairwise preferences, and show robustness to sentence-level paraphrase attacks. The central claims are that CoheMark achieves strong watermark strength while minimally impacting text quality.","tokens_in":14951,"tokens_out":3360,"duration_ms":33977,"significance":"If the detectability result were established with a properly matched null distribution, the idea of using coherence-driven sentence selection for watermarking would be a modest but useful contribution to the sentence-level watermarking literature. The paper is thorough in comparing against six baselines, two base models, and two datasets, and it honestly reports failure rates, runtime, and the absence of a k-SemStamp comparison. However, the current evidence does not yet support the headline claim: the detection metric is computed against a mismatched unwatermarked text distribution, the quality advantage rests on a single 100-prompt GPT-4o preference evaluation with no statistical analysis, and the central NSSC design choices are never ablated. The contribution is plausible but needs substantial additional validation.","major_comments":[{"comment":"The TPR@1% values in Table 1 are not comparable across models because the unwatermarked negative set is generated by Mistral-7B-Instruct-v0.3 while the watermarked positives are generated by OPT-2.7B and Llama-3-8B. As described in Section B, the unwatermarked text is produced by asking the aligned Mistral model to 'mimic human-written text', so the 1% FPR threshold is calibrated on Mistral output. Since sentence embeddings and fuzzy-cluster memberships are sensitive to model identity and style, the reported TPR may partly reflect cross-model differences rather than the watermark rule. The authors must report TPR@1% using a same-model unwatermarked negative set (e.g., unwatermarked generations from OPT-2.7B and Llama-3-8B with the same sampling settings) for each model/dataset combination. Until this is done, the central claim of 'strong watermark strength' is not established.","section":"Section B / Table 1"},{"comment":"The failure-generation rates of 0.5%–4.5% in Table 6, combined with the maximum-trial stopping rule described in Section A, mean that TPR@1% is computed only on successful generations. If prompts that hit the trial limit are dropped from the detection evaluation, the reported TPRs are conditioned on an outcome that may correlate with text difficulty, length, or topic. The authors should report intent-to-treat rates (counting failures as undetected) or demonstrate that failures are statistically independent of the detection outcome. Without this, the detection numbers may overstate the usable watermark strength in practice.","section":"Section 4 / Table 6"},{"comment":"The text-quality claim relies almost entirely on GPT-4o pairwise preference scores over 100 OpenGen prompts, with no confidence intervals, no inter-judge agreement, and no significance tests for the soft win rates. Moreover, the evaluation prompt explicitly asks the judge to assess 'coherence', while CoheMark is explicitly designed to enforce topical overlap via NSSC; the evaluation is therefore partly measuring the method's intended design rather than an independent property. The authors should validate the LLM judge against human ratings, report variance across prompts and judges, and include a non-circular quality metric such as perplexity under a held-out LM or task-specific automatic metrics, rather than dismissing the traditional metrics in Table 1 on the basis of a single unvalidated judge.","section":"Section 4.2 / Table 2"},{"comment":"The NSSC v1 and v2 green/red index sets and the switching rule are hand-crafted and never ablated; the paper explicitly states that optimization of these rules is left for future work. Because the entire watermark depends on these specific choices (e.g., green indices {1,3} vs. {2,4,5,6}, and switching after 5 matches), a reader cannot rule out that the strong detection and quality results are artifacts of a particular configuration. At minimum, an ablation varying the green-index set, the membership ranking cardinality, and the switching threshold should be provided for one model/dataset pair to show the method is not a knife-edge construction.","section":"Section 4 / NSSC definition"}],"minor_comments":[{"comment":"There are several typos and inconsistencies, including 'Qusetion 1' and 'Qusetion 2' in Section 4.2, 'explaination' in Section 3.1, and inconsistent capitalization of 'Semstamp' vs. 'SemStamp'.","section":"Throughout"},{"comment":"The authors state that CoheMark outperforms baselines on 'accuracy' and 'reliability' in the case study, but the CoheMark example in Table 4 itself contains fabricated historical details and incorrect elevation data, as acknowledged in the text; the claimed superiority in factual accuracy is therefore not persuasive.","section":"Section 4.2"},{"comment":"The robustness results after paraphrasing are presented only as a line plot without numerical values, error bars, or the number of prompts used; reporting the raw TPR values and sample sizes would allow readers to assess the significance of the robustness advantage.","section":"Figure 2"},{"comment":"SemStamp results are omitted for Llama-3-8B because of infinite loops, and k-SemStamp is not compared at all; the paper should at least discuss how the missing baselines affect the comparison, especially since SemStamp is the closest prior sentence-level method.","section":"Section 4 / Baselines"}],"recommendation":"major_revision","confidential_remarks":"The cross-model null distribution issue is the most serious technical problem; it directly affects the headline detection numbers and must be fixed. The quality evaluation also needs substantiation beyond a single unvalidated LLM judge. However, these issues are fixable with additional experiments, so the paper is not beyond repair. The authors should be encouraged to provide code and statistical rigor if they intend to pursue a journal version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoheMark is an incremental but honest extension of SemStamp-style sentence-level watermarking. What is actually new is using fuzzy c-means membership rankings to define the green/red sentence spaces and a hand-set switching rule between two next-sentence criteria. That is a plausible variation, not a new capability or theory. I would not call it a breakthrough, but it is the kind of variation the subfield needs tested.\n\nCredit where due: the paper compares six baselines, two base models, two datasets, reports runtime and generation failure rates, and openly states it did not compare k-SemStamp and leaves NSSC optimization to future work. The robustness results after paraphrase and the failure statistics are useful practical data. The citation pattern looks fine; the main missing comparison is explicitly acknowledged. The method itself is coherent and the fuzzy-clustering motivation is sensible.\n\nMain soft spot: the detection benchmark is confounded. Section B says unwatermarked text used for TPR@1% is generated by Mistral-7B-Instruct-v0.3, while the watermarked positives come from OPT-2.7B and Llama-3.1-8B. The threshold is calibrated to admit 1% of Mistral text. If sentence embeddings differ by source model, the detector can flag OPT/Llama text as watermarked regardless of the CoheMark rule. No same-model unwatermarked negative set is reported, so the absolute detectability numbers are not established. A relative ranking among baselines could survive the fix, but the headline numbers may be inflated. This is independent of the NSSC issue.\n\nSecond: NSSC v1/v2 and the switching rule are hand-crafted, never ablated, and the paper explicitly leaves their optimization open. Those choices sit at the center of the method; we do not know how sensitive the results are to them. Third: the quality advantage rests primarily on GPT-4o pairwise preferences over 100 prompts, with no error bars and with a judge explicitly asked to reward coherence and relevance—exactly what NSSC was designed to enforce. The paper's own Llama PPL/BertScore numbers show CoheMark underperforming KGW and Unbiased, so the quality story is less settled than the abstract implies. Fourth: failure rates of 0.5–4.5% are reported in the appendix but it is not clear whether failed generations are included or dropped; if dropped, both detection and quality numbers are biased upward. No code or cluster-training details are given, which makes reproduction harder.\n\nWho is this for: watermarking researchers and deployers tracking the quality–detectability trade-off. It is a worthwhile workshop-level contribution that deserves a serious referee, but the review should require same-model negatives, NSSC ablations, and clarity on failures/code. I'd send it out, and I'd expect revision rather than acceptance as-is.","headline":"A solid incremental sentence-level watermarking paper whose headline detection numbers are undercut by a cross-model null distribution, but the method is coherent and worth refereeing.","tokens_in":15433,"tokens_out":3889,"would_cite":false,"duration_ms":36212,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoheMark claims a sentence-level watermark that keeps text readable by steering each sentence into the topic clusters of the previous sentence while maintaining detection rates above 97%.","keywords":["sentence-level watermarking","fuzzy c-means clustering","cohesion","text quality","LLM watermarking","rejection sampling","watermark robustness","semantic coherence"],"falsifier":"Replace the hand-picked green clusters with random ones of the same size and re-measure quality and detection: if quality holds, cohesion is not the source of the advantage.","tokens_in":14542,"feed_emoji":"💧","tokens_out":7678,"duration_ms":64728,"temperature":0.7,"pith_summary":"The paper claims that a sentence-level watermark can be embedded without sacrificing text coherence by making the acceptance of each generated sentence depend on its semantic proximity to the previous sentence. CoheMark clusters a domain's sentence embeddings with fuzzy c-means, represents each sentence by a ranking of cluster memberships, and accepts only sentences whose top cluster falls in the 'green' slots defined by the previous sentence's ranking. On OpenGen and LFQA with OPT-2.7B and Llama-3-8B, the method reports true positive rates at 1 percent false positive of 97 to 100 percent, comparable to the strongest baselines, and pairwise GPT-4o evaluations rate its text above KGW, EXP, Unbiased, and SemStamp on coherence, relevance, interestingness, and integrity. The paper's point is that the quality-strength trade-off in watermarking is not fixed: exploiting inter-sentence cohesion can push both at once.","feed_headline":"Topic-aware watermarks keep AI text coherent and detectable","feed_subtitle":"CoheMark steers each new sentence into the previous sentence's topic clusters, keeping quality and detectability together.","key_machinery":"The machinery is the pair formed by the FuzzyClusterer and the CoheSampler, linked by the Membership Index: a ranking of a sentence's fuzzy c-means membership degrees across clusters. The FuzzyClusterer learns soft clusters over sentence embeddings from the target domain, and the Membership Index turns any new sentence into a ranked list of cluster affinities. The CoheSampler uses a hand-specified Next Sentence Selection Criterion to read the previous sentence's ranking and declare certain rank positions green; the next sentence is accepted only if its top-ranked cluster sits in one of those positions. The switching between NSSC v1 and v2 is intended to keep text locally coherent while periodically pivoting to less-represented topics. This rejection-sampling loop carries the argument: it replaces arbitrary semantic-region division with a cohesion-guided one.","core_discovery":"The central discovery is that choosing the valid semantic spaces for the next sentence by the membership profile of the previous sentence—rather than by a random red/green split—preserves textual cohesion while maintaining detectability. CoheMark first trains fuzzy c-means clusters on sentences from the target domain, then at each generation step embeds the previous sentence and ranks the clusters by membership degree. The next sentence is generated and re-embedded until its highest-membership cluster falls in the green set defined by the current Next Sentence Selection Criterion: NSSC v1 takes the 1st and 3rd clusters in the previous sentence's ranking, NSSC v2 takes the 2nd, 4th, 5th, and 6th, with a switch after five consecutive matches. Detection checks what fraction of sentences in a text obey the same criterion. The paper reports TPR@1% of 99.3 and 97.3 across models on OpenGen, 100 and 97.3 on LFQA, and GPT-4o soft win rates of 61-95 percent over four baseline families.","pith_inferences":["The exact Next Sentence Selection Criteria appear unoptimized: nothing in the paper tests alternatives, so a natural next experiment is varying which rank positions are green and how many matches trigger the switch.","If the cohesion hypothesis is the real driver, the fuzzy c-means layer is replaceable by any soft clusterer that yields the same membership rankings, so the method could transfer to other embedding spaces.","The robustness evaluation uses a particular sentence-by-sentence paraphrase attack; stronger adversarial rewrites that deliberately break topic continuity would stress the watermark differently.","The LLM-judge results are reported without a human preference study, so a direct human evaluation would settle whether the quality advantage is as large as GPT-4o says."],"forward_implications":["If CoheMark works as reported, sentence-level watermarks can avoid modifying the model's logit distribution and still exceed 97 percent true positive rate at 1 percent false positive rate.","The method makes text quality a design input rather than an afterthought: generated text is kept on the topic established by the previous sentence, which the paper supports with GPT-4o pairwise win rates over KGW, EXP, Unbiased, and SemStamp.","CoheMark reports higher true positive rates than the baselines after sentence-level paraphrase attacks, indicating the cohesive selection survives rewording.","The runtime cost is real but bounded: with OPT-2.7B on OpenGen, CoheMark averages about 27 seconds per generation, roughly seven times slower than token-level watermarks and about three times faster than SemStamp.","The paper argues traditional automatic metrics such as perplexity and BertScore conflict with LLM-based evaluation of watermarked text, so quality conclusions should not rest on the traditional measures alone."],"supporting_citations":[{"why":"Provides the KGW token-level watermark family that CoheMark is compared against and whose green/red-list idea CoheMark extends to sentences.","marker":"Kirchenbauer et al., 2023a"},{"why":"SemStamp, the prior sentence-level semantic watermark whose arbitrary red/green division CoheMark aims to improve.","marker":"Hou et al., 2023"},{"why":"k-SemStamp, the hard-clustering variant that motivates CoheMark's use of soft fuzzy clustering.","marker":"Hou et al., 2024"},{"why":"Defines fuzzy c-means clustering, the algorithm CoheMark uses to compute soft membership of sentences.","marker":"Bezdek et al., 1984"},{"why":"Supports the paper's claim that fuzzy clustering is effective for sentence-level text.","marker":"Skabar & Abdalgader, 2011"},{"why":"Provides Sentence-BERT, the embedder that maps sentences to the embeddings used by clustering and sampling.","marker":"Reimers, 2019"},{"why":"Supplies the OpenGen and LFQA datasets used for generation and detection evaluation.","marker":"Krishna et al., 2024"},{"why":"Grounds the LLM-as-a-judge evaluation approach that yields CoheMark's pairwise quality win rates.","marker":"Zheng et al., 2023"}],"fun_headline_variants":["Watermark AI text via thematic sentence chains","CoheMark: Cohesion-based watermark for better AI text","Keep AI text natural and traceable with CoheMark","Sentence-level watermark that uses topic clusters","CoheMark: Watermarking without breaking flow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim rests on hand-picked rules for which topic clusters count as green for the next sentence, and those rules are never varied or tested.","fun_headline_variants_meta":{"raw":{"variants":["Watermark AI text via thematic sentence chains","CoheMark: Cohesion-based watermark for better AI text","Keep AI text natural and traceable with CoheMark","Sentence-level watermark that uses topic clusters","CoheMark: Watermarking without breaking flow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2383,"prompt_tokens":905,"completion_tokens":1478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1404}},"tokens_in":521,"tokens_out":1478,"duration_ms":10688,"temperature":1.0,"reasoning_tokens":1404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:42:25.096691+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the hand-picked green clusters with random ones of the same size and re-measure quality and detection: if quality holds, cohesion is not the source of the advantage.","supporting_citations":[{"cited_title":"Fcm: The fuzzy c-means clustering algorithm","cited_arxiv_id":null,"evidence_quote":"Defines fuzzy c-means clustering, the algorithm CoheMark uses to compute soft membership of sentences."},{"cited_title":"Clustering sentence-level text using a novel fuzzy relational clustering algorithm","cited_arxiv_id":null,"evidence_quote":"Supports the paper's claim that fuzzy clustering is effective for sentence-level text."}],"review_version":1}