{"id":"f1dd0e47-f72e-423a-9195-957048b7bee0","arxiv_id":"2504.16046","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BloomScrub detects long verbatim quotes from a protected corpus with a Bloom filter, rewrites them iteratively, and abstains when needed, certifying that no quote longer than the threshold is emitted.","lead":"This paper introduces BloomScrub, an inference-time filter that detects long verbatim quotes from a copyrighted corpus, rewrites them, and abstains when it cannot. It offers a formal guarantee that no quote longer than a chosen threshold appears in the final output.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The certificate is corpus-relative: BloomScrub guarantees only normalized verbatim matches from the pre-indexed corpus C, not the 'copyrighted sources' of the abstract.","rationale":"I read the paper in good faith. The certificate itself is valid under its stated definitions: Bloom filters have no false negatives, and the final output is either scrubbed below the threshold or abstained. The reader's CONDITIONAL verdict is appropriate. My concern is the same weakest assumption the reader identified: the abstract's unqualified 'certified copyright takedown' could be read as a guarantee over all copyrighted sources, whereas the actual guarantee covers only normalized verbatim matches from the pre-indexed corpus C. The paper's Limitations section acknowledges non-literal copying, but the abstract does not carry the corpus-relativity qualifier. The proposed holdout test will quantify how much of the worst-case risk transfers to unindexed sources. If the held-out corpus shows vanilla-level quote rates, the title and abstract should be qualified; if it shows near-zero rates, the concern is largely rhetorical. I do not see a flaw that would reject the method within its declared scope, so the reader's CONDITIONAL verdict stands unchanged.","tokens_in":20474,"tokens_out":14051,"duration_ms":140714,"concrete_test":"Build the Bloom filter from a random 50% subset of the target corpus (call it C_A), run BloomScrub with the same tau and n as in the paper, and compute %R>Q(tau) against the held-out half C_B using exact string search rather than a Bloom filter. If long-quote rates on C_B remain near the vanilla model while C_A drops to near zero, the certificate is strictly C-relative and the abstract/title should be qualified to 'from the indexed corpus.' If C_B also shows near-zero quote rates, the scope concern is weak and the current wording is acceptable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core guarantee in §3.2(A) is mathematically sound for the indexed corpus: a Bloom filter has no false negatives, so any normalized verbatim substring of C longer than the detection width n is found, and the iterative loop either scrubs it or abstains. The load-bearing question is what this certificate applies to. It applies only to character n-grams inserted from C; any source not in C, any quote whose normalized n-gram sequence was not inserted, and any paraphrase, synonym substitution, or light edit is invisible to the detector. The abstract and title present this as 'certified copyright takedown' and 'worst-case copyright infringement' without that qualifier, and the Limitations section concedes only that non-literal copying is outside the method. In a real deployment the relevant copyrighted universe is not a fixed, fully indexed C, so the certified statement is narrower than its framing. This is not an internal inconsistency, but it is the assumption on which the practical force of the central claim rests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BLOOMSCRUB, an inference-time mitigation pipeline that interleaves Bloom-filter-based detection of verbatim spans from a copyrighted corpus C with LLM-based rewriting, and optionally replaces the response with a refusal if long spans persist. The formal claim is that, because a Bloom filter has no false negatives, any normalized verbatim substring of length at least n from the indexed corpus is detected, and the iterative loop either rewrites it below a threshold tau or abstains, so emitted responses contain no quote from C longer than tau. The authors evaluate on NewsSpan and NewsQA with a fine-tuned Llama-3.1-8B generator, comparing to SysPrompt, MemFree decoding, and R-CAD on a new corpus-level metric %R>Q(tau) plus reference-based LCS/ACS, information quality, and QA utility. Results show near-zero long-quote rates with small utility loss.","tokens_in":20622,"tokens_out":11611,"duration_ms":111348,"significance":"If the guarantee is stated at the right level of scope, this is a useful and clean result: it gives a verifiable hard constraint on verbatim regurgitation from a fixed, indexed corpus, at inference time, with black-box model access, and it scales with a Bloom filter. The paper's decision to make the evaluation corpus-level rather than reference-bound is a genuine improvement, and the iterative quote-guided rewriting is a simple and practical mechanism. The Limitations section is honest about the method's narrow scope for non-literal copying. The main gap is a mismatch between the formal, corpus-relative certificate and the broad 'certified copyright takedown' language in the title and abstract; tightening this framing is essential before publication.","major_comments":[{"comment":"The certificate is only for normalized verbatim matches against the specific, fully indexed corpus C. Since only n-grams inserted into the Bloom filter can be detected, any quote from a source outside C, and any paraphrase or lightly edited passage whose normalized n-gram sequence is not in C, is outside the guarantee. The abstract's 'worst-case LLM copyright infringement' and 'copyrighted sources' should be qualified throughout to 'the indexed corpus C'; the Limitations section already concedes non-literal copying, but the title and abstract should not imply a broader certificate. I suggest stating the guarantee as a formal theorem with explicit quantifiers: for a fixed normalized corpus C, with probability 1 (no false negatives), every emitted response is either a refusal or contains no substring s of C with |s| > tau.","section":"Abstract; §3.1; §3.2(A)"},{"comment":"The Introduction says BLOOMSCRUB provides 'certified copyright takedown—completely eliminating long verbatim quotes from copyrighted sources.' Without abstention, Table 2 and Figure 2 report %R>Q(50) values of 10.9% (NewsSpan) and 5.6% (NewsQA) for BLOOMSCRUB, and %R>Q(100) of 0.0% and 0.1%. Thus 'complete elimination' is only true when the abstention mechanism is enabled, as reported in Table 3. Please revise the claim to explicitly tie elimination to the abstention option, and report both modes consistently in the abstract and introduction.","section":"Introduction; Table 2; Figure 2"},{"comment":"Algorithm 1 checks the abstention condition at line 9 using quotes extracted at line 4, i.e., from the response before the final rewrite. As written, whenever the loop exits because the iteration limit is reached after a rewrite, the system abstains even if that final rewrite removed all long quotes; conversely, the check does not inspect the final y that would actually be returned. This is conservative for the safety guarantee but is inconsistent with the prose in §3.2(B) ('if ... the rewrite model still fails to remove all long verbatim quotes') and inflates the measured abstention cost. Please change the pseudocode to re-run the quote extractor on the final y before the abstention decision, and adjust the loop condition so that imax denotes the maximum number of rewrites (e.g., while i < imax).","section":"§3.2(B); Algorithm 1"}],"minor_comments":[{"comment":"The citation for MemFree decoding is to Ippolito et al. (2019), but the described method is from Ippolito et al. (2022); the bibliography contains both, so the citation should be corrected.","section":"§4.3"},{"comment":"Figure 6 lacks axis labels and a clear caption for the y-axis, which makes the Levenshtein-distance comparison hard to read.","section":"Figure 6"},{"comment":"Algorithm 1's line 8 contains the fragment 'i + +'; use 'i ← i+1' or 'i = i+1' for clarity.","section":"Algorithm 1"},{"comment":"In Appendix A, 'under the pertaining mode' appears to be a typo for 'pretraining mode'; please clarify the fine-tuning setup.","section":"Appendix A"},{"comment":"The empirical comparison would be stronger with standard errors or multiple fine-tuning seeds, since some utility comparisons are essentially null results (e.g., QA F1 47.8 vs 47.9 on NewsSpan) and a single run cannot establish equivalence.","section":"Tables 2–4"},{"comment":"The normalization used before n-gram insertion (whitespace, punctuation, case) should be defined precisely, since the exact coverage of the certificate depends on this normalization.","section":"§3.2(A)"}],"recommendation":"major_revision","confidential_remarks":"The authors do not include code; given that the method is simple and the certificate depends on implementation details (Bloom filter insertion, normalization), releasing code would materially help verification. I otherwise see no concerns about novelty or attribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper's core idea is genuinely useful: a black-box, inference-time loop that uses a Bloom filter to spot verbatim quotes from a corpus of interest, rewrites them with an LLM, and abstains when it can't get below threshold. The certified claim is sound within its stated scope: a Bloom filter has no false negatives, so if the final output is checked, no quote longer than tau from the indexed, normalized corpus can survive. That is a real, lightweight, updatable safety layer for the clearest copyright exposure, and it's not something the baselines offer.\n\nThe empirical work is decent. They show large reductions in long quote rates versus vanilla, system prompt, MemFree, and R-CAD, with utility and info quality roughly preserved, and an ablation showing quote guidance matters. The new metric %R>Q(tau) is a reasonable worst-case measure.\n\nNow the soft spots, in proportion. First, the guarantee is corpus-relative. The abstract says \"certified copyright takedown\" and \"worst-case\" without saying this only covers normalized verbatim matches from the pre-indexed C. Paraphrase, synonym substitution, and sources not in the index are invisible. The Limitations section does concede non-literal copying, but the title and abstract still oversell. That's a framing issue, not a mathematical flaw.\n\nSecond, Algorithm 1 has a real bug: the final rewrite's output is not re-checked. The loop extracts quotes, then rewrites, then increments. If the loop exits because max iterations are reached, the check uses quotes extracted before the last rewrite. So the certificate does not hold for the actually-returned text unless you re-extract after the final rewrite. Easy fix, but load-bearing.\n\nThird, abstention rates are not reported in Table 3. We see info quality drops slightly, but without knowing how often the system refuses, we can't judge the utility/abstention trade-off. Fourth, no error bars or multiple seeds; for the headline effect sizes that's a minor concern, but worth noting.\n\nWho is this for? Anyone working on copyright mitigation, memorization, or inference-time safety. It deserves a serious referee. The central idea is sound, the bug is fixable, and the empirical claims are promising if not fully complete.","headline":"BloomScrub's certified guarantee is real but only covers verbatim quotes from the indexed corpus; the paper is a solid, useful contribution that overstates its scope a bit and has a small algorithmic bug to fix.","tokens_in":21199,"tokens_out":2771,"would_cite":true,"duration_ms":24107,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BloomScrub certifies that large language model outputs can be kept free of long verbatim quotes from an indexed copyrighted corpus.","keywords":["copyright takedown","Bloom filter","worst-case infringement","verbatim quote detection","inference-time mitigation","certified risk reduction","LLM memorization","abstention"],"falsifier":"Run BloomScrub on a large test set and then re-check every emitted response with an exact, false-positive-free index (for example, a suffix array built from the same normalized corpus $C$); if any non-refusal response contains a substring longer than $\\tau$ that appears in $C$, the certified claim is false. A cheaper partial check is to count how often the Bloom filter's false positives trigger rewrites: a high false-positive rate would reveal the cost of the sketch, though it would not by itself break the certificate.","tokens_in":20240,"feed_emoji":"🧹","tokens_out":8490,"duration_ms":71428,"temperature":0.7,"pith_summary":"The paper contends that the copyright risk that matters most—a model occasionally emitting a long verbatim passage from protected text—can be eliminated at inference time, without retraining and without logit-level access. It introduces BloomScrub, which builds a Bloom filter over character $n$-grams of a copyrighted corpus, scans each generated response for verbatim quotes, and repeatedly rewrites detected quotes with an instruction-tuned model. If, after a fixed number of rewriting rounds, a quote longer than a user-chosen threshold $\\tau$ remains, the system refuses to answer. The certified statement is therefore that every served response either contains no normalized verbatim substring longer than $\\tau$ from the indexed corpus, or is a refusal; on two news corpora the method drives the percentage of responses containing a quote longer than 100 characters from roughly 20% down to near zero, reaching 0.0% at $\\tau=50$ with abstention, while keeping fact-answering utility essentially intact. The reason a sympathetic reader should care is that it turns a fuzzy legal worry into an auditable, corpus-relative guarantee at low deployment cost.","feed_headline":"BloomScrub certifies LLM outputs free of long verbatim quotes","feed_subtitle":"An inference-time detect-rewrite-abstain loop drives long-quote rates to zero while preserving factual answers.","key_machinery":"The central object is a fixed-width Bloom filter over character $n$-grams ($n=25$) of the copyrighted corpus, used as the quote extractor $E_C$. Its defining property is zero false negatives: every $n$-gram inserted is always reported, which makes detection of every quote of length $\\geq n$ a certainty rather than a heuristic. The extractor aggregates $k$ consecutive $n$-gram hits at one-character offsets into a single quote of length $n+k-1$, so near-verbatim stitched quotes are caught. The second component is iterative quote-guided rewriting: the extracted quotes are inserted into a template telling a rewrite model to paraphrase the text and completely rewrite those spans; detection and rewriting alternate until no quote exceeds $\\tau$ or the iteration budget is exhausted. The optional abstention step is what converts soft reduction into a hard, certified statement: when the loop cannot clean the response, the system returns a refusal, so the guarantee holds unconditionally for the indexed corpus.","core_discovery":"On its own terms, the paper establishes that a lightweight inference-time loop can provide certified protection against worst-case verbatim copying. For a protected corpus $C$ and a length threshold $\\tau$, the guarantee is: any final response is either a refusal, or no normalized substring of it longer than $\\tau$ appears in $C$. The certificate is carried by the Bloom filter's zero-false-negative property: every character $n$-gram of $C$ is inserted, so any quote of length at least $n=25$ is detected, and the extractor merges consecutive overlapping hits into one long span, catching near-verbatim stitched quotes. The paper's experiments compare BloomScrub with decoding-time baselines and show it produces the lowest maximal character- and word-level LCS, nearly eliminates quotes above 100 characters, and preserves QA F1 and information-quality scores; removing the quote-guidance from rewriting makes risk metrics worse, supporting the claim that explicit quote extraction is load-bearing. With abstention, the method reaches $0.0\\%$ on its corpus-level metric $\\%R_{>Q}(\\tau)$ at $\\tau=50$.","pith_inferences":["If certified verbatim removal becomes a standard deployment feature, the practical legal question shifts to what is loaded into the index; the guarantee is only as broad as the corpus $C$, so a provider's indexing choices become a policy decision.","The same detect-rewrite-abstain loop transfers to any exact-match constraint—personally identifying information, toxic phrases, confidential identifiers—because the certificate depends only on the zero-false-negative filter and the rewrite model, not on copyright semantics.","A natural next experiment, which the paper does not run, is to replace the Bloom filter with an exact suffix-array index: that would keep the certificate intact and remove the small false-positive cost, with memory as the trade-off.","The paper's own limitation that paraphrases are out of scope suggests room for a layered guard: a verbatim certificate from this method plus a similarity-based detector for non-literal copying; the two attack different failure modes."],"forward_implications":["A provider can publish the indexed corpus sketch and the threshold $\\tau$, and any auditor can verify compliance by exact matching, turning a claim about copyright risk into a checkable property of the served output stream.","Updating the protected corpus—for instance, after a new licensing agreement—requires only rebuilding the Bloom filter; no model retraining and no access to model logits is needed.","Because each rewrite iteration tightens the effective threshold, a single deployment can tune enforcement stringency at inference time to match the legal appetite of the use case.","The corpus-level metric $\\%R_{>Q}(\\tau)$ gives future mitigation methods a worst-case benchmark that does not depend on which ground-truth continuation happens to be paired with a prompt."],"supporting_citations":[{"why":"Supplies the Bloom filter data structure whose zero-false-negative property is the basis for certified quote extraction.","marker":"(Bloom, 1970)"},{"why":"Provides the indexing tool the paper uses to insert all character n-grams of the corpus into the Bloom filter sketch.","marker":"(Marone and Van Durme, 2023)"},{"why":"Defines the copyright-takedown task, the evaluation protocol, and the win-rate and utility measures that BloomScrub is measured against; also the source of the R-CAD baseline and the system-prompt comparison.","marker":"(Wei et al., 2024)"},{"why":"Defines MemFree decoding, the hard n-gram-removal baseline, and the 150-character threshold reference for blocking verbatim matches.","marker":"(Ippolito et al., 2022)"},{"why":"Supplies context-aware decoding, the method R-CAD inverts to steer generations away from retrieved copyrighted context.","marker":"(Shi et al., 2023)"},{"why":"Supplies evidence that literal and non-literal copying both need measurement and motivates the stitched-quote detection used in the extractor; also states the non-literal limitation.","marker":"(Chen et al., 2024)"},{"why":"Provides the NewsSpan corpus of New York Times articles used for training, evaluation, and utility QA generation.","marker":"(Cheng et al., 2024)"},{"why":"Provides the NewsQA corpus and QA pairs used as the second copyright dataset.","marker":"(Trischler et al., 2016)"},{"why":"Supplies the fair-use and transformativeness rationale that makes long verbatim quotes the legal risk to mitigate.","marker":"(Henderson et al., 2023)"}],"fun_headline_variants":["Bloom filter loop certifies LLMs against verbatim quotes","Inference-time loop scrubs long quotes with certified guarantee","Zero false negatives: BloomScrub erases long verbatim LLM outputs","Certified copyright takedown: detect, rewrite, or refuse","Worst-case copyright risk: BloomScrub certifies no long quotes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee covers only verbatim substrings of the specific normalized corpus $C$ loaded into the Bloom filter; any paraphrase, synonym substitution, light edit, or quote from a document outside that index is invisible to the detector and outside the certificate.","fun_headline_variants_meta":{"raw":{"variants":["Bloom filter loop certifies LLMs against verbatim quotes","Inference-time loop scrubs long quotes with certified guarantee","Zero false negatives: BloomScrub erases long verbatim LLM outputs","Certified copyright takedown: detect, rewrite, or refuse","Worst-case copyright risk: BloomScrub certifies no long quotes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3327,"prompt_tokens":975,"completion_tokens":2352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2262}},"tokens_in":591,"tokens_out":2352,"duration_ms":15235,"temperature":1.0,"reasoning_tokens":2262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:12:49.162227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BloomScrub on a large test set and then re-check every emitted response with an exact, false-positive-free index (for example, a suffix array built from the same normalized corpus $C$); if any non-refusal response contains a substring longer than $\\tau$ that appears in $C$, the certified claim is false. A cheaper partial check is to count how often the Bloom filter's false positives trigger rewrites: a high false-positive rate would reveal the cost of the sketch, though it would not by itself break the certificate.","supporting_citations":[],"review_version":1}