{"id":"a88f81d6-fa2a-4d88-8e50-7a0dd8bdd281","arxiv_id":"2607.13037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A record- and token-level provenance system that turns author revocation requests into precise forget sets, cutting dataset-level over-deletion from 101x to 1.3x on wiki data.","lead":"OriginBlame is a new software system that tracks which author wrote which lines in a training dataset, so companies can remove one person's data instead of deleting the whole dataset. It matters because regulators and users want data removal, and AI unlearning tools need a precise list of what to forget first.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.7's NPO comparison may evaluate forget metrics on different sets ('respective splits'), so the 42% improvement over random could be an artifact of set composition rather than provenance.","rationale":"In good faith, the paper contributes a plausible systems architecture, a clear over-deletion analysis, and a reasonable performance study. The revocation-precision numbers in Section 5.2 are internally consistent, and the reconcile experiment is a reasonable functional validation. However, the central claim in the abstract and in the reader's strongest_claim depends on the unlearning comparison in Section 5.7. The paper's wording strongly suggests that each condition's metrics are computed on its own forget set, which would invalidate the comparison. This is more load-bearing than the reader's identified weak premise (author metadata availability), because even with perfect metadata the headline unlearning result could be spurious. The reader's conditions (code release, variance reporting, O(1) clarification) are reasonable but do not address this potential flaw. I recommend keeping the verdict conditional—pending the re-evaluation described—rather than rejecting outright, because the issue is testable and the rest of the system may stand on its own. Agreement is partial: the reader focused on an external precondition, while I identify an internal evaluation-validity concern.","tokens_in":14150,"tokens_out":9223,"duration_ms":103149,"concrete_test":"Reproduce the NPO random-baseline run from Section 5.7, then evaluate forget PPL and ROUGE-L on the *same* 1,000-line sample of the target author's lines used in the line-level condition (and, for completeness, also on the random set). Compare the random-condition model's PPL on the author's lines to the line-condition model's PPL on those same lines. If the random-condition PPL is near SFT's ~3.5 while line-condition is ~8.97, the claimed 42% advantage is an artifact of mismatched evaluation sets. If the random-condition PPL on the author's lines is also ~6.4, the conclusion holds. This requires only re-running evaluation, not retraining.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that provenance-based forget sets improve unlearning by 42% over random baselines—rests on Table 8 in Section 5.7. The setup constructs two different forget sets: line uses the target author's lines; random draws a same-size random subset from the full corpus. The evaluation text says: 'All metrics are computed on sampled subsets (1,000 lines for PPL, 100 for ROUGE-L) from the respective splits.' If 'respective splits' means each condition is evaluated on its own forget set, then the random condition's forget PPL is measured on random lines, not on the target author's lines. This does not answer the compliance question: did the model forget the author's content? Comparing 8.97 (line) to 6.43 (random) across different sets only shows that NPO increases PPL on whatever set it was applied to, and that the author's lines may be easier to unlearn than a diverse random sample. The SFT baseline row is ambiguous—it cannot simultaneously be the baseline for both different forget sets. Thus the 42% improvement may be an artifact, not evidence that provenance-based selection better targets the author's data. This is an internal-validity concern, distinct from the reader's identified assumption about author metadata availability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"OriginBlame (ob) is a provenance system that propagates authorship metadata through data-processing pipelines. It stores records in a three-layer content-addressable index (authors, sections, document-index) plus an optional token-index layer for packed/tokenized data. Queries resolve contributor identities into record- or token-level forget sets. The paper evaluates on zhwiki (219,555 pages) and Linux kernel sources, measuring revocation precision, reconcile recovery, query latency, framework overhead (HuggingFace Datasets, Datatrove), and a downstream NPO/RMU/gradient-ascent unlearning study on Qwen3-1.7B. Headline results are reduced over-deletion (101x to 1.3x), modest integration overhead, and a 42% improvement in NPO forget PPL for provenance-selected vs. random forget sets.","tokens_in":14405,"tokens_out":10133,"duration_ms":113476,"significance":"If the main unlearning comparison is valid, the paper addresses a real, under-served problem: converting an author-level revocation request into a fine-grained forget set for machine unlearning. The architecture is simple and deterministic, the core has no ML/GPU dependency, and the integration surface is small. The cross-domain Linux kernel experiment is a useful generality check. The strengths include machine-checkable deterministic query design and a clear separation of propagation from authorship discovery. However, the strongest downstream claim is not currently well supported (see Major Comments), and the token-index contribution is validated only at the level of overhead and latency, not correctness or downstream use. The paper's scope is also narrower than the abstract suggests because ob propagates, not discovers, authorship; this is acknowledged in Discussion but should be foregrounded early.","major_comments":[{"comment":"The key unlearning comparison is not interpretable as reported. The two conditions use different forget sets—target-author lines vs. a same-size random subset of the corpus—and the text says all metrics are computed 'from the respective splits.' If that means the random condition's forget PPL/ROUGE is measured on the randomly selected lines, the comparison only shows that NPO degrades the model on whichever lines it was trained to forget; it does not show better targeting of the requester's content. The SFT row in Table 8 is also undefined under this reading because it cannot be a common baseline for two different splits. Fix the protocol: evaluate both conditions on a fixed held-out sample of the target author's lines and a fixed retain sample, and report the random condition's own-set metrics as secondary.","section":"§5.7, Table 8"},{"comment":"The unlearning experiments appear to be a single run per condition with no seeds, variance estimates, or significance tests. Given stochastic QLoRA/NPO, differences like retain PPL 4.23 vs 5.89 (Berthe) or 4.93 vs 6.04 (Antigng) could be within run-to-run noise. Evaluation subsets are also small (1,000 lines for PPL, 100 for ROUGE-L). Please report multiple seeds with means and standard deviations, and state whether the random forget set excludes target-author lines; if it does not, report the overlap between the random and line-level sets.","section":"§5.7, experimental protocol"},{"comment":"The token-level layer is advertised as a third contribution, but the evaluation validates only throughput, latency, and metadata preservation. There is no check that generate-set bitmasks correspond to the correct token ranges after ob clean merging, no correctness test against ground-truth token spans, and no downstream token-level unlearning experiment. Without this, the token-level claim remains a design assertion rather than a demonstrated capability. Add a correctness/alignment check and, if feasible, a small token-level forget-set experiment.","section":"§4.5, §5.5"}],"minor_comments":[{"comment":"The terms 'record,' 'line,' and 'page' are used somewhat interchangeably. Clarify the unit of provenance: is a record one wiki page, one section, or one JSONL line? This affects how to read Table 3 and the unlearning forget sets.","section":"§5.2/§5.7"},{"comment":"The reconcile mutation replaces 10% of alphabetic characters with random lowercase letters, which is ill-suited to zhwiki's predominantly Han text. Many lines may contain no alphabetic characters and therefore remain unchanged, inflating the hash-recovery rate. Report the actual fraction of mutated lines whose hashes changed, or use a CJK-appropriate mutation.","section":"§5.3, Appendix B"},{"comment":"Reconcile relies on nomic-embed-text-v1.5 with cosine threshold 0.85. No sensitivity analysis is provided; report how recovery varies with threshold and embedding model.","section":"§5.3"},{"comment":"Datatrove throughput overhead is nonmonotonic (13.8%, 19.0%, 13.4%, 2.1%). Explain the pattern, and fill the missing full-scale Datatrove entry in Table 7.","section":"§5.5, Table 6"},{"comment":"The manuscript cites GitHub URLs for code but does not include an artifact or data in the submission. Please provide a stable commit hash, evaluation scripts, and the zhwiki-derived QA data or a generation script to support reproducibility.","section":"General"},{"comment":"Minor wording issue: 'which ob clean later merges' is grammatically awkward; also, the WAL notation 'lock.{pid}→docidx.{pid}' should be defined the first time it appears.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's strongest downstream claim (42% improvement) rests on an evaluation design that is currently ambiguous. I would not reject, because the system itself and the over-deletion/overhead measurements are plausible and useful; but the unlearning experiment must be redone with a common evaluation set and multiple seeds, and the token-level claims need a correctness check. I also could not verify the code links because no artifact was included; please request one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"X,\n\nThe paper is worth reading for the architecture, but the headline unlearning number doesn't survive a close read. The system—three-layer content-addressable provenance with a token-index—fills a real gap: unlearning papers assume a forget set exists, and file-level provenance tools over-delete. The over-deletion numbers (101x down to 1.3x) are straightforward, and the overhead measurements look plausible for a JSONL metadata layer.\n\nWhat's genuinely new: the token-index layer that records document-to-token-range attribution during packing without per-token records, and the lazy revocation tag model. The reconcile two-phase recovery (hash then embedding) is a sensible design, and the 96–98% recovery is credible for the described mutations. The Linux kernel cross-domain check is a nice touch—git blame line attribution is the right comparison to The Stack's file-level approach. The limitations are honestly stated: no retroactive provenance, single-hop, no per-token attribution yet.\n\nThe soft spot is Section 5.7. The setup defines line forget sets as the target author's lines and random as a same-size random subset. Then metrics are computed \"on sampled subsets from the respective splits.\" If that means the random condition's forget PPL is measured on its own random lines, the 42% comparison is not apples-to-apples: it compares NPO's effect on author lines to NPO's effect on a random mix. The SFT baseline row can't be the baseline for both conditions simultaneously. The result may simply show that the author's lines are more homogeneous or easier to unlearn. The 42% claim is load-bearing for the abstract, and this internal-validity issue needs fixing—ideally by evaluating both methods on the same target author's lines as the compliance question.\n\nMinor points: no code or data shipped despite the GitHub links; unlearning runs are single-shot with no variance; the \"O(1) lookup\" claim is misleading given the index is opt-in and the sharding gives at best constant hashing plus bucket scan.\n\nNet: the provenance architecture is a real contribution and deserves reviewer time. The unlearning evaluation needs to be redone before the paper's headline claim can be trusted.\n\nRecommendation: send to peer review, but flag Section 5.7 for the authors to correct. I'd hold off citing until the code and corrected eval are public.","headline":"Solid provenance system with a suspicious headline unlearning result; the 42% claim likely confounds set composition with selection quality.","tokens_in":14917,"tokens_out":2228,"would_cite":false,"duration_ms":21814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hash-linked provenance index turns author revocation into precise line- and token-level forget sets.","keywords":["data provenance","machine unlearning","forget set","author revocation","content-addressable indexing","token-level provenance","NPO","over-deletion"],"falsifier":"If a real crawl or dataset passes through processing stages that strip authorship before ob.track() can run, the provenance chain is empty and the 1.3x over-deletion claim does not materialize. A direct experiment: take a corpus, build ob provenance, then run NPO with ob's forget set versus an oracle forget set of the same author's true lines — if unlearning metrics are statistically equal, the system's precision is not buying the stated improvement.","tokens_in":13973,"feed_emoji":"🏷️","tokens_out":3441,"duration_ms":33195,"temperature":0.7,"pith_summary":"The paper is trying to establish that the bottleneck in data removal from AI training is not the unlearning algorithm but the absence of a forget set. Its system, OriginBlame (ob), attaches author identity to each training record as it is written, carries that identity through tokenization and packing via SHA-256 hash links, and answers revocation requests with deterministic reverse queries. On 219k Wikipedia pages this cuts over-deletion from 101x to 1.3x, and on a 1.7B model a provenance-built forget set outperforms a same-size random one by 42% in forget perplexity while also improving retain perplexity by 23%. The system explicitly does not infer authorship: it propagates whatever attribution the source environment already has.","feed_headline":"Provenance cuts unlearning over-deletion from 101x to 1.3x","feed_subtitle":"Record-level author tracking feeds unlearning a precise forget set; on a 1.7B model it beats random by 42%.","key_machinery":"The three-layer content-addressable index (authors ← sections ← document-index) linked by SHA-256 hashes, plus an independent token-index layer storing cumulative token ranges. The workhorse is the reverse query: author id → matching section hashes → document-index entries, which yields a forget set without reading data files. The token-index's cumulative-count trick means each entry knows its token range from the sum of predecessors, giving position-indexed provenance with no per-token records.","core_discovery":"The central discovery is architectural: provenance at record and token granularity can be made cheap and queryable by storing three hash-linked layers — authors, sections, document-index — and a parallel token-index that records cumulative token counts during tokenization. Revocation is a lazy tag cascade: setting revoked=true on an author record, then traversing author → sections → document-index to produce the exact (file, line_hash) list. The token-index produces a bitmask over token-index entries directly usable by unlearning code, without per-token records. The paper's evidence that this matters is the NPO experiment: the same unlearning algorithm receives either ob's line-level forget","pith_inferences":["If provenance-recording becomes standard at ingestion, the whole unlearning pipeline can be reframed: the hard part shifts from 'which algorithm' to 'what metadata do we keep at write time'.","The token-index implies a path to per-token attribution, which would connect to token-level unlearning and fine-grained copyright claims.","The 42/23 percent split suggests forget-set precision may matter more than algorithm choice; a fair test would pit provenance-built sets against influence-function-based selection with the same budget.","In collaborative corpora, line-level revocation still deletes co-authored content; the paper's data shows 100% of records sit in multi-author sections, so true author-level compliance may require partial attribution weights."],"forward_implications":["Author revocation requests can be fulfilled at line or token granularity, eliminating dataset-level over-deletion for training corpora.","Unlearning algorithms like NPO can be fed provenance-built forget sets and show large quality gains; token-level bitmasks give a standard interface.","Provenance is auditable: an append-only operation log supports compliance checks, and the system needs no model access or GPU.","The same integration pattern works beyond text — git blame attribution for source code shows line-level provenance is domain-agnostic.","Because revocation is a reversible tag, compliance actions can be tested and rolled back before purge."],"fun_headline_variants":["Author-level provenance: unlearning over-deletion drops from 101x to 1.3x","Token-level provenance yields precise forget sets, boosting unlearning by 42%","Record-level provenance: 101x over-deletion -> 1.3x, 42% better unlearning","Record-level provenance slashes unlearning over-deletion to 1.3x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"OriginBlame's entire benefit depends on author identity being present when data first enters the pipeline — it propagates attribution but cannot discover it, and the paper concedes that incremental adoption is impossible once attribution has already been lost.","fun_headline_variants_meta":{"raw":{"variants":["Author-level provenance: unlearning over-deletion drops from 101x to 1.3x","Token-level provenance yields precise forget sets, boosting unlearning by 42%","Record-level provenance: 101x over-deletion -> 1.3x, 42% better unlearning","Record-level provenance slashes unlearning over-deletion to 1.3x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000887,"raw_usage":{"total_tokens":3637,"prompt_tokens":689,"completion_tokens":2948,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":2860}},"tokens_in":433,"tokens_out":2948,"duration_ms":21845,"temperature":1.0,"reasoning_tokens":2860,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:35:35.620552+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a real crawl or dataset passes through processing stages that strip authorship before ob.track() can run, the provenance chain is empty and the 1.3x over-deletion claim does not materialize. A direct experiment: take a corpus, build ob provenance, then run NPO with ob's forget set versus an oracle forget set of the same author's true lines — if unlearning metrics are statistically equal, the system's precision is not buying the stated improvement.","supporting_citations":[],"review_version":1}