{"id":"1f827331-9706-4fec-b347-dcb246d7bbf7","arxiv_id":"2608.10678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sampled-BPE uses a small sample and BPE token statistics to estimate token-level pollution in web-scale Chinese corpora, revealing heavy and shifting adult content in Common Crawl.","lead":"This paper presents a low-cost method to audit Chinese web-scale corpora: sample a small slice, train a BPE tokenizer, and classify the most frequent tokens to estimate pollution. Applied to 11 open corpora and six Common Crawl snapshots, it finds widespread, uneven, and shifting adult and gambling content.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline ratios inherit classifier bias: GLM-4 category labels are validated only on GPT-vocabulary annotations, never on sampled web-corpus tokens, so the 68.72% Adult Content figure is not yet grounded.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the category classifier, fine-tuned and evaluated only on annotations from Zhang et al. (2025), is assumed to transfer without bias to arbitrary Chinese web corpora and Common Crawl snapshots. I agree that this is the most load-bearing point because all substantive conclusions in the paper are category-level statements. The sampling-accuracy experiments, by contrast, are internally coherent: they compare sampled profiles against a Full profile under the same pipeline, so they at least measure sampling noise conditional on the classifier. What they do not measure is labeling error, and labeling error is exactly what would change the headline '68.72% Adult Content' result. The proposed concrete test would settle the concern by constructing a held-out gold set from the actual corpora and measuring classifier precision/recall per category, then reweighting the reported ratios. If the classifier performs as well on this gold set as on its original test split, the headline results gain real support; if not, the audit numbers require re-estimation with a corrected or recalibrated classifier. I would keep the verdict CONDITIONAL, making this external validation the explicit condition for acceptance. No other concern I identified is more direct: streaming versus random sampling is addressed with a comparison in Appendix C, and the interpolation of error curves is a weaker but secondary issue that would not by itself invalidate the central method claim.","tokens_in":22019,"tokens_out":4597,"duration_ms":49612,"concrete_test":"Build a frequency-stratified gold set from the actual audit targets (at least OSCAR, mC4, and the 2026 CC snapshot): 500 tokens per corpus, each with the same Internet-search evidence used in Step 3, labeled independently by two fluent-Chinese annotators. Run the fine-tuned GLM-4 classifier on identical inputs; report per-category precision and recall and recompute Tables 1 and 2 with annotator labels. If Adult Content precision on the 2026 CC snapshot is below about 90%, or reweighted category ratios move by more than the claimed 4.25% relative error, the headline ratios are classifier-dependent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's substantive outputs are category-level pollution ratios (Tables 1 and 2), and every ratio is a sum of token frequencies weighted by labels from a GLM-4-32B classifier fine-tuned on expert annotations of GPT Chinese vocabularies from Zhang et al. (2025). The reported 97.32% accuracy is measured on a test split drawn from that same annotation source, not from the corpora being audited. There is no held-out validation on OSCAR, mC4, Common Crawl snapshots, or any other audit target. Because the classifier sees a token plus Internet search evidence rather than the token's actual corpus context, and because many Chinese web tokens are euphemistic or overloaded with benign meanings (e.g., '在线观看', '亚洲精品'), systematic over- or under-prediction of Adult Content is plausible. The claimed 4.25% sampling error in Figure 4 is conditional on the classifier's labels: it measures sampling noise relative to a Full profile that is itself classifier-labeled. Thus classifier transfer bias, if present, propagates directly into the headline claim that 68.72% of the 2026 Chinese CC snapshot is Adult Content, and into every corpus comparison. A secondary issue is that Tables 1 and 2 report no confidence intervals, so even sampling-only uncertainty is invisible; but the classifier is the load-bearing piece.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAMPLED-BPE, a token-level auditing pipeline that samples a small fraction of a web-scale Chinese corpus, trains a BPE tokenizer on the sample, maps high-frequency tokens to six content categories using a fine-tuned GLM-4-32B classifier with Internet search evidence, and aggregates the results into corpus-level pollution profiles. The authors validate the sampling step against a full-scan reference, reporting a 148.4x runtime speedup and 35.8x memory reduction with 4.25% relative error at 0.25% sampling, and then apply the pipeline to 11 open Chinese corpora and six Chinese Common Crawl snapshots (2021–2026). The audit reports large and uneven pollution across corpora (e.g., 83.38% in OSCAR vs. 0.50% in WuDao) and a 2026 Common Crawl snapshot with 79.52% total pollution and 68.72% Adult Content. The authors also release a hierarchical dataset of 630,684 Chinese web tokens organized into 92,972 trees with category, context, and explanation fields.","tokens_in":22306,"tokens_out":5869,"duration_ms":51879,"significance":"If the sampling estimator and the classifier transfer are both valid, the paper provides a practical and transferable method for low-cost, repeatable audits of web-scale corpora, and the concrete numbers for the Chinese open corpora and Common Crawl snapshots are a useful accountability contribution to LLM data governance. The release of a hierarchical token dataset with web-context evidence is a notable strength, as it makes the token-level claims reviewable and supports future cleaning research. The sampling-vs-full comparison on the reference corpus is a reasonable empirical check, and the authors are appropriately transparent about limitations. However, the significance is conditional: the category-level results inherit any bias in the classifier, and the claimed per-corpus error rates are interpolated from a single reference curve; these points must be addressed before the headline prevalence statements can be accepted.","major_comments":[{"comment":"The category labels that feed every category-level result come exclusively from a GLM-4-32B classifier fine-tuned and tested on expert annotations of GPT Chinese vocabularies from Zhang et al. (2025). Because the reported 97.32% accuracy is measured on a test split from that same annotation source, it does not measure performance on the tokens and contexts that actually occur in OSCAR, mC4, Common Crawl snapshots, or the other audited corpora. All prevalence estimates in Tables 1 and 2 are sums of token frequencies weighted by these labels, so systematic transfer bias in the classifier would propagate directly into the headline numbers, for example the 68.72% Adult Content figure for the 2026 Chinese Common Crawl snapshot, as well as into every cross-corpus and temporal comparison. The paper should validate the classifier on a held-out set of tokens sampled from the audited corpora, or otherwise quantify and bound label-induced uncertainty. This is a load-bearing gap because the audit's central outputs are the category ratios.","section":"Section 3.1, Step 3; Tables 1 and 2"},{"comment":"The per-corpus weighted relative errors in Table 3 are not measured on the audited corpora; they are obtained by log-linear interpolation over the full-token category weighted-error curve of a single reference corpus (the Common Crawl reference used in Figure 4). This assumes that the sampling-error curve transfers across corpora with different sizes, vocabularies, and pollution compositions. The reported values, such as 0.24% for mC4, are therefore model-based estimates, yet they are presented without any caveat or validation. To make the 'all below 4%' claim load-bearing for the corpus-specific results, the authors should either run a full-scan comparison on a second corpus or provide a more direct uncertainty estimate for each sampled corpus.","section":"Table 3 and Figure 4"},{"comment":"Neither table reports confidence intervals or any sampling-uncertainty measure for the pollution ratios. Given that the ratios range from 0.50% to 83.38% and that some corpora were audited at sampling rates as low as 0.19% (Table 3), point estimates without uncertainty bounds make it impossible to tell which pairwise differences or temporal shifts are statistically reliable. Bootstrapped confidence intervals over sampled tokens, or at least a statement of the sampling variance, should accompany the headline estimates.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The number of token records in the released dataset is given as 660k+ in the abstract but as 630,684 (630k+) in Section 6 and in the body abstract; please make these consistent.","section":"Abstract vs. Section 6"},{"comment":"The left-panel annotation in the main text reads 'token coverage remains:' and then gives a value, leaving the sentence grammatically incomplete; please complete it.","section":"Figure 4"},{"comment":"The sentence immediately before Table 2 is truncated ('In 2026, to-'); please restore the missing phrase.","section":"Section 5.1"},{"comment":"The definition of pollution percentages over tokens containing at least three Chinese characters is stated only briefly; because the 68.72% headline depends on this denominator, please add a sentence explaining the choice and its effect on comparability with token-level audits that use all tokens.","section":"Section 4"},{"comment":"'Web-scale' is defined as the TB range, yet Common Crawl is described as PB-scale and is also an audit target; please clarify whether PB-scale sources are within the stated scope of the sampling method.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the sampling methodology is sound, but the category-level conclusions rest on a classifier whose transfer to the audited corpora is unvalidated. I believe this is fixable with a focused validation experiment and uncertainty reporting, so I recommend major revision rather than rejection. I was not able to verify the GitHub repository contents from the manuscript, so the dataset and code claims rely on the authors' description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is simple and genuinely useful: sample a small slice of a terabyte-scale corpus, train a BPE tokenizer on that slice, count token frequencies, classify the top tokens with an LLM using internet evidence, and aggregate into category-level pollution percentages. At 0.25% sampling they report a 148x speedup and roughly 4% relative error against a full scan. That part holds up: the sampling-vs-full comparison in Figure 4 is a real empirical check, and the streaming sampler in Appendix C is a nice systems detail. The released hierarchical token dataset (630k+ tokens, 92k+ trees) is a tangible contribution.\n\nThe soft spot is the classifier, and it is the load-bearing piece for the audit numbers. Every category label—and therefore every headline ratio like 68.72% Adult Content in the 2026 Common Crawl snapshot—comes from a GLM-4-32B model fine-tuned on expert annotations of GPT vocabulary tokens from Zhang et al. (2025). The reported 97.32% accuracy is on a test split from that same annotation pool, not on tokens sampled from OSCAR, mC4, or Common Crawl. The classifier also sees token plus internet search results, not the token's actual corpus context. So the absolute pollution figures are not yet grounded: a transfer bias in the classifier propagates directly into Tables 1 and 2. The 4.25% sampling error is conditional on the classifier's labels, so it does not cover this bias. I also note there are no confidence intervals on any pollution ratio, and the per-corpus error estimates in Table 3 are log-linear interpolations from a single reference curve rather than measured per corpus. Reproducibility is incomplete—no commit hash, sparse hyperparameters.\n\nNone of this kills the method. The sampling methodology stands on its own; the classifier is a modular component. But the paper presents the audit results as its headline findings, and those results are only as strong as the unvalidated classifier. A referee should ask for a held-out validation on a sample of the audited corpora, or for a downgrade of the absolute claims.\n\nMy recommendation: engage with this paper. Send it to review, expecting the referee to push hard on the classifier transfer question. The method and dataset are worth serious attention; the headline numbers are not yet trustworthy.","headline":"A practical sampling-based token-level audit method with a real validation, but the headline pollution ratios rest on an unvalidated classifier transfer.","tokens_in":22844,"tokens_out":2685,"would_cite":true,"duration_ms":23811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token-level pollution of a web-scale Chinese corpus is estimable from a 0.25% sample: 148x faster, roughly 4.25% relative error, and 68.72% of the 2026 Chinese Common Crawl snapshot is adult content.","keywords":["Chinese web corpora","corpus pollution audit","BPE tokenizer","sampled statistics","Common Crawl","token-level analysis","web content safety","adult content detection"],"falsifier":"Hand-label 1,000 randomly drawn tokens from the 2026 Chinese Common Crawl snapshot using the same six categories and the same Internet-search evidence, then compare with the pipeline's labels; if agreement falls materially below the 97.32% test accuracy reported on the GPT-vocabulary split, the 79.52% pollution figure is classifier bias, not measurement.","tokens_in":21835,"feed_emoji":"📊","tokens_out":10553,"duration_ms":80539,"temperature":0.7,"pith_summary":"Web-scale Chinese corpora are too large and too volatile to scan in full, yet polluted content in them leaks into downstream language models. This paper claims that a small sampled subset is enough: train a byte-pair-encoding (BPE) tokenizer on the sample, count high-frequency tokens, and map them to content categories using Internet search evidence, and you obtain a usable token- and category-level pollution profile at a fraction of the cost. At a 0.25% sampling rate the pipeline keeps relative error near 4.25% for pollution categories while running 148.4$\\times$ faster and using 35.8$\\times$ less memory; the cost of auditing a 1TB corpus drops from months to hours. Applied to 11 open Chinese corpora and six Chinese Common Crawl snapshots (2021–2026), the audit reports pollution ranging from 0.50% to 83.38% across corpora, with 79.52% of the 2026 Common Crawl snapshot classified as polluted and 68.72% as adult content. The paper also releases a hierarchical dataset of 630,684 tokens, each with web context, category, and explanation, organized into 92,972 trees for review and tracing.","feed_headline":"Sampling cuts web-corpus pollution audits from months to hours","feed_subtitle":"A BPE on a 0.25% sample keeps token-level accuracy; 68.72% of 2026 Chinese Common Crawl is adult content.","key_machinery":"The load-bearing mechanism is the byte-pair-encoding tokenizer trained on the sampled subset: BPE repeatedly merges the most frequent adjacent byte pairs, so the learned vocabulary surfaces recurrent lexical patterns without needing a predefined keyword list. The full pipeline has four stages — streaming sampling in one sequential pass, BPE training and token counting, category mapping, and corpus profiling that aggregates token ratios into category prevalences. Category mapping assigns each token to one of six categories (Normal, Adult, Gambling, Gaming, Video, Anomalous) using a fine-tuned GLM-4-32B, an open-source Chinese language model, with Internet search results as contextual evidence; the classifier inherits its training labels from expert annotations of GPT Chinese vocabularies in Zhang et al. (2025). That inherited annotation set is what makes the category-level pollution ratios possible at all.","core_discovery":"The central claim is that corpus pollution can be audited at token level from a sampled subset rather than a full scan, because the statistics that matter for pollution survive aggressive downsampling. The authors show that at a 0.25% sampling rate the sampled corpus still recovers 76.83% of the full corpus's tokens, token ratios over the recovered tokens match closely (Spearman 86.88%, Pearson 99.93%), and category-level weighted relative error stays around 5%. They then use the pipeline as a measurement instrument: pollution is widespread but uneven across the 11 open corpora; 96.386% of the 106,671 polluted tokens appear in only one corpus; and the Chinese portion of Common Crawl is both highly polluted and temporally shifting, with Adult Content dominating recent snapshots and Online Gambling collapsing from 5.31% in 2021 to 0.21% in 2026. The authors frame these numbers as usable estimates for periodic upstream auditing rather than exact full-corpus measurements.","pith_inferences":["Because nothing in the pipeline except the classifier's training data is Chinese-specific, the same sampled-BPE recipe should extend to other languages and scripts, provided fresh expert annotations and per-language search evidence are collected.","The temporal signal — Online Gambling tokens falling from 5.31% to 0.21% between 2021 and 2026 while Adult Content surges — could be read as a low-cost social measurement of enforcement and platform dynamics on the Chinese web, not just a corpus-quality metric.","The cleanest stress test for the method would be a controlled injection experiment: splice a known set of gambling or adult phrases into a clean corpus, audit at 0.25% sampling, and measure how much of the injected material is recovered; this separates sampling error from classifier error."],"forward_implications":["Full scans of terabyte corpora become unnecessary for pollution monitoring: at sampling rates near 1% or below the same estimates are reachable in hours, so audits can be rerun against each new Common Crawl snapshot.","Static keyword lists and blocklists are structurally insufficient, since 96.386% of polluted tokens appear in only one of the 11 corpora; BPE-based discovery keeps surfacing new surface forms as tokenizers are retrained.","Cleaning pipelines reduce pollution but do not eliminate it — every curated corpus retains a measurable residual tail dominated by Anomalous tokens — so even 'clean' corpora still warrant periodic re-audit.","Pollution that arises from composing two individually normal tokens (e.g., 'Philippines' plus an abbreviation becoming a gambling brand) defeats substring matching, and the released token trees make such compositional cases traceable to their roots."],"supporting_citations":[{"why":"Supplies the byte-pair-encoding algorithm that the pipeline trains on sampled text to surface high-frequency tokens.","marker":"(Sennrich et al., 2016)"},{"why":"Supplies the expert-annotated Chinese polluted-token labels and category framework on which the GLM-4-32B classifier is fine-tuned and evaluated.","marker":"(Zhang et al., 2025)"},{"why":"Provides the upstream web archive whose Chinese WET-derived snapshots from 2021–2026 are audited in the temporal case study.","marker":"(Common Crawl, 2011)"},{"why":"Documents spam- and pornography-related Chinese tokens in GPT-4o's vocabulary, the downstream symptom that motivates upstream corpus auditing.","marker":"(Yang, 2024)"},{"why":"Reports Chinese gambling and lottery strings in Codex outputs, a second motivating symptom of corpus pollution.","marker":"(OpenAI Developer Community, 2026)"},{"why":"Defines the mC4 corpus whose Chinese portion is audited and reported 25.41% polluted, dominated by Online Gambling.","marker":"(Xue et al., 2021)"},{"why":"Defines the OSCAR corpus whose Chinese portion is audited and reported 83.38% polluted, the highest in the study.","marker":"(Abadji et al., 2022)"},{"why":"Defines the WanJuan corpus, the cleanest curated family at 0.75% pollution, used to support the claim that curation reduces pollution.","marker":"(He et al., 2023)"}],"fun_headline_variants":["Sampled BPE audit of Chinese web: 148x faster, 4.25% error","0.25% sample catches token pollution: 76.83% token recall","Chinese web pollution mapped: adult content spikes to 68.72% in 2026","Token-level audit: hours instead of months for web-scale Chinese corpora","Rapid audit: sampled tokens expose shifting pollution in Chinese web"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The category percentages all inherit the accuracy of a classifier trained and tested on annotations of GPT Chinese vocabulary tokens, and that classifier was never checked against annotations drawn from the web corpora it is asked to judge.","fun_headline_variants_meta":{"raw":{"variants":["Sampled BPE audit of Chinese web: 148x faster, 4.25% error","0.25% sample catches token pollution: 76.83% token recall","Chinese web pollution mapped: adult content spikes to 68.72% in 2026","Token-level audit: hours instead of months for web-scale Chinese corpora","Rapid audit: sampled tokens expose shifting pollution in Chinese web"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1863,"prompt_tokens":975,"completion_tokens":888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":782}},"tokens_in":591,"tokens_out":888,"duration_ms":7738,"temperature":1.0,"reasoning_tokens":782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:35:22.607373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-label 1,000 randomly drawn tokens from the 2026 Chinese Common Crawl snapshot using the same six categories and the same Internet-search evidence, then compare with the pipeline's labels; if agreement falls materially below the 97.32% test accuracy reported on the GPT-vocabulary split, the 79.52% pollution figure is classifier bias, not measurement.","supporting_citations":[{"cited_title":"2024 , month = may, howpublished =","cited_arxiv_id":null,"evidence_quote":"Documents spam- and pornography-related Chinese tokens in GPT-4o's vocabulary, the downstream symptom that motivates upstream corpus auditing."}],"review_version":1}