REVIEW 3 major objections 5 minor 51 references
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Token-level pollution of a web-scale Chinese corpus is estimable from a 0.25% sample: 148x faster, roughly 4.25% relative error, and 68.72% of the 2026 Chinese Common Crawl snapshot is adult content.
desk verdict A practical sampling-based token-level audit method with a real validation, but the headline pollution ratios rest on an unvalidated classifier transfer. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the byte-pair-encoding tokenizer trained on the sampled subset: BPE repeatedly merges the most frequent adjacent byte pairs, so the learned vocabulary surfaces recurrent lexical patterns without needing a predefined keyword list. The full pipeline has four stages — streaming sampling in one sequential pass, BPE training and token counting, category mapping, and corpus profiling that aggregates token ratios into category prevalences. Category mapping assigns each token to one of six categories (Normal, Adult, Gambling, Gaming, Video, Anomalous) using a fine-tuned GLM-4-32B, an open-source Chinese language model, with Internet search results as contextual evidence; the classifier inherits its training labels from expert annotations of GPT Chinese vocabularies in Zhang et al. (2025). That inherited annotation set is what makes the category-level pollution ratios possible at all.
What would settle it
Hand-label 1,000 randomly drawn tokens from the 2026 Chinese Common Crawl snapshot using the same six categories and the same Internet-search evidence, then compare with the pipeline's labels; if agreement falls materially below the 97.32% test accuracy reported on the GPT-vocabulary split, the 79.52% pollution figure is classifier bias, not measurement.
Extended reading notes
Core claim
The central claim is that corpus pollution can be audited at token level from a sampled subset rather than a full scan, because the statistics that matter for pollution survive aggressive downsampling. The authors show that at a 0.25% sampling rate the sampled corpus still recovers 76.83% of the full corpus's tokens, token ratios over the recovered tokens match closely (Spearman 86.88%, Pearson 99.93%), and category-level weighted relative error stays around 5%. They then use the pipeline as a measurement instrument: pollution is widespread but uneven across the 11 open corpora; 96.386% of the 106,671 polluted tokens appear in only one corpus; and the Chinese portion of Common Crawl is both highly polluted and temporally shifting, with Adult Content dominating recent snapshots and Online Gambling collapsing from 5.31% in 2021 to 0.21% in 2026. The authors frame these numbers as usable estimates for periodic upstream auditing rather than exact full-corpus measurements.
Load-bearing premise
The category percentages all inherit the accuracy of a classifier trained and tested on annotations of GPT Chinese vocabulary tokens, and that classifier was never checked against annotations drawn from the web corpora it is asked to judge.
Editorial extensions
If this is right
- Full scans of terabyte corpora become unnecessary for pollution monitoring: at sampling rates near 1% or below the same estimates are reachable in hours, so audits can be rerun against each new Common Crawl snapshot.
- Static keyword lists and blocklists are structurally insufficient, since 96.386% of polluted tokens appear in only one of the 11 corpora; BPE-based discovery keeps surfacing new surface forms as tokenizers are retrained.
- Cleaning pipelines reduce pollution but do not eliminate it — every curated corpus retains a measurable residual tail dominated by Anomalous tokens — so even 'clean' corpora still warrant periodic re-audit.
- Pollution that arises from composing two individually normal tokens (e.g., 'Philippines' plus an abbreviation becoming a gambling brand) defeats substring matching, and the released token trees make such compositional cases traceable to their roots.
Reading between the lines
- Because nothing in the pipeline except the classifier's training data is Chinese-specific, the same sampled-BPE recipe should extend to other languages and scripts, provided fresh expert annotations and per-language search evidence are collected.
- The temporal signal — Online Gambling tokens falling from 5.31% to 0.21% between 2021 and 2026 while Adult Content surges — could be read as a low-cost social measurement of enforcement and platform dynamics on the Chinese web, not just a corpus-quality metric.
- The cleanest stress test for the method would be a controlled injection experiment: splice a known set of gambling or adult phrases into a clean corpus, audit at 0.25% sampling, and measure how much of the injected material is recovered; this separates sampling error from classifier error.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAMPLED-BPE, a token-level auditing pipeline that samples a small fraction of a web-scale Chinese corpus, trains a BPE tokenizer on the sample, maps high-frequency tokens to six content categories using a fine-tuned GLM-4-32B classifier with Internet search evidence, and aggregates the results into corpus-level pollution profiles. The authors validate the sampling step against a full-scan reference, reporting a 148.4x runtime speedup and 35.8x memory reduction with 4.25% relative error at 0.25% sampling, and then apply the pipeline to 11 open Chinese corpora and six Chinese Common Crawl snapshots (2021–2026). The audit reports large and uneven pollution across corpora (e.g., 83.38% in OSCAR vs. 0.50% in WuDao) and a 2026 Common Crawl snapshot with 79.52% total pollution and 68.72% Adult Content. The authors also release a hierarchical dataset of 630,684 Chinese web tokens organized into 92,972 trees with category, context, and explanation fields.
Significance. If the sampling estimator and the classifier transfer are both valid, the paper provides a practical and transferable method for low-cost, repeatable audits of web-scale corpora, and the concrete numbers for the Chinese open corpora and Common Crawl snapshots are a useful accountability contribution to LLM data governance. The release of a hierarchical token dataset with web-context evidence is a notable strength, as it makes the token-level claims reviewable and supports future cleaning research. The sampling-vs-full comparison on the reference corpus is a reasonable empirical check, and the authors are appropriately transparent about limitations. However, the significance is conditional: the category-level results inherit any bias in the classifier, and the claimed per-corpus error rates are interpolated from a single reference curve; these points must be addressed before the headline prevalence statements can be accepted.
major comments (3)
- [Section 3.1, Step 3; Tables 1 and 2] The category labels that feed every category-level result come exclusively from a GLM-4-32B classifier fine-tuned and tested on expert annotations of GPT Chinese vocabularies from Zhang et al. (2025). Because the reported 97.32% accuracy is measured on a test split from that same annotation source, it does not measure performance on the tokens and contexts that actually occur in OSCAR, mC4, Common Crawl snapshots, or the other audited corpora. All prevalence estimates in Tables 1 and 2 are sums of token frequencies weighted by these labels, so systematic transfer bias in the classifier would propagate directly into the headline numbers, for example the 68.72% Adult Content figure for the 2026 Chinese Common Crawl snapshot, as well as into every cross-corpus and temporal comparison. The paper should validate the classifier on a held-out set of tokens sampled from the audited corpora, or otherwise quantify and bound label-induced uncertainty. This is a load-bearing gap because the audit's central outputs are the category ratios.
- [Table 3 and Figure 4] The per-corpus weighted relative errors in Table 3 are not measured on the audited corpora; they are obtained by log-linear interpolation over the full-token category weighted-error curve of a single reference corpus (the Common Crawl reference used in Figure 4). This assumes that the sampling-error curve transfers across corpora with different sizes, vocabularies, and pollution compositions. The reported values, such as 0.24% for mC4, are therefore model-based estimates, yet they are presented without any caveat or validation. To make the 'all below 4%' claim load-bearing for the corpus-specific results, the authors should either run a full-scan comparison on a second corpus or provide a more direct uncertainty estimate for each sampled corpus.
- [Tables 1 and 2] Neither table reports confidence intervals or any sampling-uncertainty measure for the pollution ratios. Given that the ratios range from 0.50% to 83.38% and that some corpora were audited at sampling rates as low as 0.19% (Table 3), point estimates without uncertainty bounds make it impossible to tell which pairwise differences or temporal shifts are statistically reliable. Bootstrapped confidence intervals over sampled tokens, or at least a statement of the sampling variance, should accompany the headline estimates.
minor comments (5)
- [Abstract vs. Section 6] The number of token records in the released dataset is given as 660k+ in the abstract but as 630,684 (630k+) in Section 6 and in the body abstract; please make these consistent.
- [Figure 4] The left-panel annotation in the main text reads 'token coverage remains:' and then gives a value, leaving the sentence grammatically incomplete; please complete it.
- [Section 5.1] The sentence immediately before Table 2 is truncated ('In 2026, to-'); please restore the missing phrase.
- [Section 4] The definition of pollution percentages over tokens containing at least three Chinese characters is stated only briefly; because the 68.72% headline depends on this denominator, please add a sentence explaining the choice and its effect on comparability with token-level audits that use all tokens.
- [Section 2.1] 'Web-scale' is defined as the TB range, yet Common Crawl is described as PB-scale and is also an audit target; please clarify whether PB-scale sources are within the stated scope of the sampling method.
Circularity Check
Category-level pollution ratios rest on a load-bearing self-citation: the GLM-4 classifier is trained and validated only on GPT-vocabulary annotations from Zhang et al. (2025), which shares authors with this paper, and Tables 1-2 are direct sums of that classifier's labels.
-
self citation load bearing
[Section 3.1, Step 3 (category mapping); Section 4 normalization]
"The model is fine-tuned to predict a category from each token and its corresponding Internet search results, using a train/test split constructed from expert annotations of GPT Chinese vocabularies in Zhang et al. (2025). It achieves 97.32% classification accuracy on the test split after fine-tuning. ... Following (Zhang et al., 2025), percentages are computed over tokens containing at least three Chinese characters over six categories."
Every category-level pollution ratio in Tables 1 and 2 is a frequency-weighted sum of labels produced by this GLM-4 classifier. The classifier's only reported validation is a test split drawn from the same GPT-vocabulary annotation source as its training data, and that source is the authors' own prior paper (Zhang et al. 2025, sharing Qingjie Zhang, Tianwei Zhang, and Han Qiu). No held-out validation is performed on the audited corpora (OSCAR, mC4, Common Crawl snapshots, etc.), so the headline claim that 68.72% of the 2026 Chinese Common Crawl snapshot is Adult Content, and all corpus comparisons, reduce by construction to labels inherited from the authors' prior annotations.
full rationale
The sampling and cost claims are self-contained: Figure 4 compares sampled token statistics against the paper's own Full profile, and the 148.4x speedup and 35.8x memory reduction in Figure 5 are direct measurements on the authors' pipeline. The temporal and token-turnover analyses in Section 5 are likewise internal comparisons across snapshots. The one load-bearing external input is the category classifier's training and test data, which come entirely from Zhang et al. (2025), a prior paper with overlapping authorship. Because Tables 1 and 2 are weighted sums of that classifier's labels, the headline pollution ratios inherit the authors' prior annotation scheme and its potential domain shift from GPT vocabularies to arbitrary web corpora. This is not a full equivalence: the annotations were not made on the audited corpora, and the sampling, BPE, and cost contributions are independent. Still, the central empirical result that '68.72% of 2026 Chinese Common Crawl snapshot is Adult Content' is not validated against any external ground truth for web-corpus tokens, so the paper's main quantitative claim is supported by a self-citation chain rather than by independent evidence. Hence a score of 4: some self-citation that is load-bearing, while the central methodology retains independent content.
Assumptions & free parameters
free parameters (4)
- Sampling rate per corpus =
0.19% to 14.00% (Table 3)
- GLM-4-32B fine-tuned classifier weights =
not disclosed
- BPE merge budget and vocabulary size =
not disclosed
- Token length threshold =
at least three Chinese characters
assumptions (5)
- domain assumption BPE trained on a small sample approximates the full-corpus token distribution
- domain assumption Streaming (contiguous prefix) sampling is representative
- domain assumption The six categories and the Chinese-character token filter capture meaningful pollution
- domain assumption GLM-4-32B classifier trained on GPT-vocabulary annotations transfers to web corpora
- domain assumption February Common Crawl snapshots represent each year
Cite this review
Pith. "Pith review of Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics." pith.science (2026). https://pith.science/paper/BGSNM53S
@misc{pith2026260810678,
author = {Pith},
title = {Pith review of: Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics},
year = {2026},
howpublished = {\url{https://pith.science/paper/BGSNM53S}},
note = {Machine review of arXiv:2608.10678}
}
abstract
Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes full scan costly; (2) prior analyses are often too coarse to expose token-level pollution; (3) Chinese web pollution is implicit and rapidly changing. We propose Sampled-BPE, a lightweight token-level auditing pipeline that sample a small subset and train BPE tokenizer to surface polluted tokens. Experiments show that Sampled-BPE preserves usable estimates while substantially reducing runtime and memory: a 148.4 $\times$ speedup and a 35.8 $\times$ memory reduction induce only 4.25% relative error for pollution categories. We apply the pipeline to 11 open Chinese corpora and 6 Chinese Common Crawl snapshots from 2021 to 2026. The audit reveals widespread but uneven pollution across open corpora, as well as highly polluted and temporally shifting Chinese web content. We further release a hierarchical Chinese web token dataset with 660k+ token records, each with web context, category, and explanation fields, organized as trees to support review and tracing of pollution.
Figures
Figures from the paper (21 more)
Reference graph
Works this paper leans on
-
[1]
Journal of machine learning research , volume=
Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=
-
[2]
arXiv preprint arXiv:2101.00027 , year=
The pile: An 800gb dataset of diverse text for language modeling , author=. arXiv preprint arXiv:2101.00027 , year=
-
[3]
International conference on machine learning , pages=
Pythia: A suite for analyzing large language models across training and scaling , author=. International conference on machine learning , pages=. 2023 , organization=
2023
-
[4]
arXiv preprint arXiv:2306.01116 , year=
The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only , author=. arXiv preprint arXiv:2306.01116 , year=
-
[5]
arXiv preprint arXiv:2309.10818 , year=
Slimpajama-dc: Understanding data combinations for llm training , author=. arXiv preprint arXiv:2309.10818 , year=
-
[6]
arXiv preprint arXiv:2402.19173 , year=
Starcoder 2 and the stack v2: The next generation , author=. arXiv preprint arXiv:2402.19173 , year=
-
[7]
Advances in neural information processing systems , volume=
Redpajama: an open dataset for training large language models , author=. Advances in neural information processing systems , volume=
-
[8]
Advances in Neural Information Processing Systems , volume=
The fineweb datasets: Decanting the web for the finest text data at scale , author=. Advances in Neural Information Processing Systems , volume=
Show all 51 references
-
[9]
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
INFINI-GRAM MINI: Exact n-gram Search at the Internet Scale with FM-Index , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
2025
-
[10]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Dolma: An open corpus of three trillion tokens for language model pretraining research , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[11]
Common Crawl , year =
-
[12]
arXiv preprint arXiv:2308.10755 , year=
Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models , author=. arXiv preprint arXiv:2308.10755 , year=
-
[13]
Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages=
Towards a cleaner document-oriented multilingual crawled corpus , author=. Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages=
-
[14]
2023 , howpublished =
2023
-
[15]
Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies , pages=
mT5: A massively multilingual pre-trained text-to-text transformer , author=. Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies , pages=
2021
-
[16]
2024 , howpublished =
2024
-
[17]
Mono-and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models , author=
HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono-and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models , author=. arXiv preprint arXiv:2511.01066 , year=
-
[18]
2025 , howpublished =
2025
-
[19]
and Nguyen, Thien Huu
Nguyen, Thuat and Nguyen, Chien Van and Lai, Viet Dac and Man, Hieu and Ngo, Nghia Trung and Dernoncourt, Franck and Rossi, Ryan A. and Nguyen, Thien Huu. C ultura X : A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. Proceedings of the ...
2024
-
[20]
arXiv preprint arXiv:2311.01149 , year=
Chinesewebtext: Large-scale high-quality Chinese web text extracted with effective evaluation model , author=. arXiv preprint arXiv:2311.01149 , year=
-
[21]
Advances in Neural Information Processing Systems , volume=
The bigscience roots corpus: A 1.6 tb composite multilingual dataset , author=. Advances in Neural Information Processing Systems , volume=
-
[22]
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=
The ROOTS search tool: Data transparency for LLMs , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=
-
[23]
arXiv preprint arXiv:2211.05100 , year=
Bloom: A 176b-parameter open-access multilingual language model , author=. arXiv preprint arXiv:2211.05100 , year=
-
[24]
arXiv preprint arXiv:2404.04167 , year=
Chinese tiny llm: Pretraining a chinese-centric large language model , author=. arXiv preprint arXiv:2404.04167 , year=
-
[25]
2024 , eprint=
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model , author=. 2024 , eprint=
2024
-
[26]
arXiv preprint arXiv:2310.19341 , year=
Skywork: A more open bilingual foundation model , author=. arXiv preprint arXiv:2310.19341 , year=
-
[27]
0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models , author=
CCI3. 0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models , author=. arXiv preprint arXiv:2410.18505 , year=
-
[28]
Ai Open , volume=
Wudaocorpora: A super large-scale chinese corpora for pre-training language models , author=. Ai Open , volume=. 2021 , publisher=
2021
-
[29]
Transactions of the Association for Computational Linguistics , volume=
Data statements for natural language processing: Toward mitigating system bias and enabling better science , author=. Transactions of the Association for Computational Linguistics , volume=. 2018 , publisher=
2018
-
[30]
Communications of the ACM , volume=
Datasheets for datasets , author=. Communications of the ACM , volume=. 2021 , publisher=
2021
-
[31]
Proceedings of the twelfth language resources and evaluation conference , pages=
CCNet: Extracting high quality monolingual datasets from web crawl data , author=. Proceedings of the twelfth language resources and evaluation conference , pages=
-
[32]
Proceedings of the 2021 conference on empirical methods in natural language processing , pages=
Documenting large webtext corpora: A case study on the colossal clean crawled corpus , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=
2021
-
[33]
Transactions of the Association for Computational Linguistics , volume=
Quality at a glance: An audit of web-crawled multilingual datasets , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=
2022
-
[34]
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Deduplicating training data makes language models better , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[35]
arXiv preprint arXiv:2303.09540 , year=
Semdedup: Data-efficient learning at web-scale through semantic deduplication , author=. arXiv preprint arXiv:2303.09540 , year=
-
[36]
Findings of the association for computational linguistics: EMNLP 2020 , pages=
Realtoxicityprompts: Evaluating neural toxic degeneration in language models , author=. Findings of the association for computational linguistics: EMNLP 2020 , pages=
2020
-
[37]
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
Speculating LLMs’ Chinese Training Data Pollution from Their Tokens , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
2025
-
[38]
Proceedings of the International AAAI Conference on Web and Social Media , volume=
Practical Datasets for Analyzing LLM Corpora Derived from Common Crawl , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume=
-
[39]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
TiC-LM: A web-scale benchmark for time-continual LLM pretraining , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[40]
Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=
GLUE: A multi-task benchmark and analysis platform for natural language understanding , author=. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=
2018
-
[41]
International conference on machine learning , pages=
The flan collection: Designing data and methods for effective instruction tuning , author=. International conference on machine learning , pages=. 2023 , organization=
2023
-
[42]
Advances in neural information processing systems , volume=
Language models are few-shot learners , author=. Advances in neural information processing systems , volume=
-
[43]
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , pages=
Languagecrawl: A generic tool for building language models upon common-crawl , author=. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , pages=
-
[44]
2024 , month = may, howpublished =
Yang, Zeyi , title =. 2024 , month = may, howpublished =
2024
-
[45]
2026 , month = jan, howpublished =
Chinese Gambling Characters in. 2026 , month = jan, howpublished =
2026
-
[46]
The C Users Journal , volume=
A new algorithm for data compression , author=. The C Users Journal , volume=. 1994 , publisher=
1994
-
[47]
Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=
Neural machine translation of rare words with subword units , author=. Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=
-
[48]
ACM Transactions on Mathematical Software (TOMS) , volume=
Random sampling with a reservoir , author=. ACM Transactions on Mathematical Software (TOMS) , volume=. 1985 , publisher=
1985
-
[49]
Marine pollution bulletin , volume=
Marine water quality monitoring: A review , author=. Marine pollution bulletin , volume=. 2013 , publisher=
2013
-
[50]
PloS one , volume=
Neologisms are epidemic: Modeling the life cycle of neologisms in China 2008-2016 , author=. PloS one , volume=. 2021 , publisher=
2008
-
[51]
PloS one , volume=
Hate speech detection: Challenges and solutions , author=. PloS one , volume=. 2019 , publisher=
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.