Pith. sign in

REVIEW 4 major objections 3 minor 13 references

C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read C2LEVA claims that combining test-set renewal, contamination filtering, and data watermarking produces LLM rankings that are not inflated by memorized test data.

desk verdict Genuinely new active-prevention angle on a solid bilingual benchmark, but the 'contamination-free' claim outruns the evidence in the paper. read the letter →

arxiv 2412.04947 v3 pith:XAMRNRMK submitted 2024-12-06 cs.CL

Yanyang Li , Tin Long Wong , Cheung To Hung , Jianqiao Zhao , Duo Zheng , Ka Wai Liu , Michael R. Lyu , Liwei Wang This is my paper · ORCID
classification cs.CL
keywords datacontaminationLLMevaluationbenchmarkdetectionwatermarkingbilingualtestsetrenewal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

C2LEVA is a bilingual (English and Simplified Chinese) benchmark that the authors present as both comprehensive and contamination-free, covering 22 tasks across application assessment and ability evaluation (language, knowledge, reasoning, harms). The paper's central claim is that its systematic prevention strategy, which automates test-data renewal and adds data protection, yields trustworthy LLM rankings that are not inflated by models having memorized the test set. The authors support this by evaluating 15 open and proprietary models and by showing that C2LEVA's mean win rates track Chatbot Arena Elo with a Spearman correlation of 0.948. A careful reader would care because benchmark contamination currently undermines the reliability of LLM leaderboards, and this paper attempts a defense that combines passive renewal with active protection. The paper also documents that its active protection (data watermarking) degrades measured performance in most cases, which is a cost that future benchmarks will have to manage.

What carries the argument

The load-bearing mechanism is the two-part contamination prevention pipeline. Passive prevention automates test-set construction from fresh web content, rule-based generators, and LLM assistants, then filters candidate test instances with Min-K% (a per-instance contamination risk score computed from token probabilities of a representative model, Llama-3-8B) and augments scarce data with synonym substitution. Active prevention applies data-protection techniques: a CC BY-NC-ND license, ZipCrypto encryption, and sparse random-sequence watermarking designed to allow provable membership inference with a stated maximum 5% performance loss and p-value near 0.05. The paper's validation metric is the mean win rate across tasks, and its key external check is the Spearman correlation (0.948) between C2LEVA rankings and Chatbot Arena Elo.

What would settle it

Take the released C2LEVA test set, compute Min-K% scores with Llama-3-8B as the paper does, and then check a sample of instances against the training corpora or internal memorization behavior of several newly released LLMs; if a non-trivial fraction of instances that the filter scored as clean are memorized, the contamination-free guarantee fails. A cheaper version uses a deliberately contaminated instance that Min-K% scores as clean, trains a small model on it, and shows that the model then answers it correctly.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that contamination prevention for LLM evaluation can be made systematic by pairing passive prevention with active prevention. Passive prevention continuously renews test data through crawling, rule-based synthesis, and LLM-assisted generation, and filters risky instances using Min-K% token-probability contamination detection; active prevention makes the released data harder to misuse by licensing it, encrypting the archive, and watermarking a subset of test inputs so that unauthorized memorization can be proven. Applied across 22 tasks in two languages, this framework yields a benchmark whose model ranking correlates strongly with an independent, human-vote-based leaderboard, which the paper treats as evidence that the benchmark is both comprehensive and not contaminated. A secondary, cautionary finding is that watermarking measurably distorts evaluation results, with an average performance loss of about 11.59% across models in the fact completion task.

Load-bearing premise

The guarantee that every test instance is unseen by any evaluated model rests on a single filter, Min-K% scores computed with Llama-3-8B, catching all instances that appear in any model's training data, plus the assumption that newly crawled web text has not already been absorbed into those training corpora.

Editorial extensions

If this is right

  • If C2LEVA's contamination prevention works as claimed, current and future LLM rankings from the benchmark are not inflated by test-set memorization.
  • The benchmark provides a reusable 22-task, bilingual template for evaluating application skills and four ability dimensions without relying on stale test data.
  • The documented watermarking distortion implies that contamination-free evaluation carries a measurable accuracy cost that must be traded off against protection strength.
  • The strong correlation with Chatbot Arena Elo suggests that benchmark rankings derived from renewed, filtered test data can reproduce independent human-preference rankings.
  • Because the framework is automated, the leaderboard can be continuously updated as new models and new data appear, without rebuilding the benchmark from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The contamination-free claim is only as strong as the Min-K% detector's ability to generalize: if a future model was trained on data that the Llama-3-8B-based filter scored as clean, that model's C2LEVA score could still be inflated, so the benchmark should publish the detector's operating characteristics.
  • A natural stress test is to train a small model deliberately on a held-out portion of C2LEVA and see whether Min-K% flags those instances before release; if it does not, the renewal pipeline needs a stronger filter.
  • The watermarking distortion finding suggests that active prevention may be viable only for tasks where small performance shifts do not change ranking conclusions, or where stronger watermarks can be developed that preserve task semantics.
  • The two-language design invites extension to more languages and modalities, where contamination risk from web-scale training data is at least as severe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces C2LEVA, a bilingual (English/Chinese) LLM benchmark with 22 tasks spanning application assessment and ability evaluation. Its central contribution is a systematic contamination-prevention pipeline: passive prevention via crawling, rule-based synthesis, LLM assistants, contamination detection (Min-K%), and data augmentation; plus an active prevention layer using watermarking, licensing, and encryption. The authors evaluate 15 open-source and proprietary LLMs, report a leaderboard, and validate the benchmark by correlating mean win rates with Chatbot Arena Elo (Spearman rank correlation 0.948). The paper claims that C2LEVA provides a 'contamination-free' and trustworthy assessment.

Significance. If the contamination-free claim were rigorously supported, C2LEVA would be a valuable contribution: it covers tasks often missing from dynamic benchmarks (harms, knowledge, language), provides bilingual coverage, uses multiple prompt templates, and combines passive and active prevention in a principled framework. The large-scale evaluation of 15 models and the public leaderboard are also useful. However, the load-bearing assertion that the benchmark is contamination-free is not directly demonstrated. The contamination-detection filter is unvalidated, the watermarking design goal contradicts the measured distortion, and the Arena correlation does not test contamination. These gaps make the central claim currently overreaching.

major comments (4)
  1. [§3.3 (contamination detection)] The Min-K% filter is the only passive safeguard for crawled and LLM-generated data, but the paper reports no threshold value, no detection recall or precision, and no cross-model transfer experiments. Using Llama-3-8B as a 'representative model' is an assumption that needs empirical support. Please add a validation experiment with known contaminated samples (e.g., documents from pretraining corpora or simulated membership) and report the ROC/AUC of Min-K% for several of the evaluated models at the chosen threshold.
  2. [§3.3 vs. Table 3] The text states that watermarked test cases are 'designed to ensure a maximum performance loss of 5%', but Table 3 shows an average loss of 11.59% across models and a per-model Chinese loss up to 38.39% for Claude-3.5. This is an internal contradiction. The design guarantee must either be revised to match the measured distortion, or the watermarking strength must be recalibrated to meet the 5% target. As written, the active-prevention component itself introduces nontrivial evaluation distortion.
  3. [§4.3, Fig. 6] The Spearman correlation of 0.948 with Chatbot Arena Elo is high, but this only shows that C2LEVA ranks models similarly to a human-preference leaderboard. A contaminated benchmark can also achieve high rank correlation if contamination is diffuse or correlated with general capability. The sentence 'This supports the conclusion that C2LEVA is comprehensive and mitigates data contamination' is not justified by the evidence; the correlation is a consistency check, not a contamination test. Please either remove this claim or add a direct test, such as comparing model performance on instances flagged versus not flagged by the detector, or using a known contaminated subset.
  4. [Table 2 and §3.3 (coverage of prevention)] The systematic-prevention claim is not uniform across tasks. Contamination detection is not applied to reasoning-primitive or realistic-reasoning tasks, nor to copyright; active watermarking is applied to only 5% of one task (fact completion). At minimum, the paper should state plainly which tasks have which safeguards and qualify the benchmark-level claim accordingly. As written, the abstract's 'contamination-free tasks' overstates the coverage of the proposed pipeline.
minor comments (3)
  1. [Figures 3, 4, 6, 8, 9] These figures contain garbled placeholder strings (e.g., '/uni...' sequences) instead of readable labels and values; they must be regenerated with proper text rendering.
  2. [Appendix E.3] The human quality assessment of generated theses reports scores of 0.626 (Chinese) and 0.616 (English), but does not state the number of annotators or inter-annotator agreement; a single annotator is insufficient to establish reliability.
  3. [§4.3] The statement 'The Spearman's rank correlation is 0.948 with p < 0.05' should include the exact p-value and the number of paired observations used in the correlation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's validation is external and its contamination-prevention components are borrowed, not fitted to the paper's own outputs.

full rationale

The paper contains no derivation whose output is assumed as input. The contamination-prevention pipeline relies on external methods (Min-K% detection from Shi et al. 2023b, watermarking from Wei et al. 2024, and licensing/encryption per Jacovi et al. 2023), none of which are fitted to C2LEVA's own evaluation results. The central validation in Section 4.3 compares C2LEVA mean win rates to Chatbot Arena Elo, an external leaderboard, and the reported Spearman correlation is an independent check rather than a restatement of the paper's own outputs. The task taxonomy is adopted from the authors' own CLEVA paper (Li et al. 2023), but this is disclosed provenance and it is not used to forbid alternatives or to force a conclusion; comprehensiveness is additionally anchored to HELM and tested via external correlation. The 'contamination-free' claim rests on an unvalidated assumption that Llama-3-8B's Min-K% token-probability scores generalize to other models, and the measured watermark distortion in Table 3 contradicts the stated 5% design target; these are empirical or correctness weaknesses, not circularity. No equation, fitted parameter, or self-citation chain makes the claimed result equivalent to its inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the effectiveness of several external methods and on an unverified assumption that recency and Min-K% filtering guarantee non-contamination. The watermarking step introduces a new active defense but also a measurable performance distortion (average 11.59% loss) that is acknowledged in Section 4.3. No new entities are postulated.

free parameters (4)
  • Butter-finger augmentation probability = 0.01
    Chosen by hand for typo-fixing task to control input noise; not justified by data.
  • Sentiment analysis threshold = 5 (English), 2 (Chinese)
    Hand-chosen rating thresholds to split reviews into positive/negative; affects task difficulty.
  • Watermarking ratio = 5% of fact completion test data
    Chosen to limit performance distortion to claimed 5% while retaining detectability; not optimized or independently validated.
  • Min-K% contamination detection threshold = not reported
    The cutoff for excluding test cases as contaminated is not specified, making the passive prevention step un-replicable.
assumptions (4)
  • domain assumption Chatbot Arena Elo is a contamination-free ground-truth ranking
    Section 4.3 uses the Arena Elo correlation as evidence of C2LEVA's effectiveness, assuming Arena is resistant to contamination and manipulation.
  • domain assumption Min-K% with Llama-3-8B can detect contaminated test instances
    Section 3.3 relies on Min-K% token-probability scores from a single model to filter test data, assuming transferability to all other models.
  • domain assumption Newly crawled web data is unseen by LLMs
    The passive prevention assumes recency of crawl data implies absence from training corpora, but training data can include recent content and no comparison to actual corpora is made.
  • domain assumption Data watermarking provides provable membership inference
    Active prevention relies on Wei et al. (2024) watermarking to detect cheating, but the claimed statistical detection (p~0.05) is not demonstrated in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation." pith.science (2026). https://pith.science/paper/XAMRNRMK

@misc{pith2026241204947,
  author       = {Pith},
  title        = {Pith review of: C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAMRNRMK}},
  note         = {Machine review of arXiv:2412.04947}
}
abstract

Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C$^2$LEVA.

Figures

Figures reproduced from arXiv: 2412.04947 by the authors.

Figure 1
Figure 1. Contamination prevention overview. Solid [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The task taxonomy of C2LEVA. ing human values and knowledge. We propose data augmentation to mitigate data scarcity. Active prevention, though promising, is rarely explored (Jacovi et al., 2023). Our work pioneers active prevention with data protection (Wei et al., 2024; Zhao et al., 2024), and explores the effective￾ness of these algorithms and the trustworthiness of the corresponding evaluation results. Our findin… view at source ↗
Figure 3
Figure 3. The framework of C2LEVA for contamination prevention. Claude-3.5 Gemini-1.5 GPT-4o DeepSeek-V2 Yi-Large Llama-3-70B Qwen-Max GLM-4 Llama-3-8B InternLM2-20B GLM-4-9B Yi-1.5-9B Qwen2-7B Vicuna-13B Baichuan2-13B 0.0 0.2 0.4 0.6 0.8 English Claude-3.5 Gemini-1.5 GPT-4o Yi-Large DeepSeek-V2 Qwen-Max GLM-4 Llama-3-70B InternLM2-20B Llama-3-8B Yi-1.5-9B GLM-4-9B Qwen2-7B Baichuan2-13B Vicuna-13B 0.0 0.1 0.2 0.3 0.4 0.5 0.6… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The mean win rate of 15 models in 22 tasks of C [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Mean win rate comparison among models in different task groups. We choose the top-5 best-performing [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The English mean win rate of C 2LEVA scales linearly with style-controlled Chatbot Arena Elo. is the observed value. indicates the linear fit. ρ and e denote the Spearman’s ranking correlation and the root mean square error of the linear fit respectively. tasks such as…
Figure 7
Figure 7. Figure 7: Illustration of the first two rounds of the proposed thesis grouping stage. All theses that will participate in [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Human and automatic evaluation on narrative reiteration. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: The accuracy standard deviation of different models in different prompt templates from different tasks. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

  1. [1]

    In each cluster, theses whose cosine similarities to the centroid are of the lowest 25% portion are treated as outliers and discarded

    We perform K-Means clustering with cosine similarity on the thesis embeddings. In each cluster, theses whose cosine similarities to the centroid are of the lowest 25% portion are treated as outliers and discarded. We use fast- text (Bojanowski et al., 2017) to encode each thesis by taking the averaged word embed- dings as the thesis embedding

    work page 2017

  2. [2]

    The claim sug- gests

    We use the remaining theses as the initial candidates and repeat the following steps for grouping. The grouping stage stops once the maximum number of iterations is reached or no candidate theses are left to merge. (a) We divide the candidate theses into groups of two that have the highest co- sine similarity to each other in the thesis embedding space. W...

    work page 2022

  3. [3]

    open”) or has limited-access via APIs (“limited

    World · · · Answer: (give the option index only) 1 B Data Sources For reasoning tasks in C 2LEVA, we follow HELM (Liang et al., 2022) to synthesize bilingual data of reasoning primitive tasks and DyVal (Zhu et al., 2023a) for data of realistic reasoning tasks. Readers can refer to these papers for more imple- mentation details. Other tasks in C 2LEVA rely...

    work page 2023

  4. [5]

    In Findings of the Association for Computational Lin- guistics: EMNLP 2024, pages 16131–16161, Miami, Florida, USA

    VarBench: Robust language model bench- marking through dynamic variable perturbation. In Findings of the Association for Computational Lin- guistics: EMNLP 2024, pages 16131–16161, Miami, Florida, USA. Association for Computational Lin- guistics. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, ...

    work page 2024

  5. [6]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.CoRR, abs/2403.05530. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023a. Large language models can be easily distracted by irrelevant context. InInterna- tional Conference on Machine Learning, ICML 202...

    arXiv 2023

  6. [8]

    Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng

    Chatglm: A family of large language mod- els from GLM-130B to GLM-4 all tools.CoRR, abs/2406.12793. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Pre- training data detection for large language models: A divergence-based calibration method. InProceed- ings of the 2024 Conference on Empirical Methods in Natu...

    arXiv 2024

  7. [9]

    Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training

    Ghost sentence: A tool for everyday users to copyright data from large language models.CoRR, abs/2403.15740. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Adv...

    work page Pith review arXiv 2024

  8. [10]

    求你了，”她说，“请听我说。我是无辜的，我不应 该盗这儿来。 这一刻，特蕾西彻底明白了。她心里开始紧张起来了。“求你 了，

    with the probability of 0.01 to construct in- puts. We use Exact Match as the evaluation metric. A bilingual example is shown below: Chinese Example: 请修改以下文本里的错别字并输出修改好的文本。不包含 错别字的部分请逐字输出原文。 这一刻，特蕾西彻底明白了。她心里开始紧张起来 了。“求你了，”她说，“请听我说。我是无辜的，我不应 该盗这儿来。 这一刻，特蕾西彻底明白了。她心里开始紧张起来了。“求你 了，”她说，“请听我说。我是无辜的，我不应该到这儿来。” English Example: Please output this exact text, with...

    work page 2023

Show all 13 references
  1. [2020]

    InAd- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

    Language models are few-shot learners. InAd- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ben Buchanan, Andrew Lohn, Micah Musser, and Kate- rina Sedova. 2021....

    2020 arXiv

  2. [2021]

    Training verifiers to solve math word prob- lems.CoRR, abs/2110.14168. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingx- uan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guan...

    arXiv

  3. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, 2023, pages 186–217

    CLEV A: chinese language models evaluation platform. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, 2023, pages 186–217. Association for Computational Linguistics. Yucheng ...

    2023 arXiv

  4. [2024]

    Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures.CoRR, abs/2406.06565. OpenAI. 2023. GPT-4 technical report.CoRR, abs/2303.08774. Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. 2023. Proving test set contaminatio...

    2023 arXiv

  5. [4012]

    European Language Resources Association. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chin- may Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2024. ...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.