Pith. sign in

REVIEW 5 major objections 5 minor 40 references

CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CCI4.0 is a 35TB bilingual pretraining corpus whose 4.5 billion reasoning templates and model-based quality filters yield higher average benchmark scores than two curated baselines in 100B-token, 0.5B-parameter runs.

desk verdict The assembled bilingual corpus and staged CoT extraction are genuinely new, but the headline evaluation is confounded by language coverage, so the paper's central claim about pipeline quality is not established as written. read the letter →

arxiv 2506.07463 v1 pith:VEPYSK4K submitted 2025-06-09 cs.CL cs.AI

Guang Liu , Liangdong Wang , Jijie Li , Yang Yu , Yao Xu , Jiabei Chen , Yu Bai , Feng Liao
show 1 more author
Yonghua Lin
This is my paper
classification cs.CLcs.AI
keywords CCI4.0pretrainingdatasetbilingualcorpuschain-of-thoughtsynthesisdataqualityfilteringdeduplicationreasoningabilitylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CCI4.0 is a 35TB Chinese-English pretraining corpus built from Nemotron-CC English data, multiple Chinese web corpora, and math, wiki, arxiv, and code sources, then cleaned through two-stage deduplication, multi-classifier quality scoring, and per-domain loss filtering. The paper's central claim is that these cleaner signals improve downstream performance: at 100B training tokens with a 0.5B-parameter model, CCI4.0 averages 33.09 across Chinese and English benchmarks, ahead of 32.92 for Nemotron-CC-HQ and 31.64 for CCI3-HQ. The paired CCI4.0-M2-CoT component supplies 4.5 billion synthesized human thinking templates, totaling over 400B tokens, and the paper argues that mixing them in accelerates the model's preference for correct over incorrect reasoning chains. A sympathetic reader would care because the result suggests that structured, reasoning-oriented pretraining data can be produced automatically at scale rather than distilled from larger models, and that data quality choices matter as much as model size at small scale.

What carries the argument

The load-bearing mechanism is the staged CoT extraction pipeline. An instruction-tuned 32B language model first splits a document into semantically independent segments and summarizes each; the summaries are then consolidated into a coherent chain of thought, and a core question is derived, yielding triples of the form {core question, chain of thought, original document}. This is what converts ordinary pretraining text into 4.5 billion reasoning templates without distilling from a larger model. Supporting machinery includes three Chinese quality classifiers (two transformer-based classifiers plus a fastText classifier) trained on LLM-labeled data, and a domain-aware fluency filter that removes the highest-loss 0.5% of samples per domain at the 99.5th percentile.

What would settle it

Train two identical 0.5B models on the same 100B tokens, with and without the CoT component, then have both write free-form solutions to held-out GSM8K problems and compare final-answer accuracy; if the perplexity preference improves while final-answer accuracy does not, the claimed reasoning gain is an artifact of style, not competence.

Watch

Extended reading notes

Core claim

The paper claims that CCI4.0-M2-Base and CCI4.0-M2-CoT give language models cleaner training signals that transfer to knowledge and reasoning benchmarks, with the largest cross-baseline gains in Chinese tasks and in reflection-style evaluations. In controlled 100B-token runs of a 0.5B-parameter model, CCI4.0 beats CCI3-HQ on nearly every English benchmark, beats Nemotron-CC-HQ on the Chinese average (28.30 vs 27.29), and edges the overall average (33.09 vs 32.92). On adversarial reasoning-chain pairs from GSM8K and CRUXEval, models trained with CoT data show a faster rise in the fraction of samples where the correct chain receives lower perplexity than the incorrect one, which the paper interprets as reduced hallucination and earlier acquisition of reasoning ability. The paper also reports that CoT mixing improves some downstream tasks such as HellaSwag and TriviaQA but leaves other English tasks effectively flat or slightly lower.

Load-bearing premise

The reasoning evidence assumes that a small model's relative preference, measured by perplexity or how surprising it finds each chain, for the correct reasoning chain over the incorrect one reflects logical competence, not just familiarity with reasoning-style text.

Editorial extensions

If this is right

  • At 10B training tokens, CCI4.0 reaches about the same average score as Nemotron-CC-HQ at 30B tokens, so the corpus packs more learnable signal per token at small scale.
  • The Chinese average rises from 27.29 for Nemotron-CC-HQ to 28.30 for CCI4.0, with CMMLU improving from 26.84 to 28.92, because the mix deliberately includes about 20% Chinese data.
  • Mixing CCI4.0-M2-CoT accelerates the growth of the correct-chain perplexity preference on adversarial GSM8K and CRUXEval items, which the paper reads as earlier and more reliable reflection.
  • CoT mixing is not uniformly positive on downstream tasks: in the 10B-token comparison the English average is slightly lower with CoT included, so the claimed benefit is specific to reasoning signals rather than all benchmarks.
  • Removing the top 0.5% per-domain loss outliers from English web data alone improves CommonsenseQA from 23.91 to 25.55 and TriviaQA from 0.92 to 1.25, showing the filtering pipeline contributes independently of CoT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the perplexity-preference metric is right, the staged extraction method provides a teacher-free scaling path for reasoning data: the 800B-token MoE run in the appendix suggests the signal keeps improving with compute, without needing a frontier model to generate chains.
  • A natural extension is to test free-form answer accuracy on held-out math problems; the paper's own Table 4 leaves open whether the perplexity gain transfers to correct final answers at this model scale.
  • The 99.5th percentile loss cutoff is one operating point; sweeping it on a single domain would show whether the quality gain saturates or whether rare high-loss samples carry information.
  • Because the CoT triples are extracted from the same base corpus, a comparison against a control corpus with shuffled or surface-similar question-answer formatting would clarify whether the gain comes from reasoning structure or format familiarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces CCI4.0, a roughly 35 TB bilingual (Chinese/English) pretraining corpus built by combining a Chinese web corpus, Nemotron-CC English data, and additional math, code, wiki, arxiv, and book sources. The proposed pipeline includes two-stage deduplication, multiclassifier quality scoring, domain-aware fluency filtering, and LLM-based synthesis of 4.5 billion Chain-of-Thought (CoT) templates. To validate the dataset, the authors train Qwen2-0.5B models on 100B-token samples and report average benchmark scores of 33.09 for CCI4.0 versus 32.92 for Nemotron-CC-HQ and 31.64 for CCI3-HQ, claiming consistent downstream improvements, especially in Chinese tasks and in math/code reflection tasks. The CoT contribution is evaluated with a perplexity-preference metric on adversarial CoT datasets, with additional scaling experiments in the appendix.

Significance. If the empirical claims were fully supported, CCI4.0 would be a substantial bilingual pretraining resource, and the proposed pipeline would be a useful reference for data-centric LLM research. The paper documents a detailed and mostly reproducible processing flow, and the appendix contains scaling curves and ablations that are a positive feature. However, the main empirical comparison is confounded by language composition: CCI4.0 trails Nemotron-CC-HQ on the English average (37.89 vs. 38.55), and the overall win comes from Chinese benchmarks where the English-only baseline has no training data. The CoT reasoning evaluation is also not validated as a measure of logical competence rather than stylistic familiarity. These issues strike at the central claims, so the paper needs substantial additional experiments and analysis before it can be accepted.

major comments (5)
  1. [Section 5.1, Table 2] The headline comparison between CCI4.0 and Nemotron-CC-HQ is not a valid test of the curation pipeline because the corpora differ in language composition: Nemotron-CC-HQ is English-only, while CCI4.0 adds a Chinese web corpus. On the English benchmarks, CCI4.0 scores 37.89 versus 38.55 for Nemotron-CC-HQ, which contradicts the prose claim that CCI4.0 'consistently outperforms' the baselines. The overall average gain (33.09 vs. 32.92) is driven by CEval and CMMLU, where Nemotron-CC-HQ has essentially no training data and scores near floor (27.74 and 26.84). Since no ablation holds language constant while varying the pipeline, the contribution of deduplication, quality scoring, fluency filtering, and CoT synthesis is untested. Please add a language-matched control, such as an English-only CCI4.0 subset or a mixed corpus with the same Chinese/English ratio but without the CCI4.0 processing, and report per-language averages as primary outcomes.
  2. [Section 5.1, Table 2 prose] The sentence 'It significantly outperforms Nemotron-CC-HQ in both CEval (27.67 vs. 27.74) and CMMLU (28.92 vs. 26.84)' misreports the CEval numbers: 27.67 is lower than 27.74. This is a factual error in a key result and should be corrected, and all numerical claims in the prose should be systematically checked against the tables.
  3. [Section 5.2] The adapted PPL-preference evaluation is not established as a measure of reasoning ability for a 0.5B base model. Training on correct CoT chains increases the model's probability of CoT-style text, so a lower perplexity on correct versus incorrect CoT may reflect stylistic familiarity rather than logical competence. The observation in Table 4 that the CoT mix lowers the English downstream average (32.99 to 32.75) is consistent with this concern. Please validate the metric on held-out adversarially corrupted chains that preserve surface style, and/or show a positive correlation with downstream accuracy on tasks where CoT can be elicited.
  4. [Section 3.4 and Appendix A.6] The 99.5th percentile per-domain loss cutoff is an ad-hoc threshold whose effectiveness is validated only on a 10B-token English sample (A.6, Table 5). The paper applies it to all domains and to Chinese data without a sensitivity analysis or a per-domain ablation. Since the assumption that high-loss samples are noise rather than informative outliers is load-bearing for the fluency-filtering contribution, please provide evidence that the threshold is robust across domains and languages.
  5. [Section 5.3 and Table 4] Section 5.3 states that 'Results demonstrate that our synthetic CoT data contributes to performance gains in downstream tasks during model pretraining,' but Table 4 shows the English average drops from 32.99 to 32.75 with CoT, with mixed results on individual tasks (PIQA 62.79 to 61.26, CommonsenseQA 25.31 to 23.67, SIQA 40.99 to 40.43). The current claim is too strong and should be qualified to Chinese benchmarks or to the specific tasks that actually improve.
minor comments (5)
  1. [Abstract and Introduction] The abstract says the dataset 'occupies roughly 35 TB of disk space,' while the introduction describes a '35T tokens' dataset; these units should be reconciled.
  2. [Table 1] The checkmark and cross symbols in Table 1 are garbled in the text (e.g., '0.8✓×✓×'), making the comparison difficult to read; a properly formatted table is needed.
  3. [References] References [32] and [34] appear to be the same Nemotron-CC technical report and should be consolidated.
  4. [Section 5.1, bullet list] The bullet 'CCI4.0 at 10B tokens is comparable to Nemotron-CC-HQ at 30B tokens' is not backed by a visible figure or table; either include the supporting data or remove the claim.
  5. [Conclusion] The conclusion contains a typo, 'coppora,' which should be 'corpora.'

Circularity Check

1 steps flagged · score 6.0 of 10

The CoT 'reflection' evaluation is self-definitional: the PPL metric measures exactly the probability mass that CoT pretraining is trained to increase, so the claimed reasoning and hallucination gains reduce to the training objective.

  1. self definitional [Section 5.2 (Reasoning Abilities from CoT Dataset), Figure 3; Appendix A.3; abstract's 'math and code reflection' claim]
    "For each test sample containing both a correct and an incorrect CoT, we measure the model’s perplexity (PPL) on both CoTs. A sample is considered passed if the model assigns a lower PPL to the correct CoT compared to the incorrect one. ... We trained a 0.5B parameter dense language model on two distinct 100 billion-token datasets: one dataset included a mix of CoT data, while the other did not."

    The scored quantity is PPL on correct versus incorrect CoT. The treatment is adding correct CoT text to the pretraining corpus. Pretraining minimizes loss, i.e., PPL, over the training distribution, so the CoT-mixed model is directly optimized to lower PPL on exactly the correct CoT examples, while no incorrect CoT examples are added. The improved pass rate is therefore entailed by training-data inclusion rather than by independent reasoning competency. Calling this 'reasoning ability' and saying it 'significantly reduces the model’s tendency to hallucinate incorrect CoT examples' presents the training-signal effect as a measured outcome. Table 4 corroborates that the effect does not transfer to external English benchmarks (Average English 32.99 without CoT vs. 32.75 with CoT).

full rationale

The paper is mostly self-contained and empirical: the main dataset-quality comparison in Table 2 trains 0.5B models on 100B tokens and reports external benchmark scores against CCI3-HQ and Nemotron-CC-HQ; that is a legitimate external comparison. The language-composition imbalance between CCI4.0 and the English-only Nemotron-CC-HQ is a validity concern about the headline average, but it is not circularity. No load-bearing self-citation chain or imported uniqueness theorem is present: CCI3-HQ is an empirical baseline, and the adversarial CoT framework [1] is external to the authors. The one genuine circular step is the CoT reasoning evaluation in Section 5.2 (and its extension in Appendix A.3): the metric is PPL preference for correct CoT, and the intervention is training on correct CoT text, so the measured 'reasoning ability' is the training objective itself. Because that self-definitional metric underlies the paper's headline claim of improvements 'especially in math and code reflection tasks,' the circularity is partial but substantive, warranting a score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims depend on hand-chosen curation thresholds (loss cutoff, dedup parameters, language mix) and on assumptions that model-based scores track data usefulness and reasoning ability. None of these assumptions is externally benchmarked; each is validated only by the paper's own small-scale training runs, so the reader must accept them on the authors' say-so. The 4.5 billion CoT templates are synthesized data artifacts whose value is exactly what the paper tries to establish, and the supporting evidence is the disputed PPL proxy rather than an independent falsifiable handle.

free parameters (4)
  • Fluency filtering loss cutoff = 99.5th percentile, per domain
    Hand-chosen in section 3.4 and appendix A.1; the paper says lower percentiles would be too aggressive, so this threshold directly shapes the final corpus and the reported results.
  • Deduplication thresholds = length-threshold 800, min-doc-words 35
    Set by hand in section 3.2 to avoid over-deduplicating short texts; affects corpus diversity and downstream scores.
  • Chinese/English mix ratio = Chinese roughly 20% of the corpus
    Stated in section 5.1; the headline overall gain over Nemotron-CC-HQ is mostly driven by Chinese benchmarks, so this ratio is load-bearing for the main result.
  • Chinese sample length bounds = 100 to 20,000 characters; minimum 10 characters per line
    Preprocessing choices in section 3.1 balancing semantic richness and efficiency; they determine which Chinese documents enter the corpus.
assumptions (4)
  • domain assumption Perplexity is a valid within-domain proxy for text quality, so removing the top 0.5% loss samples improves the corpus.
    Invoked in section 3.4 and appendix A.6; the only validation is the 10B-token English filtering experiment, which is a small-scale self-check rather than an independent benchmark.
  • domain assumption LLM-generated labels from Qwen2.5-72B-Instruct, DeepSeek-V3, and GPT-4o are reliable ground truth for training the Chinese quality classifiers.
    Used in section 3.3 to build the classifiers; the classifiers inherit any bias or error in the labeling models, and agreement between two classifiers is treated as evidence of quality.
  • domain assumption Relative perplexity on correct versus incorrect CoT chains measures reasoning ability in small models.
    The entire CoT evaluation in section 5.2 depends on this, imported from the Essential AI framework [1]; it is not independently validated in this paper.
  • standard math Pretraining compute can be quoted as 6nT per data point.
    Standard scaling estimate taken from reference [1] in section 5.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/VEPYSK4K

@misc{pith2026250607463,
  author       = {Pith},
  title        = {Pith review of: CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEPYSK4K}},
  note         = {Machine review of arXiv:2506.07463}
}
abstract

We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered for superior data quality and diverse human-like reasoning trajectory. CCI4.0 occupies roughly $35$ TB of disk space and comprises two sub-datasets: CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a $5.2$ TB carefully curated Chinese web corpus, a $22.5$ TB English subset from Nemotron-CC, and diverse sources from math, wiki, arxiv, and code. Although these data are mostly sourced from well-processed datasets, the quality standards of various domains are dynamic and require extensive expert experience and labor to process. So, we propose a novel pipeline justifying data quality mainly based on models through two-stage deduplication, multiclassifier quality scoring, and domain-aware fluency filtering. We extract $4.5$ billion pieces of CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the distillation of CoT from larger models, our proposed staged CoT extraction exemplifies diverse reasoning patterns and significantly decreases the possibility of hallucination. Empirical evaluations demonstrate that LLMs pre-trained in CCI4.0 benefit from cleaner, more reliable training signals, yielding consistent improvements in downstream tasks, especially in math and code reflection tasks. Our results underscore the critical role of rigorous data curation and human thinking templates in advancing LLM performance, shedding some light on automatically processing pretraining corpora.

Figures

Figures reproduced from arXiv: 2506.07463 by the authors.

Figure 1
Figure 1. The figure shows the overall processing pipeline of our dataset. Our main processing flow [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance Comparison of Different Datasets Across Training Scales. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Reasoning ability scores on adversarial datasets for a 0.5B dense model trained with (w/ [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance Comparison of Models With and Without CoT Synthesis [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Loss values across domains and percentiles. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Reasoning ability scores on adversarial datasets for a 1.4B MoE model as pre-training [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Performance Comparison of Different Datasets Across Training Scales. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Performance Comparison of Models With and Without Loss-Based Filtering [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 22 canonical work pages

  1. [1]

    Rethinking reflection in pre-training, 2025

    Essential AI, :, Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Anthony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Phil...

    work page 2025

  2. [2]

    Coig-cqia: Quality is all you need for chinese instruction fine-tuning.ArXiv, abs/2403.18058, 2024

    Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Moore Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, and Ge Zhang. Coig-cqia: Quality is all you need for chinese instruction fine-tuning.ArXiv, abs/2403....

    arXiv 2024

  3. [3]

    Careful selection of knowledge to solve open book question answering

    Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. Careful selection of knowledge to solve open book question answering. In Anna Korhonen, David Traum, and Lluís Màrquez, editors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6120–6129, Florence, Italy, July 2019. Association for Computatio...

    work page 2019

  4. [4]

    CCI-Data [Data set]

    Beijing Academy of Artificial Intelligence. CCI-Data [Data set]. https://huggingface.co /datasets/BAAI/CCI-Data

    work page

  5. [5]

    CCI2-Data [Data set]

    Beijing Academy of Artificial Intelligence. CCI2-Data [Data set]. https://huggingface. co/datasets/BAAI/CCI2-Data

    work page

  6. [6]

    WuDaoCorporaText [Data set]

    Beijing Academy of Artificial Intelligence. WuDaoCorporaText [Data set]. https://data.b aai.ac.cn/datadetail/WuDaoCorporaText

    work page

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019

    2019

  8. [8]

    Andrei Z. Broder. On the resemblance and containment of documents.Proceedings. Compres- sion and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21–29, 1997

    work page 1997

Show all 40 references
  1. [9]

    Data- juicer: A one-stop data processing system for large language models.Companion of the 2024 International Conference on Management of Data, 2023

    Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data- juicer: A one-stop data processing system for large language models.Companion of the 2024 International Con...

    2024

  2. [10]

    Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023

    Jianghao Chen, Pu Jian, Tengxiao Xi, Dongyi Yi, Qianlong Du, Chenglin Ding, Guibo Zhu, Chengqing Zong, Jinqiao Wang, and Jiajun Zhang. Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023

    2023

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    2018

  4. [12]

    Unsupervised cross-lingual representation learning at scale.ArXiv, abs/1911.02116, 2019

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale.ArXiv, abs/1911.02116, 2019

    1911 arXiv

  5. [13]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025

  6. [14]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023

    2023

  7. [15]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  8. [16]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Chris Callison-Burch, Laurence Cohen, Esin Durmus, Ethan Fenoglio, Josh Firestone, Jordan Foster, Sam Gehman, Shachar Gretz, Kristen Halla- han, Dieuwke Hupkes, Nathan Lambert, Ron Le Bras, Zachary Levonian, Luca Lisi, Annika McMillan-Major...

    2020 arXiv

  9. [17]

    Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023

    Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023

    2023

  10. [18]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  11. [19]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InAdvances in Neural I...

    2023

  12. [20]

    Bag of tricks for efficient text classification.arXiv preprint arXiv:1607.01759, 2016

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification.arXiv preprint arXiv:1607.01759, 2016

    2016 arXiv

  13. [21]

    Deduplicating training data makes language models better, 2022

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better, 2022

    2022

  14. [22]

    Levesque, Ernest Davis, and Leora Morgenstern

    Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Represen- tation and Reasoning, KR’12, page 552–561. AAAI Press, 2012. 11

    2012

  15. [23]

    Cmmlu: Measuring massive multitask language understanding in chinese, 2024

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024

    2024

  16. [24]

    Datacomp-lm: In search of the next generation of training sets for language models, 2024

    Jeffrey Li and Alex Fang etc. Datacomp-lm: In search of the next generation of training sets for language models, 2024

    2024

  17. [25]

    Openhermes 2.5-zh: A partial chinese translation of openhermes-2.5, 2024

    Wenbo Pan. Openhermes 2.5-zh: A partial chinese translation of openhermes-2.5, 2024

    2024

  18. [26]

    Fineweb2: A sparkling update with 1000s of languages, December 2024

    Guilherme Penedo, Hynek Kydlíˇcek, Vinko Sabolˇcec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, December 2024

    2024

  19. [27]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116, 2023

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alibe, Quentin Chanu, Baptiste Launay, Jean-Baptiste Dehaene, and Hugo Touvron. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data on...

    2023 arXiv

  20. [28]

    Deduplicate Text Datasets

    Google Research. Deduplicate Text Datasets. https://github.com/google-research/d eduplicate-text-datasets, 2021. Accessed: 2025-05-16

    2021

  21. [29]

    Socialiqa: Com- monsense reasoning about social interactions, 2019

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions, 2019

    2019

  22. [30]

    Dolma: An open corpus of 3 trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159, 2024

    Luca Soldaini, Rodney Kin Lo, Wajdi Yazdan, Ahmed El-Kishky, Faisal Ladhak, Daniel Murray, Shaked Yom Din, Winston Li, Yingbo Liu, Yanai Elazar, Akshita Bhagia, Dirk Groeneveld, Tim Dettmers, Aleksandra Piktus, Nicola Cancedda, Allie De Lucia, Orr Katz, Leshem Choshen, Qiao Li...

    2024 arXiv

  23. [31]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2024

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2024

    2024

  24. [33]

    The fineweb datasets: Decanting the web for the finest text data at scale

    HuggingFace Team. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024

    2024 arXiv

  25. [34]

    Nemotron-4 340b technical report.arXiv preprint arXiv:2412.02595, 2024

    NVIDIA Team. Nemotron-4 340b technical report.arXiv preprint arXiv:2412.02595, 2024

    2024 arXiv

  26. [35]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  27. [36]

    Cci3.0-hq: a large-scale chinese dataset of high quality designed for pre-training large language models, 2024

    Liangdong Wang, Bo-Wen Zhang, Chengwei Wu, Hanyu Zhao, Xiaofeng Shi, Shuhao Gu, Jijie Li, Quanyue Ma, TengFei Pan, and Guang Liu. Cci3.0-hq: a large-scale chinese dataset of high quality designed for pre-training large language models, 2024

    2024

  28. [37]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  29. [38]

    Opencsg chinese corpus: A series of high-quality chinese datasets for llm training, 2025

    Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. Opencsg chinese corpus: A series of high-quality chinese datasets for llm training, 2025. 12

    2025

  30. [39]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  31. [40]

    Games" and

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. 13 A Appendix A.1 Loss values across domains Figure 5: Loss values across domains and percentiles. To systematically analyze the model’s perform...

    2019

  32. [41]

    Results demonstrate that the model trained with CoT data performs well in reasoning tasks like HellaSwag and reading comprehension tasks like TriviaQA. However, the performance gains brought by CoT data to pretrained models on downstream reasoning tasks are inconsistent, and h...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.