Pith. sign in

REVIEW 13 cited by

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.02595 v2 pith:ZKTL75HA submitted 2024-12-03 cs.CL

Dan Su , Kezhi Kong , Ying Lin , Joseph Jennings , Brandon Norick , Markus Kliegl , Mostofa Patwary , Mohammad Shoeybi
show 1 more author
Bryan Catanzaro
This is my paper · ORCID
classification cs.CL
keywords datadatasetdclmtokentokenshorizonmmlutraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

  2. OLMoASR: Open Models and Data for Training Robust Speech Recognition Models

    cs.SD 2025-08 conditional novelty 7.0 of 10

    An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.

  3. HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.

  4. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.

  5. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  6. Essential-Web v1.0: 24T tokens of organized web data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.

  7. Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    JQL trains small multilingual quality scorers from LLM judgments and human annotations, and filtering pretraining data with them improves downstream multilingual model performance over heuristic baselines.

  8. Multi-Head Attention Residuals

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Splitting the depth-routing query into per-subspace heads (a parameter-free reshape) improves Transformer validation loss at 100M–1B and mid-training at 8B.

  9. RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A deletion-only program refiner, trained on expert end-to-end edits converted via minimum edit distance, improves LLM pretraining data and downstream accuracy.

  10. CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models

    cs.CL 2025-06 reject novelty 5.0 of 10

    A new 35 TB bilingual pretraining dataset with 4.5 billion chain-of-thought templates is described, but the evidence for its benefits is marginal, confounded, and contradicted by the paper's own tables.

  11. MuLoCo: Muon is a practical inner optimizer for DiLoCo

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.

  12. Observation of momentum dependent charge density wave gap in EuTe4

    cond-mat.mes-hall 2025-08 unverdicted novelty 4.0 of 10

    EuTe4 shows a momentum-dependent charge density wave gap at the Fermi level, largest along Gamma-Y and smallest along Gamma-X, plus a low-temperature magnetic phase diagram near TN = 6.9 K.

  13. PARAM-1 BharatGen 2.9B Model

    cs.CL 2025-07 reject novelty 3.0 of 10

    A technical report on a 2.9B English-Hindi model whose headline evaluation numbers are internally inconsistent and whose promoted tokenizer was not used to train the final model.

Pith tools