BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

Aldo Carranza; Alex Fang; Alvin Deng; Amro Abbas; Ari Morcos; Bogdan Gaza; Brett Larsen; Charvi Bannur; Christina Baek; Cody Blakeney

REVIEW 4 major objections 3 minor 4 cited by

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that BeyondWeb, a jointly tuned synthetic-data generation framework, outperforms existing synthetic and web corpora for LLM pretraining and lets a smaller model beat a larger one at equal token budgets.

desk verdict BeyondWeb's abstract promises large synthetic-data gains, but without full methodology it's an unverified claim that deserves a careful referee rather than belief or citation. read the letter →

arxiv 2508.10975 v2 pith:RBUMVUJZ submitted 2025-08-14 cs.LG cs.CL

DatologyAI: Pratyush Maini , Vineeth Dorna , Parth Doshi , Aldo Carranza , Fan Pan , Jack Urbanek , Paul Burstein , Alex Fang

show 21 more authors

Alvin Deng Amro Abbas Brett Larsen Cody Blakeney Charvi Bannur Christina Baek Darren Teh David Schwab Haakon Mongstad Haoli Yin Josh Wills Kaleigh Mentzer Luke Merrick Ricardo Monti Rishabh Adiga Siddharth Joshi Spandan Das Zhengping Wang Bogdan Gaza Ari Morcos Matthew Leavitt

This is my paper · ORCID

classification cs.LGcs.CL

keywords syntheticpretrainingdatalargelanguagemodelswallscalingrephrasingweb-scaledatasetsBeyondWebbenchmarkevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the quality of synthetic pretraining data is governed by many jointly tuned design choices, not by a single trick. It introduces BeyondWeb, a synthetic-data generation framework whose output, when used to pretrain a 3-billion-parameter model for 180 billion tokens, outperforms the same-budget model trained on open web data by up to 5.1 percentage points averaged over 14 benchmarks. It also reports that this synthetic dataset trains faster than web data and lets a 3B model beat an 8B model trained on an existing synthetic dataset at the same token count. The paper's broader assertion is that naive synthetic-data scaling can be costly and yield little, while careful joint optimization of what to rephrase, how, and with which model family is what unlocks large gains. A sympathetic reader should care because this targets the 'data wall' in LLM pretraining and claims a practical path past it.

What carries the argument

BeyondWeb itself, a synthetic-data generation framework, is the central object. It is best understood as a pipeline whose inputs are web documents and a generator model, and whose output is a synthetic pretraining corpus; the pipeline's work is the joint optimization of three levers: selection (which documents to rephrase), transformation (how to rephrase them, including style and instruction), and generation scaling (model size and family used to synthesize the text).

What would settle it

Train the same three data sources on the same model and token budget in a held-out comparison with a different evaluation suite, more seeds, and no tuning of the synthetic pipeline to those particular benchmarks; if BeyondWeb no longer beats Cosmopedia and Nemotron-Synth on average, the central claim fails.

Watch

Extended reading notes

Core claim

BeyondWeb is a framework for generating synthetic pretraining data at trillion-token scale rather than a single fixed dataset. The central discovery claimed is that synthetic-data quality for pretraining is not a property of the generator alone; it emerges from jointly optimizing several factors: which web documents are selected for rephrasing, the instruction or style used for rephrasing, the scale of the resulting corpus, and the size and family of the model used to generate the data. The paper reports that when these factors are tuned together, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same budget on Cosmopedia, and outperforms Cosmopedia and

Load-bearing premise

The load-bearing premise is that the 14 benchmarks, averaged together, faithfully measure general LLM capability, and that every comparison used identical token budgets and training configurations, so the reported gaps isolate the synthetic-data generation method rather than evaluation quirks or extra compute.

Editorial extensions

If this is right

A 3B model trained for 180B tokens on BeyondWeb can beat an 8B model trained on Cosmopedia for the same token budget, implying that data choice can offset model-size advantage at equal token count.
BeyondWeb data trains 7.7x faster than open web data and 2.7x faster than Nemotron-Synth, so the reported gains are not bought with extra wall-clock training compute.
Averaged over 14 benchmarks, BeyondWeb leads Cosmopedia by up to 5.1pp and Nemotron-Synth by up to 2.6pp, indicating the synthetic-data generation method, not ordinary data abundance, drives the gain.
Naive synthetic-data generation can yield only modest improvements at high cost; the best outcomes require joint optimization of selection, rephrasing, and generator scale, making synthetic-data quality a multi-factor engineering problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: If the 14-benchmark average is representative, then the 'data wall' is less a wall than a budget trade-off; organizations with fixed training compute may get more capability by spending on high-quality synthetic generation rather than on larger model parameters.
Inference: Because the paper emphasizes model size and family as data-quality levers, a direct consequence is that synthetic-data recipes optimized with one generator model may not transfer to another; practitioners should re-tune the pipeline when the generator changes.
Inference: A testable extension is to use BeyondWeb's synthetic corpus as an auxiliary dataset for continual pretraining or domain adaptation, steering the 'what to rephrase' selection toward target benchmarks.
Inference: The reported speedups suggest a cost model in which the real price of synthetic data is generation compute plus the search over choices; if the selection and rephrasing rules are released, independent labs could reproduce the pipeline and measure sensitivity to each lever.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You're going to see this paper cited a lot if its numbers hold. BeyondWeb, a synthetic-data pipeline from DatologyAI, claims to beat Cosmopedia by up to 5.1pp averaged over 14 benchmarks, and to train 7.7x faster than open web data. It also claims a 3B model trained on 180B tokens of BeyondWeb beats an 8B model trained on the same token budget of Cosmopedia. Those are transformative if true. But here's the thing: I only have the abstract. The full text, tables, figures, and methodology are not in front of me, so I can't verify any of it.

What the paper does well: the abstract is honest about complexity. It explicitly says there's no silver bullet, that naive approaches yield modest gains at great cost, and that joint optimization is hard. That's a sane, non-hyped framing. The comparisons to state-of-the-art datasets (Cosmopedia, Nemotron-Synth) are the right baselines, not strawmen. And the claimed speedups, if real, are large enough to matter for anyone doing large-scale pretraining.

The soft spots are exactly what you'd expect from an abstract-only submission. The 5.1pp and 2.6pp gaps are reported without error bars or per-benchmark breakdowns. At 3B scale, run-to-run noise can be a couple of points, so a 2.6pp average gap could be within variance unless they used multiple seeds. The comparison to an 8B model is only meaningful if the token budgets, model architectures, and hyperparameters are matched otherwise. The 14 benchmark suite could be overfit or cherry-picked; we don't know if it was held out from the synthetic data generation process. None of these concerns are accusations; they are the minimum evidence needed to move from unverified to credible.

The citation pattern and self-citations aren't something I can check from the abstract. I'd want to see whether the baselines were trained by the same team with identical budgets, and whether the code and data are released for reproducibility.

My take: this paper deserves peer review, not desk rejection, because the claims are high-impact and the abstract is plausible. But I wouldn't cite it or build on it until I've seen the full methodology. For a reading group, I'd wait until the paper is available in full; the abstract alone is too thin for a productive session. Hand it to a careful referee who can stress-test the baseline fairness and benchmark selection.

Referee Report

4 major / 3 minor

Summary. The paper proposes BeyondWeb, a synthetic-data generation framework for pretraining LLMs. Based on the abstract, the authors claim that models trained on BeyondWeb outperform those trained on established synthetic datasets, Cosmopedia and Nemotron-Synth, by up to 5.1pp and 2.6pp, respectively, when averaged over 14 benchmarks. They further report 7.7x and 2.7x faster training than open web data and Nemotron-Synth, and that a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained with the same budget on Cosmopedia. The abstract also advertises several qualitative insights about synthetic-data generation. The full manuscript was not available, so this review is limited to the abstract.

Significance. If the reported results are correct and reproducible, the contributions are potentially significant for the pretraining community. The core claims—that synthetic data can substantially outperform web-scale corpora and that data efficiency can compensate for model size—would strengthen the case for synthetic-data-centric pretraining pipelines. The claimed speedups also have practical implications for reducing training cost. However, the abstract alone provides no methodological detail, ablations, baseline specifications, or statistical analysis. The value of the work therefore cannot be assessed until the full experimental protocol is available. The qualitative insights mentioned in the abstract are only enumerated, not substantiated.

major comments (4)

[Abstract (claims)] All headline numbers (5.1pp, 2.6pp, 7.7x, 2.7x, 3B-vs-8B) are not accompanied by any description of the evaluation setup. The central premise is that these gaps isolate the effect of the synthetic-data generation method, but the abstract does not state whether baselines used identical model architectures, token budgets, optimizer settings, or data mixtures. Without this parity, the claims cannot be interpreted. The full manuscript must provide a detailed training configuration for each baseline and demonstrate that the comparisons are compute- and hyperparameter-matched.
[Abstract (benchmark suite)] The abstract reports an average over 14 benchmark evaluations but does not list the benchmarks, explain how they were chosen, or address potential contamination. In particular, it is not shown that the synthetic-data generation pipeline (e.g., prompt sources, filtering criteria) did not have access to the evaluation benchmarks or their distributions. The full paper must specify the benchmark suite, its public availability, and the contamination-mitigation measures, ideally with per-benchmark results rather than only an average.
[Abstract (statistical reliability)] No error bars, confidence intervals, or number of random seeds are reported. At 3B scale and a 180B token budget, evaluation variance across runs is known to be nontrivial; a 2.6pp average gap over 14 benchmarks could be within run-to-run noise unless multiple seeds and significance tests are used. The full manuscript should provide standard deviations across independent runs and, where applicable, per-benchmark significance tests.
[Abstract (insights section)] The abstract promises 'several insights' on what drives synthetic-data benefits, which data to rephrase and how, and the impact of model size/family on data quality, but gives no evidence or ablation. These are load-bearing for the practical guidance the paper claims to offer. The manuscript needs to include ablations that support each insight and report their magnitudes, not merely assert them.

minor comments (3)

[Abstract (presentation)] The phrase 'up to 5.1pp' is ambiguous: it might mean the maximum improvement on a single benchmark or a maximum over some subsets, while the text also says 'when averaged across a suite of 14 benchmark evaluations.' Clarify whether the 5.1pp and 2.6pp figures are averages or maxima.
[Abstract (related works)] Cosmopedia and Nemotron-CC are named without citations or version information. The full paper should cite the exact datasets and versions used as baselines.
[Abstract (language)] Terms like 'data wall' and 'silver bullet' are informal; consider more precise wording, but this does not affect technical correctness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected in abstract; claims are empirical comparisons with no derivation chain to reduce.

full rationale

This is an abstract-only review. The abstract contains no equations, no fitted parameters renamed as predictions, no self-citations, and no definitional relationships that could force a result by construction. The central claims are empirical comparisons: BeyondWeb outperforms Cosmopedia and Nemotron-Synth on a suite of 14 benchmarks, trains faster, and enables a 3B/180B model to beat an 8B/Cosmopedia model. These are statements about measured performance against external baselines, not derivations from assumed premises. There is no visible 'derivation chain' whose output could be equivalent to its input. The reader's concern about post hoc benchmark selection or unfair baselines is a validity/experimental-design risk, not circularity, and the instructions explicitly prohibit speculation about intent or 'feels circular' judgments. No specific step can be quoted as circular because no such step appears in the abstract. Therefore the paper merits a score of 0: no significant circularity found on the available text.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review: no free parameters, axioms, or invented entities can be identified from the abstract. The full text is required to audit these.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining." pith.science (2026). https://pith.science/paper/RBUMVUJZ

@misc{pith2026250810975,
  author       = {Pith},
  title        = {Pith review of: BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBUMVUJZ}},
  note         = {Machine review of arXiv:2508.10975}
}

read the original abstract

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a promising paradigm for pushing the frontier of performance. Despite this, the factors affecting synthetic data quality remain poorly understood. In this work, we introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining. BeyondWeb significantly extends the capabilities of traditional web-scale datasets, outperforming state-of-the-art synthetic pretraining datasets such as Cosmopedia and Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1 percentage points (pp) and 2.6pp, respectively, when averaged across a suite of 14 benchmark evaluations. It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia. We also present several insights from BeyondWeb on synthetic data for pretraining: what drives its benefits, which data to rephrase and how, and the impact of model size and family on data quality. Overall, our work shows that there's no silver bullet for generating high-quality synthetic pretraining data. The best outcomes require jointly optimizing many factors, a challenging task that requires rigorous science and practical expertise. Naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield transformative improvements, as exemplified by BeyondWeb.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Bridging Compute- and Data-Optimal Pretraining
cs.LG 2026-07 conditional novelty 7.0 of 10

Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
cs.CL 2026-07 conditional novelty 6.0 of 10

A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.
Controllably Efficient Language Models
cs.LG 2025-11 conditional novelty 6.0 of 10

A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.
Linguistics and Human Brain: A Perspective of Computational Neuroscience
q-bio.NC 2026-02 unverdicted novelty 2.0 of 10

A narrative review arguing that computational neuroscience, powered by LLM-based model–brain alignment, serves as the bridge between linguistic theory and neural data.

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.