Pith. sign in

REVIEW 9 cited by

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.16380 v1 pith:O2LWNDF3 submitted 2024-01-29 cs.CL

Pratyush Maini , Skyler Seto , He Bai , David Grangier , Yizhe Zhang , Navdeep Jaitly This is my paper
classification cs.CL
keywords datacomputepre-trainingmodelstyleacrossbecausehigher
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.

  3. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  4. ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.

  5. Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

    cs.CL 2026-07 conditional novelty 5.0 of 10

    For closed-book QA via LoRA adapters, once capacity is sufficient, training-data quality—especially answer brevity—outperforms rank, learning rate, and architecture changes, lifting 15-doc accuracy from 57.7% to 85.7%.

  6. Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Running multiple short annealing runs at different token scales can reveal per-source utility scaling curves that change data-source rankings compared with single point estimates.

  7. Assessing the Role of Data Quality in Training Bilingual Language Models

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.

  8. MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution

    cs.SE 2025-06 conditional novelty 5.0 of 10

    MCTS-REFINE uses tree search plus strict ground-truth matching to build chain-of-thought training data that lifts open-source LLM issue-resolution scores on SWE-bench.

  9. Generative Data Refinement: Just Ask for Better Data

    cs.LG 2025-09 conditional novelty 4.0 of 10

    A pretrained LLM can rewrite individual data samples to strip out PII or toxic content while preserving useful information, creating safer training data.

Pith tools