Pith. sign in

REVIEW 3 major objections 7 minor 29 references

Edit, don't rewrite: programmatic data cleaning beats raw scaling at 1B scale

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 03:40 UTC pith:W7JIC7LS

load-bearing objection Solid data refinement method with real code/data release; effect sizes are modest and lack significance testing the 3 major comments →

arxiv 2607.08646 v1 pith:W7JIC7LS submitted 2026-07-09 cs.CL cs.AI

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

Xinlong Zhao , Dongsheng Liu , Hengyu Zhao , Zixuan Fu , Zheng Wang , Jie Cai , Jie Zhou , Qiang Ma
show 4 more authors
Xuanhe Zhou Xu Han Yudong Wang Zhiyuan Liu
This is my paper
classification cs.CL cs.AI
keywords dataultraxlarge-scalereliabilityapproacheseditingefficiencyquality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that as raw training data becomes scarce, the bottleneck for improving language models shifts from collecting more text to cleaning what already exists. Existing approaches face a tension: rule-based cleaners are fast but too coarse, while large language models can rewrite text well but are too slow and unreliable for billion-document corpora. The authors propose UltraX, a method that trains a small, fast model to emit structured editing commands—delete these lines, replace this substring, insert this line—rather than regenerating text end-to-end. The key design move is completing the editing vocabulary: prior programmatic methods supported only deletion or replacement, which forces them to discard content that needs insertion or mixed edits; UltraX adds insertion, enabling structural repair of crawler-corrupted documents. The supervision pipeline uses an expert model to produce clean reference texts, then automatically converts the before-and-after difference into executable function calls. At inference time, a lightweight model predicts these commands over sliding windows of raw web text, and a deterministic executor applies them. The paper claims this approach yields better downstream model performance across five corpora while using fewer training tokens, because it removes noise without over-deleting useful content.

Core claim

A small model trained to predict executable editing operations—spanning deletion, replacement, and insertion—can refine large-scale pre-training data more effectively than both rule-based pipelines and prior deletion-only programmatic methods. Across five corpora and ten benchmarks, models trained on UltraX-refined data achieve the highest average scores, and reach baseline-matching performance with 16B instead of 20B training tokens. The gain comes not from deleting more aggressively but from finer-grained, instance-level editing that preserves useful content while removing noise. The complete function space (including insertion) matters: ablations show that removing any single operation类别—

What carries the argument

UltraX's machinery has three load-bearing components. (1) A complete function space of five operations: keep_all, remove_all, remove_lines(start, end), replace_str(line, old, new), and add_line(base, sub, content). The insertion operation is the paper's distinctive addition over prior work. (2) A supervision pipeline that converts expert-model-refined text pairs into function-call training data via Line Alignment Mapping (matching original and refined lines by content/context/position similarity) and Dynamic Context Replacement (converting character-level edits into uniquely locatable replace_str calls). (3) An inference pipeline with sliding-window prediction (20% overlap), global operation

Load-bearing premise

The observed gains are demonstrated only at the 1B-parameter scale with 20B training tokens and ten early-signal benchmarks; whether the same refinement advantages persist at the 7B–70B+ scale, with longer training schedules, and on more comprehensive evaluation suites remains unverified.

What would settle it

Pre-train models at 7B+ scale on UltraX-refined vs. baseline data with 100B+ tokens and a broader benchmark suite. If the average performance gap shrinks toward zero or reverses, the method's benefit may be an artifact of small-scale training dynamics rather than a genuine data-quality improvement.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If programmatic refinement scales to larger corpora and model sizes, data quality engineering could partially substitute for data quantity, extending the practical lifetime of existing web crawls.
  • The function-calling paradigm—small models emitting structured commands rather than free text—could generalize to other large-scale text transformation tasks beyond pre-training data, such as format normalization, deduplication, or domain adaptation.
  • The finding that insertion capability matters for structural repair suggests that crawler-corrupted text, previously discarded as unrecoverable, may contain salvageable training signal.
  • The supervision pipeline (expert output → automatic operation extraction) could reduce the cost bottleneck of creating high-quality training data, since it avoids per-document human annotation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces UltraX, a function-calling framework for refining large-scale pre-training data. UltraX extends prior programmatic refinement methods (ProX, RefineX) by introducing a complete editing function space—adding insertion (add_line) to deletion (remove_lines) and modification (replace_str)—and by building a more reliable supervision pipeline. The pipeline uses dataset-adaptive prompt optimization to guide an expert LLM (DeepSeek-V3.2) to produce end-to-end refined texts, which are then converted into structured function-call supervision via Line Alignment Mapping and Dynamic Context Replacement. During inference, UltraX employs sliding-window prediction, global operation aggregation, and systematic post-processing to ensure stable large-scale execution. The authors pretrain 1B-parameter MiniCPM models from scratch on five corpora (FineWeb, RedPajama-v2, AICC, Ultra-FineWeb, FineWeb-ProX-Doc) and evaluate on ten downstream benchmarks, reporting the highest average performance across all corpora and improved data efficiency (matching baselines with fewer training tokens). Ablation studies isolate the contributions of instruction guidance, seed operation distribution, quality-stratified refinement, and function space completeness.

Significance. The paper addresses a timely and practically important problem: scaling data refinement to pre-training-scale corpora while maintaining fine-grained, instance-level control. The complete function space design (including insertion) is a meaningful advance over ProX (replacement + deletion) and RefineX (deletion only), and the hierarchical text-to-operation mapping pipeline is a well-engineered solution for generating reliable supervision from LLM-refined text. The release of datasets, model checkpoints, and code is a notable strength that enhances reproducibility. The from-scratch pre-training experiments across five heterogeneous corpora provide a reasonable evaluation surface, and the ablation on function space (Table 8) convincingly demonstrates the synergy among insertion, deletion, and modification. The token-efficiency result (Figure 2: UltraX at 16B tokens surpassing Raw and ProX-C at 20B) is the most compelling individual finding.

major comments (3)
  1. §4.2, Table 2, and Abstract: The headline claim of 'relative improvements exceeding 2%' (Abstract) holds primarily against the unrefined Raw baseline, not against ProX-C. Against ProX-C, the average improvements are 1.09 (FineWeb), 0.51 (RedPajama-v2), 0.28 (AICC), 0.86 (Ultra-FineWeb), and 0.74 (FineWeb-ProX-Doc) points—roughly 0.3–2.4% relative. More critically, no statistical significance testing is reported. With 1,000 samples per benchmark (§C.2), the standard error for a ~45% accuracy task is approximately 1.6 percentage points, meaning many individual task differences fall within noise. The aggregate pattern (34/50 task-corpus wins) is harder to dismiss, but the paper should either (a) report confidence intervals or significance tests, (b) run multiple seeds, or (c) at minimum explicitly acknowledge that individual task-level differences are not statistically distinguishable andre
  2. §4.3, Tables 6–7: The quality-stratified refinement ablation produces a counterintuitive result that is not adequately discussed. In Table 6 (Tail refinement, Head fixed as Raw), UltraX with No-Instruction achieves the best average (45.82), outperforming the default UltraX (45.59). In Table 7 (Head refinement, Tail fixed as Raw), UltraX with Preservation-Weighted achieves 45.38, again outperforming the default UltraX (45.36). This means the default configuration (Instruction-Guided + Edit-Weighted) is not the best strategy in either stratified setting. The paper should clarify whether the default configuration was chosen based on the full-corpus results (Table 2) rather than these stratified ablations, and discuss whether the stratified results suggest that adaptive strategy selection (aggressive on Tail, conservative on Head) could yield further gains.
  3. §3.3 and Appendix A.1.3: The seed supervision pipeline depends on DeepSeek-V3.2 for end-to-end refinement, and the quality of UltraX's training data is bounded by this expert model's output. The paper acknowledges this dependency in §6 but does not quantify it. It would strengthen the contribution to report what fraction of seed examples pass the low-confidence filtering (Algorithm 1, lines 31–39) and how sensitive the final downstream performance is to the expert model choice. If, for example, a weaker expert model were used, would the hierarchical mapping pipeline still produce usable supervision? Even a brief sensitivity analysis or a discussion of the filtering yield would address this.
minor comments (7)
  1. §4.3, Table 6: The finding that UltraX (No-Instruction) outperforms the default UltraX on the Tail subset is interesting but the explanation ('stronger editing tendency can instead lead to better refinement effects') is speculative. A more direct analysis of what No-Instruction changes in the generated function distribution would be informative.
  2. Table 9: The token reduction percentages for AICC are nearly identical between ProX-C (-33.1%) and UltraX (-33.2%), yet UltraX preserves substantially more non-empty documents (18.59M vs 15.92M). This is a strong point that is underemphasized in the text; explicitly highlighting that UltraX achieves comparable compression while retaining 17% more documents would strengthen the 'information preservation' narrative.
  3. §2: The related work section discusses ProX and RefineX limitations in detail, but the distinction between 'refinement' and 'filtering' could be sharper. The current text blends the two in the second paragraph.
  4. Figure 1: The workflow diagram is informative but dense. The Stage I and Stage II labels are mentioned in the caption but not visually prominent in the figure itself.
  5. Table 10: The column headers use abbreviations (RL, RS, AL) defined in the caption, but the table would be more readable with slightly more descriptive headers or a footnote.
  6. §4.4, Table 12: The LLM-based quality evaluation uses DeepSeek-V3.2 as the judge—the same model used for seed data generation (§A.1.2). This creates a potential evaluator bias that should be acknowledged.
  7. Appendix A.1.2: The prompt optimization process runs for 'up to 200 iterations' per dataset. It would be useful to report the actual number of iterations and whether the prompt converged for each dataset.

Circularity Check

0 steps flagged

No circularity: UltraX's claims are grounded in external downstream benchmarks, not in self-referential definitions or fitted-input predictions.

full rationale

UltraX is an empirical data-refinement framework whose central claim—improved downstream LLM performance—is evaluated on external benchmarks (ARC, MMLU, SciQ, etc.) using models trained from scratch. The derivation chain proceeds as follows: (1) an expert LLM (DeepSeek-V3.2) generates end-to-end refined text; (2) a hierarchical mapping converts original-refined text pairs into structured function-call supervision; (3) a lightweight model is trained on this supervision; (4) the trained model refines large-scale corpora; (5) language models are pretrained on the refined corpora and evaluated on independent benchmarks. No step in this chain reduces to its own inputs by construction. The seed supervision depends on the expert model's quality, but this is standard distillation, not circularity—the evaluation metric (downstream accuracy) is distinct from the training signal (programmatic editing operations). The paper does not invoke any self-authored uniqueness theorem, does not fit a parameter to data and then rename the fit as a prediction, and does not define its target quantity in terms of itself. The self-citations present (e.g., MiniCPM, Ultra-FineWeb) are to prior infrastructure work, not to load-bearing mathematical claims that would force the present result. The modest effect sizes and absence of significance testing are correctness concerns, not circularity concerns.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 2 invented entities

The paper introduces several hand-tuned thresholds and parameters for its pipeline. The core axioms are standard domain assumptions in NLP and LLM evaluation. The invented entities are well-justified by the experimental results.

free parameters (5)
  • Line-pair scoring weights = 0.6/0.2/0.2
    Weights for content, context, and position similarity in Line Alignment Mapping (Appendix A.1.3).
  • Low-confidence filtering thresholds = 20 ops, 150 chars, 200 chars, 10 add_lines
    Thresholds for discarding low-confidence samples during function construction (Appendix A.1.3).
  • Quality classifier threshold = 0.05
    Threshold used to partition FineWeb into Head/Tail subsets for quality-stratified refinement ablation (§4.3).
  • Sliding window parameters = 12K tokens, 20% overlap
    Maximum window size and overlap ratio for segment-wise prediction (Appendix A.1.1, A.3.1).
  • Duplicate-pattern detection thresholds = 2, 3, 3, 30, 0.85
    Thresholds for detecting repetitive add_line, remove_lines, and replace_str patterns (Appendix A.3.3).
axioms (3)
  • domain assumption A lightweight model can learn to predict structured editing programs reliably when trained on high-quality supervision.
    The entire approach rests on the assumption that a 0.6B model is sufficient for the refinement task, which is empirically supported by the results but is not a proven fact.
  • domain assumption The 'early signal' benchmarks used for evaluation are representative of general LLM capabilities.
    The evaluation relies on ten benchmarks (§4.1) as a proxy for model quality, a standard but imperfect assumption in LLM evaluation.
  • domain assumption The expert LLM (DeepSeek-V3.2) produces sufficiently high-quality end-to-end refined text to serve as ground truth.
    The seed supervision quality is bounded by the expert model's refinement capability (§3.3).
invented entities (2)
  • UltraX refinement model independent evidence
    purpose: A lightweight (0.6B) model fine-tuned to predict executable function-call sequences for data refinement.
    The model is trained and evaluated on downstream tasks, providing evidence of its effectiveness.
  • UltraX function space independent evidence
    purpose: A set of five editing functions (keep_all, remove_all, remove_lines, replace_str, add_line) designed for programmatic text refinement.
    The function space is evaluated through ablation studies (Table 8) demonstrating the necessity of its components.

pith-pipeline@v1.1.0-glm · 40066 in / 2595 out tokens · 333282 ms · 2026-07-10T03:40:59.921945+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing." pith.science (2026). https://pith.science/paper/W7JIC7LS

@misc{pith2026260708646,
  author       = {Pith},
  title        = {Pith review of: UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7JIC7LS}},
  note         = {Machine review of arXiv:2607.08646}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.

Figures

Figures reproduced from arXiv: 2607.08646 by Dongsheng Liu, Hengyu Zhao, Jie Cai, Jie Zhou, Qiang Ma, XinLong Zhao, Xuanhe Zhou, Xu Han, Yudong Wang, Zheng Wang, Zhiyuan Liu, Zixuan Fu.

Figure 1
Figure 1. Figure 1: Overall workflow of UltraX, covering program-supervision generation, refinement model training, and inference-time program execution. 3.1 Overall Workflow As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Average downstream performance on FineWeb under different training token budgets. UltraX Demonstrates Strong Performance across Diverse Pre-Training Corpora [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of document-level token count distributions before and after refinement. Overall, both ProX-C and UltraX reduce the total number of tokens, but they exhibit different refinement behaviors. ProX-C often performs more aggressive token removal, reducing the total token count by 21.8%, 33.1%, and 8.0% on RedPajama-v2, AICC, and FineWeb-ProX-Doc, respectively. Although ProX-C does not explicitly defi… view at source ↗
Figure 4
Figure 4. Figure 4: Per-benchmark accuracy curves on FineWeb as a function of consumed training tokens. Each subplot corresponds to one of the ten evaluation benchmarks. D Case Study Despite the strong performance demonstrated in large-scale evaluations, we further select eight representative cases from randomly sampled FineWeb documents to qualitatively illustrate the behavioral differences between UltraX and ProX-C across d… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

    URLhttps://arxiv.org/abs/2306.01116. – 15 – UltraX Guilherme Penedo, Hynek Kydlíˇcek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557, 2024. Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, Jia Yu...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/n19-1421 2024

  2. [2]

    style improvement

    Prescribedataset-specific rules based on the dataset profile, such as noise patterns, content types, and formatting conventions. 3.Outputthe complete improved prompt between<IMPROVED_PROMPT>and</IMPROVED_PROMPT>tags. Rules for the Improved Prompt. • The output contract is either cleaned text or[Content valueless, deleted]. • Emphasize the no-rewrite princ...

    work page

  3. [3]

    If the original text has poor grammar or awkward phrasing but is understandable, KEEP IT AS IS

    Retain core information and ORIGINAL WORDING.Do NOT summarize, rewrite, or "polish" the text. If the original text has poor grammar or awkward phrasing but is understandable, KEEP IT AS IS

    work page

  4. [4]

    follow for more

    Remove all advertisements and traffic-driving info(phone numbers, email addresses, social media handles like Twitter/WeChat/Discord, QR code descriptions, "follow for more", affiliate links, paid course promos, etc.). 3.Remove engagement bait(e.g., "Smash that like button", "Don’t forget to subscribe", "Share this post")

    work page

  5. [5]

    Do NOT treat them as decorative symbols

    Remove purely decorative symbols, repetitive separators, and excessive ASCII art/borders.CRITICAL EXCEPTION: Strictly PRESERVE Markdown structural characters (e.g., # for headers, * or - for lists, | for tables, > for quotes). Do NOT treat them as decorative symbols

    work page

  6. [6]

    However, do NOT remove Markdown formatting unless it is broken

    Remove HTML/XML tags(e.g., <div>, <p>, &nbsp;, <s>). However, do NOT remove Markdown formatting unless it is broken. Specifically, ALW AYS keep the # symbols in headers (e.g., convert <h1>Title</h1> to # Title, and keep existing # Title as is)

    work page

  7. [7]

    Clean up obvious gibberishand unreadable characters (including long strings of meaningless garbage, encoding error symbols like “)

    work page

  8. [8]

    commu-\nication

    Fix broken formatting ONLY:Merge lines that were unnecessarily split by newlines (e.g., "commu-\nication"). DO NOT rephrase sentences to improve flow or style

    work page

  9. [9]

    Retain all sub-headers/titles exactly as they are (including their # prefix)

    Merge paragraphs cautiously:Maintain the document hierarchy. Retain all sub-headers/titles exactly as they are (including their # prefix). Do not merge a header into the following paragraph or strip its formatting level

    work page

  10. [10]

    See Figure 1

    Remove meaningless image placeholders(e.g., text like "See Figure 1", "Image missing", "[Image]" if they are just placeholders without the actual image)

    work page

  11. [11]

    Simplify excessive slang or internet shorthand(make the expression more professional and clear without changing the original meaning); however, if the slang aids understanding or is part of the original flavor/context, it can be retained

    work page

  12. [12]

    Retain useful citations/sources(if it is a necessary academic reference, keep the citation marker and identifiable info, but remove unparsed, long, raw URLs)

    work page

  13. [13]

    The text should be fluent, professional, and ready for use

    Output format:Remove extra spaces, repetitive blank lines, and tabs. The text should be fluent, professional, and ready for use. Extra Rules. •Output Language:English. Keep the output in English! Do NOT translate English text to other languages. • Final Output:Only the refined English text (without any quotes or prefixes) OR the exact string [Content valu...

    work page

  14. [14]

    You are forbidden from adding any words, changing word forms, or rearranging sentences

    STRICT SUBSET ONLY:Your output must be a strict character-level subset of the input. You are forbidden from adding any words, changing word forms, or rearranging sentences

    work page

  15. [15]

    fix" awkward phrasing, non-native English, or

    NO LINGUISTIC NORMALIZATION:Do NOT "fix" awkward phrasing, non-native English, or "broken" translations. If the input says "thorough new fabric on wind energy," you MUST keep "fabric." Do NOT change it to "material." If it says "moment version," do NOT change it to "updated version."

    work page

  16. [16]

    404 Not Found,

    PRESERVE TECHNICAL JARGON:Keep all typos, archaic terms, and specialized academic terminology exactly as-is. These are essential data signals for model training. 4.ALREADY CLEAN:If the text requires no deletions, output it exactly as-is. Valueless Content (Deletion Marker).Output exactly[Content valueless, deleted]if the document is: • Pure Spam/SEO:Gambl...

    work page 2022

  17. [17]

    ). • E-commerce Noise:Cart status (

    All rights reserved"). • E-commerce Noise:Cart status ("Your cart is empty"), price tags, "Add to Cart," shipping info, "item unavailable" notices, and coupon/discount codes. • Non-English Blocks:Delete blocks of non-English text (German, French, etc.) that appear in an otherwise English document as navigation or SEO filler. •HTML Residue:Isolated tags li...

    work page

  18. [18]

    Click here to subscribe

    The Line-Level Rule:If an entire line is noise (e.g., "Click here to subscribe"), delete the entire line and its newline

    work page

  19. [19]

    The car is fast [Share on Twitter] and red

    The Fragment Rule:If noise is embedded in a sentence (e.g., "The car is fast [Share on Twitter] and red"), delete only the noise fragment

    work page

  20. [20]

    Never rewrite the sentence to fix the grammar

    The Coherence Rule:If deleting a noise fragment makes the sentence ungrammatical, you must either keep the whole sentence (noise included) or delete the whole sentence. Never rewrite the sentence to fix the grammar. 4.Preserve Structure:Maintain original paragraph breaks and list structures. Task.Clean the following text using the surgical protocols above...

    work page

  21. [21]

    Share"/"Subscribe

    REMOVE:navigation/breadcrumbs, ads/banners, copyright/cookie notices, "Share"/"Subscribe"/"Sign up" prompts, SEO stuffing, boilerplate templates, e-commerce UI elements (prices, "Add to cart", stock status), "Related posts"/"You may also like" sections, and comment section headers. 2.PRESERVE:article body, factual content, quotes, data, author info with c...

    work page

  22. [22]

    When in doubt,KEEPthe content — over-deletion is worse than under-deletion

    work page

  23. [23]

    Prefer remove_lines when entire lines are noise; use replace_str only for inline noise within otherwise valuable lines

    work page

  24. [24]

    Useremove_all()only when the document has absolutely zero informational value

    work page

  25. [25]

    Output operations only, one per line. – 23 – UltraX Algorithm 1Converting End-to-End Refinement into Function-Call Supervision Require:Original textx, refined textˆx Ensure:A training example(u, y)or DISCARD 1:ifx= ˆxthen 2:O ←[keep_all()] 3:else ifˆxis[Content valueless, deleted]then 4:O ←[remove_all()] 5:else 6:Splitxandˆxinto line sequencesLand ˆL 7:Co...

    work page 2025

  26. [26]

    Ultra-FineWeb (Wang et al.,

    is a large-scale corpus designed for web content extraction, emphasizing fine-grained text parsing and cleaning from raw HTML; we randomly sample 20B tokens from its full corpus. Ultra-FineWeb (Wang et al.,

    work page

  27. [27]

    early signal

    is a quality-enhanced web corpus built upon FineWeb-style data curation, from which we also randomly sample 20B tokens. FineWeb-ProX-Doc (Zhou et al., 2024) is a high-quality FineWeb subset obtained through ProX document-level filtering. We use its sample-350BT subset, which contains approximately 350B tokens, and select 20B high-quality tokens using the ...

    work page 2024

  28. [28]

    as a tenth benchmark, which has been widely adopted in recent works (Mehta et al., 2024; Wettig et al.,

    work page 2024

  29. [29]

    Add To Cart

    and shown to be an informative proxy for broader model capabilities. The complete list of evaluated datasets includes ARC-Easy and ARC-Challenge (Clark et al., 2018), Com- monSenseQA (Talmor et al., 2019), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2...

    work page 2018