Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Essential-Web v1.0: 24T tokens of organized web data

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Twenty-four trillion web tokens, each document tagged with a twelve-category taxonomy, reduce domain data curation to SQL queries that match or beat specialist pipelines on math, code, STEM, and medical benchmarks.

desk verdict Substantial dataset release, but the abstract's 'SQL-style filters' headline conflates pure taxonomy with taxonomy+classifier hybrids; the per-dataset tables tell a more modest and honest story. read the letter →

arxiv 2506.14111 v2 pith:Z5QZSGAL submitted 2025-06-17 cs.CL cs.AIcs.LG

Essential AI: Andrew Hojel , Michael Pust , Tim Romanski , Yash Vanjani , Ritvik Kapila , Mohit Parmar , Adarsh Chaluvaraju , Alok Tripathy
show 15 more authors
Anil Thomas Ashish Tanwer Darsh J Shah Ishaan Shah Karl Stratos Khoi Nguyen Kurt Smith Michael Callahan Peter Rushton Philip Monk Platon Mazarakis Saad Jamal Saurabh Srivastava Somanshu Singla Ashish Vaswani
This is my paper
classification cs.CLcs.AIcs.LG
keywords Essential-Webweb-scalepretrainingcorpusdatacurationdocumenttaxonomyCommonCrawldistillationSQL-stylefilteringdomain-specificdatasets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the costliest part of open model-building — bespoke, months-long pipelines that extract domain corpora (math, code, medicine) from the web — can be replaced by one organized corpus plus a query. The authors release Essential-Web v1.0, a 24-trillion-token Common Crawl corpus in which each of 23.6 billion documents carries labels for twelve taxonomy categories spanning subject matter, page type, content complexity, and quality. Using nothing more than SQL-style filters over those labels, they curate datasets that score within 8.0% of the best open web-curated baseline on math and beat it by 14.3% on web code, 24.5% on STEM, and 8.6% on medical benchmarks. If the claim holds, pre-training data curation becomes a search problem any researcher can solve in minutes, and the one-off cost of labeling the corpus is amortized across everyone who uses it.

What carries the argument

The load-bearing object is EAI-Taxonomy, a fixed twelve-category labeling scheme whose categories act as orthogonal axes: each document receives one primary and one optional secondary label per category, so a curated dataset is exactly a conjunction of predicates over those columns. Subject matter uses Free Decimal Correspondence, a public-domain analogue of the Dewey Decimal System, whose three hierarchical levels let a filter like 'FDC = 51' isolate mathematics at 98% recall while keeping 0.5% of the crawl. The second machinery piece is EAI-Distill-0.5b, a 0.5B-parameter annotator fine-tuned with context distillation and condensed outputs, which raises throughput to 70 requests per second per GPU and makes labeling 23.6 billion documents feasible at roughly 90k GPU-hours. A third piece is the paper's metric suite — normalized mutual information for category independence, a set-overlap variant of Cohen's kappa for label clarity, and URL-based domain recall for retrieval power — which is what justifies the taxonomy's shape before any downstream training.

What would settle it

Take a fresh sample of roughly 20,000 documents from the released corpus, have the two reference annotators (GPT-4o and Claude Sonnet-3.5) relabel them, and compare per-category agreement against EAI-Distill-0.5b's stored labels. If categories such as Extraction Artifacts (kappa 0.27 on the current 2,017-document set) and Missing Content (kappa 0.48) show similarly low agreement at this larger scale, and if re-running the paper's four published filter algorithms on the corrected labels no longer reproduces the reported margins (math within 8.0%, web code +14.3%, STEM +24.5%, medical +8.6%), the central claim that taxonomy quality carries to the full corpus fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a document-level taxonomy turns corpus curation into a query interface. Every document in Essential-Web v1.0 is annotated with EAI-Taxonomy, a fixed twelve-category scheme covering subject (three levels of the public-domain Free Decimal Correspondence), cognitive demand (Bloom's taxonomy), page type (two overlapping label sets), content quality (reasoning depth, educational level, technical correctness), and extraction health (artifacts, missing content), with a primary and optional secondary label per category. The labels come from EAI-Distill-0.5b, a 0.5B-parameter model fine-tuned on 82B tokens of teacher labels, that matches Qwen2.5-32B-Instruct within 3% on annotator agreement while running roughly 50 times faster. With filters alone — for example FDC code 51 (Mathematics) plus thresholds on reasoning depth and technical correctness — annealed 2.3B-parameter models land within standard error of specialist corpora: from 8.0% behind state of the art on math to 24.5% ahead on STEM, with the strongest results coming from combining taxonomy filters with an instruction-density classifier.

Load-bearing premise

The paper assumes that the 0.5B annotator's labels are as accurate across all 23.6 billion documents as they are on the small held-out sets it was checked on (2,017 random and 871 STEM documents), even though agreement drops sharply for some categories in the student model, with Extraction Artifacts falling from kappa 0.74 to 0.27.

Editorial extensions

If this is right

  • A practitioner who wants, say, a chemistry corpus can write a metadata query in under 15 minutes instead of training a high-recall classifier, running it over hundreds of millions of pages, and manually inspecting the output.
  • Combining taxonomy filters with an existing instruction-density classifier is the paper's strongest recipe: it lifts STEM 24.5% and medical 8.6% over baselines and adds 15.0 percentage points over DCLM-baseline on MMLU computer science.
  • The taxonomy is dense enough to audit other datasets: annotating FineMath 3+ with it shows only 61.9% of that 'math' corpus is labeled Mathematics, with physics, economics, and applied physics making up much of the rest.
  • Domain-recall measurements show simple filters retrieve 95.8-97.6% of human-vetted math and code pages while keeping under 5% of the crawl, a far higher density than fastText classifiers at comparable volume.
  • The one-off labeling cost (about 90k MI300x GPU-hours) is a fixed investment that the community amortizes, so subsequent domain corpus construction is nearly free.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My read: if label quality survives at scale, the quality axes of the taxonomy (reasoning depth, technical correctness) could replace monolithic quality classifiers in general-purpose pipelines, because they are per-document, interpretable, and queryable rather than a single opaque score.
  • The paper's own numbers suggest a testable recipe for new domains: a subject-code filter plus reasoning/correctness thresholds may be a sufficient universal prior, making even the DCLM classifier combination unnecessary for domains with clean subject codes.
  • The largest internal risk is the student model's per-category agreement drops — Extraction Artifacts kappa falls from 0.74 to 0.27 and Missing Content from 0.66 to 0.48 — so the two extraction-health categories are the ones most likely to fail at full-corpus scale; a larger held-out sample would settle this.
  • Because the taxonomy is built on a public-domain subject hierarchy, the same filter definitions could become standardized, reusable data-selection recipes — effectively turning every domain benchmark into a benchmark of data curation methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces Essential-Web v1.0, a 24-trillion-token, deduplicated Common Crawl dataset in which every document carries labels from a 12-category taxonomy (subject, document type, content complexity, quality, extraction issues). The labels are produced by EAI-Distill-0.5b, a distilled 0.5B model trained on synthetic labels from Qwen2.5-32B-Instruct. The authors argue that simple SQL-style filters over the taxonomy yield competitive domain-specific pretraining datasets for math, code, STEM, and medicine, without the bespoke pipelines used by prior work. They also propose a metric suite (NMI, annotator kappa, domain recall) for taxonomy evaluation and report downstream annealing experiments on 2.3B-parameter models with decontamination and controlled base models.

Significance. If the central claim holds, this is a substantial contribution to reproducible data-centric LM research: a 24T-token public corpus with interpretable document-level metadata could let practitioners assemble domain-specific datasets in hours rather than months, and the release of the dataset and annotator model would be a community resource. The downstream evaluation is careful in design: identical base models, the same annealing protocol, and a 13-gram decontamination filter. The domain-recall metric and the use of external gold annotators (GPT-4o, Claude) are also strengths. However, the headline empirical claims overstate what the pure taxonomy filters deliver, and the full-scale label-quality evidence is thinner than the scale of the release requires. The paper needs a major revision to align the abstract and contributions with the actual pipelines and to provide stronger validation of label quality at corpus scale.

major comments (3)
  1. [Abstract; §4.2–4.5, Tables 3/5/7/9, Algorithms 3/5/7/9] The abstract's claim that 'with nothing more than SQL-style filters' yields math (-8.0%), web code (+14.3%), STEM (+24.5%), and medical (+8.6%) is not supported by the tables. The math figure is achieved by Algorithm 3, which applies the FineMath classifier to 116M recalled documents; the pure-taxonomy Top Math dataset trails FineMath 3+ by 19.3% relative on GSM8K (21.3 vs 26.4, Table 3), even though it is close on MATH and MMLU-Math. The STEM +24.5% and medical +8.6% figures correspond to the w/ DCLM variants (Algorithms 9 and 7), while pure taxonomy STEM gains are +5.1% relative (29.1 vs 27.7, Table 9) and pure medical results are mixed (e.g., PubMedQA 67.0 vs 70.6 for DCLM-baseline, Table 7). The paper is transparent in the per-dataset tables, but the abstract, the introduction, and contribution 2 in §2.1 overstate what the pure taxonomy filters deliver. Please revise the abstract and contributions to attribute each headline number to the exact pipeline, or report pure-taxonomy headline numbers.
  2. [§6.3, Table 13; §5.1.4] The quality of the student model's labels at full 23.6B-document scale is a load-bearing premise, but the evidence is thin. The student is evaluated on only 2,017 random and 871 STEM documents, and on categories that drive the filters it shows large agreement drops relative to the teacher: Extraction Artifacts random kappa falls from 0.74 to 0.27, and Missing Content from 0.66 to 0.48 (Table 13). At the same time, the student anomalously exceeds the teacher on Reasoning Depth (0.87 vs 0.67 random) and Technical Correctness (0.72 vs 0.52 random). Since Algorithms 2–9 rely on exactly these categories, and no held-out evaluation is performed at corpus scale, the label distribution in Essential-Web v1.0 could differ substantially from these small sets. Please add a stratified validation sample (e.g., 5–10k documents) comparing EAI-Distill-0.5b labels against the teacher or gold annotators, at least for the categories used in the downstream filters.
  3. [§4.1, Algorithms 2–9] The paper does not report how the taxonomy filter rules and thresholds were selected. If any of the label sets or thresholds (e.g., DCLM_baseline_thresh = 0.01811, the reasoning-depth and technical-correctness criteria, or the FDC+document-type combinations) were chosen after inspecting results on the same benchmarks, the 'simple filters' comparison is partially circular. Please state the filter development procedure, hold out the downstream benchmarks when tuning, or provide ablation results showing that reasonable perturbations of the filters do not change the qualitative conclusions.
minor comments (3)
  1. [Appendix A.8, Algorithm 6] Algorithm 6 references d.dds.primary and d.dds.secondary, but the dataset schema described elsewhere uses d.fdc; please fix this typo so the medical filter is reproducible.
  2. [§2.1, contribution 4; §6.3.2, Table 14] The statement that EAI-Distill-0.5b remains 'within 3%, 14%, 1% of the teacher' on annotator agreement, NMI, and domain-recall is not accurate as written: Table 14 shows the average inter-category NMI increases by 16.5% on the random set (0.079 to 0.092), and Table 13 shows the random-set kappa drops by 4.1% (0.74 to 0.71). Please qualify these numbers by evaluation set or report the exact range.
  3. [Abstract] The abstract says EAI-Distill-0.5b achieves 'annotator agreement within 3%' of Qwen2.5-32B-Instruct, but Table 13 shows a 4.1% relative drop on the random set; please specify that the 3% figure refers to the STEM set or report the average over both sets accurately.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the abstract overstates pure-taxonomy support, but the results are externally benchmarked.

full rationale

I find no circular step. The taxonomy labels are validated against external gold annotators (GPT-4o and Claude Sonnet 3.5) on held-out sets, and the distilled student is measured against those same external references plus human-vetted URL sets, not against its own outputs. The downstream math, code, STEM, and medical results are new annealing experiments on held-out benchmarks (GSM8K, MATH, HumanEval+, MMLU subsets, and medical QA), so they do not reduce by construction to the taxonomy labels. The one notable issue is presentation: the abstract's 'nothing more than SQL-style filters' headline numbers for math (-8.0%), STEM (+24.5%), and medical (+8.6%) come from hybrid pipelines that also use the FineMath or DCLM classifiers (Table 2 and Algorithms 3, 7, and 9), while the pure-taxonomy variants are weaker (e.g., Top Math trails FineMath 3+ by 19.3% relative on GSM8K). This is an overclaim about what the headline numbers measure, not a circular derivation: the hybrid results are still externally benchmarked and fully disclosed in the body. No fitted parameter is renamed as a prediction, and no load-bearing premise depends on a self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities or forces. The taxonomy is a labeling scheme, not an invented entity. Free parameters are limited to thresholds adopted from prior work. The central claim rests on assumptions about label reliability, evaluation protocol, and the FDC subject scheme.

free parameters (2)
  • DCLM_baseline_thresh = 0.01811
    Threshold inherited from DCLM-baseline and reused in the combined Code/Med/STEM w/ DCLM filters; not fitted in this paper.
  • Legacy quality thresholds (Algorithm 1) = various (e.g., 0.20, 0.18, 0.50)
    Adopted from RedPajama/FineWeb-style filters; used in preprocessing, not fitted to the central claim.
assumptions (4)
  • domain assumption Gold annotators (GPT-4o, Claude Sonnet) provide valid ground truth for all 12 taxonomy categories.
    A human study on 4 categories shows humans agree better with LLMs than with each other, but this does not fully validate subjective categories like Technical Correctness.
  • domain assumption Annealing a 2.3B model for 80B tokens is a reliable indicator of dataset quality.
    The paper uses this protocol to compare datasets; it may not generalize to larger models or longer training.
  • domain assumption The Free Decimal Correspondence (FDC) taxonomy is a suitable subject classification for web documents.
    The paper relies on the public-domain FDC system to label subjects; coverage of non-English or niche topics is untested.
  • domain assumption Small held-out evaluation sets represent the full 23.6B-document corpus for label quality.
    Taxonomy quality is measured on 2,017 random and 871 STEM documents; label accuracy at scale is assumed to match.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Essential-Web v1.0: 24T tokens of organized web data." pith.science (2026). https://pith.science/paper/Z5QZSGAL

@misc{pith2026250614111,
  author       = {Pith},
  title        = {Pith review of: Essential-Web v1.0: 24T tokens of organized web data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z5QZSGAL}},
  note         = {Machine review of arXiv:2506.14111}
}
read the original abstract

Data plays the most prominent role in how language models acquire skills and knowledge. The lack of massive, well-organized pre-training datasets results in costly and inaccessible data pipelines. We present Essential-Web v1.0, a 24-trillion-token dataset in which every document is annotated with a twelve-category taxonomy covering topic, format, content complexity, and quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned 0.5b-parameter model that achieves an annotator agreement within 3% of Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain competitive web-curated datasets in math (-8.0% relative to SOTA), web code (+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0

Figures

Figures reproduced from arXiv: 2506.14111 by the authors.

Figure 1
Figure 1. LLM pre-training dataset sizes over time. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of our five-stage methodology for developing and deploying EAI-TA [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. We train two base models: (1) General Base (100% web data) and (2) Code Base (50% web data, 50% code data) for 320B tokens. We then evaluate all domain-specific EAI-TAXONOMY and top-performing, web-curated datasets by annealing one of the two base models for 80B tokens. and GSM8K, as opposed to training on each dataset from scratch for 80B tokens. After annealing, each model has seen 400B tokens, 10× Chinchilla Pare… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of NMI heatmaps on random and STEM eval sets for [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Change in inter-category NMI per category from [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: NMI heatmaps on random and STEM evaluation sets for [PITH_FULL_IMAGE:figures/full_fig_p042_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

Reference graph

Works this paper leans on

67 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Smollm2: When smol goes big -- data-centric training of a small language model, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    arXiv 2025

  2. [2]

    Anderson and D.R

    L.W. Anderson and D.R. Krathwohl. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Longman, 2001

    work page 2001

  3. [3]

    Claude 3.5 Sonnet Model Card Addendum

    Anthropic . Claude 3.5 Sonnet Model Card Addendum . Model card, Anthropic, June 2024. URL https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf. Accessed 23 May 2025

    work page 2024

  4. [4]

    Automatic evaluation of healthcare llms beyond question-answering, 2025

    Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urcelay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, and Dario Garcia-Gasulla. Automatic evaluation of healthcare llms beyond question-answering, 2025. URL https://arxiv.org/abs/2502.06666

    arXiv 2025

  5. [5]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  6. [6]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27), June 2024. ISSN 1091-6490. doi:10.1073/pnas.2311878121. URL http://dx.doi.org/10.1073/pnas.2311878121

    doi:10.1073/pnas.2311878121 2024

  7. [7]

    Qwen technical report, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  8. [8]

    xlstm: Extended long short-term memory, 2024

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL https://arxiv.org/abs/2405.04517

    arXiv 2024

Show all 67 references
  1. [9]

    Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl

    Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl . In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski, editors, Advances in Information Retrieval. 40th European Confe...

    2018

  2. [10]

    Enriching word vectors with subword information, 2017

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. URL https://arxiv.org/abs/1607.04606

    2017 arXiv

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  4. [12]

    A coefficient of agreement for nominal scales

    Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 0 37 -- 46, 1960. URL https://api.semanticscholar.org/CorpusID:15926286

    1960

  5. [13]

    xxhash, 2025

    Yann Collet. xxhash, 2025. URL https://xxhash.com

    2025

  6. [14]

    Common Crawl Corpus

    Common Crawl Foundation . Common Crawl Corpus . https://commoncrawl.org, 2025. all shards uo to CC-MAIN-2024-38

    2025

  7. [15]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024 a

    DeepSeek-AI. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024 a . URL https://arxiv.org/abs/2406.11931

    2024 arXiv

  8. [16]

    Deepseek llm: Scaling open-source language models with longtermism, 2024 b

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism, 2024 b . URL https://arxiv.org/abs/2401.02954

    2024 arXiv

  9. [17]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024 c

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024 c . URL https://arxiv.org/abs/2405.04434

    2024 arXiv

  10. [18]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437

    2025 arXiv

  11. [19]

    High agreement but low kappa: I

    Alvan R Feinstein and Domenic V Cicchetti. High agreement but low kappa: I. the problems of two paradoxes. Journal of clinical epidemiology, 43 0 (6): 0 543--549, 1990

    1990

  12. [20]

    Thebluescrubs-v1, a comprehensive curated medical dataset derived from the internet, 2025

    Luis Felipe, Carlos Garcia, Issam El Naqa, Monique Shotande, Aakash Tripathi, Vivek Rudrapatna, Ghulam Rasool, Danielle Bitterman, and Gilmer Valdes. Thebluescrubs-v1, a comprehensive curated medical dataset derived from the internet, 2025. URL https://arxiv.org/abs/2504.02874

    2025 arXiv

  13. [21]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  14. [22]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma 2 Team . Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  15. [23]

    Gemma 3 technical report, 2025

    Gemma 3 Team . Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  16. [24]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  17. [25]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Sim...

    2022

  18. [26]

    Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier co...

    2025 arXiv

  19. [27]

    Qwen2.5-coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 arXiv

  20. [28]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL https://arxiv.org/abs/2009.13081

    2020 arXiv

  21. [29]

    P ub M ed QA : A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. P ub M ed QA : A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural La...

    2019 doi

  22. [30]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  23. [31]

    Justification for the use of cohen’s kappa statistic in experimental studies of nlp and text mining

    AS Kolesnyk and NF Khairova. Justification for the use of cohen’s kappa statistic in experimental studies of nlp and text mining. Cybernetics and Systems Analysis, 58 0 (2): 0 280--288, 2022

    2022

  24. [32]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180

    2023 arXiv

  25. [33]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2025 arXiv

  26. [34]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023. URL https://arxiv.org/abs/2305.01210

    2023 arXiv

  27. [35]

    Muon is scalable for llm training, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Z...

    2025 arXiv

  28. [36]

    The llama 3 herd of models, 2024

    Llama Team . The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  29. [37]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  30. [38]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025

    Meta AI . The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Accessed: 2025-06-15

    2025

  31. [39]

    Free decimal correspondence

    John Mark Ockerbloom. Free decimal correspondence. https://everybodyslibraries.com/free-decimal-correspondence/, August 2010. Released 19 August 2010; dedicated to the public domain (CC0)

    2010

  32. [40]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  33. [41]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors, Proceedings of the Conferen...

    2022

  34. [42]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL https://arxiv.org/abs/2310.06786

    2023 arXiv

  35. [43]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. UR...

    2023 arXiv

  36. [44]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  37. [45]

    Qwen2.5 technical report, 2025

    Qwen. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  38. [46]

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks,...

    2022 arXiv

  39. [47]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    2023 arXiv

  40. [48]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  41. [49]

    Scaling laws for linear complexity language models, 2024 a

    Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models, 2024 a . URL https://arxiv.org/abs/2406.16690

    2024 arXiv

  42. [50]

    Slimpajama-dc: Understanding data combinations for llm training, 2024 b

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-dc: Understanding data combinations for llm training, 2024 b . URL https://arxiv.org/abs/2309.10818

    2024 arXiv

  43. [51]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Ague...

    2022 arXiv

  44. [52]

    Learning by distilling context, 2022

    Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022. URL https://arxiv.org/abs/2209.15189

    2022 arXiv

  45. [53]

    Software Heritage Archive

    Software Heritage Foundation . Software Heritage Archive . https://archive.softwareheritage.org, 2025. Snapshot accessed 10 June 2025

    2025

  46. [54]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Mu...

    2024 arXiv

  47. [55]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, 2023. URL https://arxiv.org/abs/2206.14486

    2023 arXiv

  48. [56]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2025

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2025. URL https://arxiv.org/abs/2412.02595

    2025 arXiv

  49. [57]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  50. [58]

    Llama: Open and efficient foundation language models, 2023 a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  51. [59]

    Llama 2: Open foundation and fine-tuned chat models, 2023 b

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  52. [60]

    Redpajama: an open dataset for training large language models, 2024 a

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpaja...

    2024 arXiv

  53. [61]

    Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redp...

    2024

  54. [62]

    Organize the web: Constructing domains enhances pre-training data curation, 2025

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation, 2025. URL https://arxiv.org/abs/2502.10341

    2025 arXiv

  55. [63]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  56. [64]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  57. [65]

    Naturalreasoning: Reasoning in the wild with 2.8m challenging questions, 2025

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li. Naturalreasoning: Reasoning in the wild with 2.8m challenging questions, 2025. URL https://arxiv.org/abs/2502.13124

    2025

  58. [66]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104

    2024 arXiv

  59. [67]

    Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. Megamath: Pushing the limits of open math corpora, 2025. URL https://arxiv.org/abs/2504.02807

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.