REVIEW 3 major objections 7 minor 3 cited by
ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models
T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read ChemPile gathers over 75 billion tokens of curated chemical data into an open corpus built for training chemical foundation models.
desk verdict ChemPile is a genuinely large, openly released chemical corpus that the field will likely use, but the quality of its largest paper subset rests on a classifier validated on 150 examples from a different domain; the paper should be published only after the authors either tighten that validation or soften the curation claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is ChemPile itself, built as reproducible infrastructure rather than a single static file. Three mechanisms carry the argument: a transformer-based text classifier that filters a large literature corpus down to chemistry-related papers; a sampling engine that converts tabular chemistry datasets into natural-language questions by filling 1,636 expert-written templates with randomized synonyms, enumeration schemes, and multiple-choice options, then expands every entry into SMILES, SELFIES, InChI, IUPAC names, and images; and a split protocol that assigns every molecule to a global scaffold-based train/validation/test partition so the same molecular core never appears in both training and evaluation.
What would settle it
Draw a random sample of about 1,000 documents from the Paper subset and have two independent chemists label each as chemistry research or not; if the measured precision falls well short of the approximately 0.77 F1 reported, or if a large fraction of the corpus's biggest text subsets is clearly off-topic, the curated-chemical-corpus claim is contradicted.
Extended reading notes
Core claim
ChemPile's central claim is that a single, openly released corpus can supply the volume, diversity, and quality that chemical foundation models have been missing. The dataset spans seven subsets built from very different sources: textbooks and lecture transcripts; research articles filtered from large abstract and full-text corpora; tabular chemistry datasets converted into natural language through 1,636 hand-written templates; those same tabular data expanded into multiple molecular representations and rendered molecular images; code filtered from large permissively licensed code corpora; community question-answer data; and synthetic reasoning traces for interpreting molecular spectra. Totaling about 255 GB, 76.7 billion tokens, and 260 million documents, the corpus is, the paper states, larger than the 34-billion-token corpus behind the largest previously reported chemical foundation model and orders of magnitude larger than released chemical instruction datasets. The paper also provides scaffold-based splits designed to keep the same molecular scaffold out of both training and test partitions, and it reports that embeddings of IUPAC names track molecular similarity more closely (r=0.722) than embeddings of SMILES strings (r=0.521), evidence that representation choice matters for how well the corpus can teach chemistry.
Load-bearing premise
The entire 'curated chemical corpus' claim rests on the assumption that the classifier used to select the paper subset is accurate enough, but it was validated on only about 150 manually labeled examples with an F1 near 0.77, so a modest drop in precision would admit substantial non-chemical text into the 14.1-billion-token Paper subset.
Editorial extensions
If this is right
- Chemical foundation models can be pretrained entirely on open data at a scale (roughly 76.7 billion tokens) that was previously available only to much smaller or closed chemical datasets.
- Because the same molecules appear in SMILES, SELFIES, IUPAC, InChI, and rendered images, researchers can directly test which representation or combination transfers best to property prediction and inverse design.
- The scaffold-based splits make benchmark results comparable across labs by preventing the same molecular scaffold from appearing in both training and test partitions.
- The modular subsets support data-mixing studies, including how much code, reasoning-trace, or image-caption data improves chemical reasoning and multimodal understanding.
Reading between the lines
- If the scale claim holds, chemical benchmarks should show scaling-law-like log-linear gains in downstream accuracy as ChemPile token counts grow; the paper does not itself present such scaling curves.
- The reported IUPAC-embedding advantage suggests a single-representation ablation could show that IUPAC-heavy pretraining beats SMILES-only training for property prediction, but that model comparison is not in the paper.
- Because much of the reasoning subset was generated by LLMs, models trained on ChemPile may inherit model-specific spectral-assignment errors; comparing downstream reasoning accuracy with and without the synthetic traces would separate distillation gains from distillation noise.
- The template-sampling engine is domain-agnostic in design, so the same curation recipe could convert tabular data from other sciences into language-interfaced instruction data.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ChemPile is an open, multimodal chemical corpus reported as 255 GB of compressed Parquet data (76.7B GPT-2 tokens, 260M documents) assembled from seven subsets: ChemPile-Education (LibreTexts textbooks, MIT OCW materials, YouTube lecture transcripts, US Olympiad problems), ChemPile-Paper (chemistry-filtered EuroPMC literature, ChemRxiv/BioRxiv/MedRxiv preprints, arXiv materials-science and physical-chemistry categories, and material safety data sheets), ChemPile-LIFT and ChemPile-mLIFT (template-generated language-interfaced tabular data with SMILES/SELFIES/InChI/IUPAC representations and, for mLIFT, molecular images), ChemPile-Code (keyword-filtered StarCoder and CodeParrot), ChemPile-Reasoning (Stack Exchange Q&A plus LLM-distilled spectral-elucidation traces), and ChemPile-Caption (100K image-caption pairs from LibreTexts). The paper claims that ChemPile is the largest open curated chemical corpus at a scale suited to foundation-model pretraining, with expert-reviewed curation, consistent HuggingFace interfaces, and leakage-controlled splits. The headline arithmetic is internally consistent (Table 1 sums to roughly 255 GB and 76.7B tokens). The principal weakness is the quality control of the chemistry-filtered Paper subset (14.1B tokens), which rests on a classifier validated on only 150 out-of-domain annotations; this is the load-bearing issue examined below.
Significance. If the quality claims hold, ChemPile is a substantial community resource: no openly released chemistry corpus at this scale (about 75B tokens) exists, and the comparison with ChemDFM's unreleased 34B-token corpus makes that gap concrete. The paper ships reproducible infrastructure: curation scripts on GitHub, a documented sampling engine with 1,636 expert-reviewed templates, OPSIN-validated SMILES-to-IUPAC conversion, RDKit-graph-based verification of distilled reasoning traces, and a consistent HuggingFace API. The internal accounting is consistent across Table 1 and the appendix, and the multiple-representation design for identical molecules is a real contribution for representation studies. The main uncertainty is the curation quality of the Paper subset, which is the largest component of naturally occurring scientific prose and is filtered by a modestly validated classifier; the significance of the whole corpus is conditional on substantiating that filter on the target distribution. The diversity evidence in Figure 2b is suggestive rather than quantitative.
major comments (3)
- [§4.7, Appendix K, datasheets 'Data Splits'] The central 'curated chemical data' claim is load-bearing on the ChemPile-Paper subset (14.1B tokens, 11.7M documents), whose EuroPMC portion is selected by a BERT multilabel classifier trained on CAMEL data and validated on about 150 manually annotated entries from FineWebMath (F1 approximately 0.77). Three points make this validation insufficient for the claim. First, the validation set comes from a different distribution than the target corpus, so the reported F1 does not estimate precision on EuroPMC; with N = 150 the uncertainty is large (a 95% confidence interval on F1 = 0.77 spans roughly plus or minus 0.07), and F1 alone does not reveal the contamination rate, so precision and recall should be reported separately. Second, the manuscript does not state whether the ChemRxiv/BioRxiv/MedRxiv preprints that feed the same 14.1B-token subset pass through this or any chemistry filter; since BioRxiv and MedRxiv are broad biomedical servers, an unfiltered inclusion would be a large off-topic source. Third, the Appendix E datasheet itself concedes that 'some of the articles in the dataset might not include chemical research and only be related to chemistry.' The manuscript also reports '3.3 billion tokens' for the EuroPMC chemistry-filtered content while Table 1 lists 14.1B tokens for the whole Paper subset, so a per-source breakdown (documents and tokens for EuroPMC, each preprint server, arXiv, and MSDS) is needed to audit the total. I request: (i) classifier precision and recall on a held-out EuroPMC sample with confidence intervals; (ii) an explicit statement of the filtering applied to each Paper source; (iii) per-source size statistics; and (iv) release of the classifier and the 150-entry annotation set.
- [§4.3, Appendix P] The split documentation is internally inconsistent and does not support the leakage-prevention claim. Every datasheet reports train/validation/test ratios of 0.9, 0.1, and 0.1, which sum to 1.1; taken literally this is not a valid partition. In the Appendix K pseudocode, with train_fraction = 0.9 and val_fraction = 0.1, the test set (all_molecules_list[train_size + val_size:end]) is empty, and the fallback random-assignment branch can never assign rows to the test class because train_fraction + val_fraction = 1.0. In addition, Section 4.7 states that SMILES-based datasets are split by RDKit Murcko scaffold, but the provided 'scaffold splitting' pseudocode performs a plain shuffle-and-cut of a molecule list with no scaffold computation anywhere in the algorithm. As written, the protocol is not reproducible, and a user cannot tell whether the shipped splits are scaffold-based (leakage-controlled, as claimed) or random. Please provide corrected pseudocode that matches the released implementation, state the actual split fractions, and describe how scaffolds are assigned across the global molecule list.
- [§4.3, Appendix P] Appendix P reports approximately 91% accuracy for the SMILES-to-IUPAC model, but the manuscript does not specify the acceptance criterion used in the OPSIN-based 'automatic verification' of generated IUPAC names, nor the fraction of entries that were accepted or discarded. If verification only checks that OPSIN can parse the generated name (syntactic validity), names that parse to a molecule different from the intended one will pass; if it checks round-trip equivalence with the input SMILES, the residual error rate is much lower. Since ChemPile-mLIFT is the largest subset by size (155 GB), the correctness of its IUPAC field materially affects the corpus-wide quality claim. Please report the verification criterion (syntactic parse versus canonical equivalence to the source SMILES), the acceptance rate, and the post-verification error rate on a held-out sample.
minor comments (7)
- [§2.3, §3, Figure 2b] The diversity comparison does not specify which datasets are embedded, how many samples per dataset were used, or whether sampling was balanced; the conclusion that ChemPile 'spans a larger space' is qualitative, so please report the full dataset list, sample counts, and, ideally, a quantitative coverage or volume metric.
- [§1, §3, Appendix J] The text describes ChemPile as released under a 'permissive license,' but the actual licenses include CC BY-NC-SA 4.0 (Education, LIFT, mLIFT, Caption) and CC BY-NC-ND 4.0 (Paper), which are non-commercial and, for Paper, no-derivatives; this wording is misleading for potential commercial pretraining use and should be corrected.
- [§3, Figure 2a] Token counts are computed with different tokenizers across corpora (GPT-2 tiktoken for ChemPile versus undisclosed tokenizers for ChemDFM and BioGPT), so a caveat that the comparison is tokenizer-dependent would keep the 'more than 50% larger' statement within its uncertainty.
- [§N.4] The statement that 'no assessment was conducted regarding the model's adherence to instructed output formatting guidelines' should be reconciled with the claim that reasoning traces were parsed via the [START_REASONING]/[END_REASONING] tags; the appendix should also report the fraction of LLM generations discarded for failing parsing or correctness checks.
- [§4.2, reference [77]] The PaperScraper tool is cited to a drug-design review (Born and Manica, Current Medicinal Chemistry 2021); a direct citation of the PaperScraper repository or its accompanying paper would help readers locate the tool.
- [Appendix M] The representation-embedding correlation analysis uses OpenAI's text-embedding-3-large, which is not reproducible and was not trained on chemical text; the conclusion that IUPAC embeddings are closest to established similarity measures is model-dependent and should be flagged as such.
- [Appendix F (ChemPile-Code datasheet)] The Known Limitations entry refers to 'the accuracy of the classifier used to select the code,' but Section 4.4 describes regular-expression-based keyword filtering; the terminology should be aligned across the paper and the datasheet.
Circularity Check
No circularity: ChemPile is an assembly/curation paper whose central claims rest on external sources, documented construction pipelines, and independent validations.
full rationale
ChemPile is a dataset-construction paper, not a derivation of a predictive law. The central claims — token counts, scale comparisons, diversity, and curation quality — are supported by construction artifacts and external comparators. The LIFT and mLIFT subsets are generated by filling manually curated templates with external property datasets, and the SMILES-to-IUPAC conversion is checked against the independent OPSIN tool, so no output is equivalent to its input by construction. The only fitted model is the BERT classifier used to filter EuroPMC; its performance is reported as F1 ≈ 0.77 on roughly 150 manually annotated entries (Section 4.2 and Appendix N.3.1). That is a validation/quality concern — and the datasheet explicitly concedes that 'Some of the articles in the dataset might not include chemical research and only be related to chemistry' — but it is not circular: the classifier is not used to define the claimed result, and no quantity is predicted from a parameter fitted to that same quantity. Several self-citations appear (e.g., [3], [5], [24]), but only as contextual background or prompt-format precedent; none is load-bearing for the scale, diversity, or curation claims. No circular step can be exhibited from the paper's own text, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- EuroPMC document chunk size =
512 tokens, 50-token overlap
- Minimum caption length for ChemPile-Caption =
200 characters
- Minimum solution length for ChemPile-Education Olympiad =
250 characters
- Sampling engine MCQ option count =
2 to 5 options
assumptions (5)
- domain assumption Larger and more diverse pretraining corpora improve foundation model capabilities (scaling laws and data mixing studies).
- domain assumption The CAMEL-trained BERT classifier provides valid chemistry labels for filtering EuroPMC content.
- domain assumption LLM rewriting and synthetic reasoning generation preserve scientific accuracy.
- domain assumption Random splitting of amino-acid sequences and non-SMILES datasets prevents data leakage across ChemPile subsets.
- standard math Tanimoto similarity on molecular fingerprints is an appropriate ground truth for evaluating chemical representation embeddings.
Cite this review
Pith. "Pith review of ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models." pith.science (2026). https://pith.science/paper/37ZIMQEU
@misc{pith2026250512534,
author = {Pith},
title = {Pith review of: ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/37ZIMQEU}},
note = {Machine review of arXiv:2505.12534}
}
read the original abstract
Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality datasets that reflect the field's multifaceted nature. We present the ChemPile, an open dataset containing over 75 billion tokens of curated chemical data, specifically built for training and evaluating general-purpose models in the chemical sciences. The dataset mirrors the human learning journey through chemistry -- from educational foundations to specialized expertise -- spanning multiple modalities and content types including structured data in diverse chemical representations (SMILES, SELFIES, IUPAC names, InChI, molecular renderings), scientific and educational text, executable code, and chemical images. ChemPile integrates foundational knowledge (textbooks, lecture notes), specialized expertise (scientific articles and language-interfaced data), visual understanding (molecular structures, diagrams), and advanced reasoning (problem-solving traces and code) -- mirroring how human chemists develop expertise through diverse learning materials and experiences. Constructed through hundreds of hours of expert curation, the ChemPile captures both foundational concepts and domain-specific complexity. We provide standardized training, validation, and test splits, enabling robust benchmarking. ChemPile is openly released via HuggingFace with a consistent API, permissive license, and detailed documentation. We hope the ChemPile will serve as a catalyst for chemical AI, enabling the development of the next generation of chemical foundation models.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 3 Pith papers
-
Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
A ModernBERT-based encoder trained with masked language modeling on SMILES-annotated scientific documents plus a contrastive stage yields embeddings that are competitive on both molecular property prediction and scien...
-
SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation
SciCodePile contributes a 128GB scientific code corpus and a 200-task executable benchmark showing that current LLMs still fall far short of reliable scientific code generation.
-
Molecular Machine Learning in Chemical Process Design
This paper argues that integrating molecular machine learning into chemical process design could accelerate discovery of novel molecules and processes, but requires better data, benchmarks, and industry collaboration.
Reference graph
Works this paper leans on
-
[1]
The future of chemistry is language
Andrew D White. “The future of chemistry is language”. In: Nature Reviews Chemistry 7.7 (2023), pp. 457–458
2023
-
[2]
A review of large language models and autonomous agents in chemistry
Mayk Caldas Ramos, Christopher J. Collison, and Andrew D. White. “A review of large language models and autonomous agents in chemistry”. In: Chemical Science 16.6 (2025), pp. 2514–2572. ISSN : 2041-6539. DOI: 10.1039/d4sc03921a. URL: http://dx.doi.org/ 10.1039/D4SC03921A
-
[3]
Kevin Maik Jablonka et al. “14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon”. In: Digital Discovery 2.5 (2023), pp. 1233–1250. ISSN : 2635-098X. DOI: 10 . 1039 / d3dd00113j. URL: http : //dx.doi.org/10.1039/D3DD00113J
-
[4]
Machine learning for a sustainable energy future
Zhenpeng Yao et al. “Machine learning for a sustainable energy future”. In:Nature Reviews Materials 8.3 (Oct. 2022), pp. 202–215. ISSN : 2058-8437. DOI: 10.1038/s41578- 022- 00490-5. URL: http://dx.doi.org/10.1038/s41578-022-00490-5
doi:10.1038/s41578- 2022
-
[5]
Are large language models superhuman chemists?
Adrian Mirza et al. “Are large language models superhuman chemists?” In: arXiv preprint arXiv: 2404.01475 (2024)
arXiv 2024
-
[6]
Language agents achieve superhuman synthesis of scientific knowledge
Michael D. Skarlinski et al. “Language agents achieve superhuman synthesis of scientific knowledge”. In: arXiv preprint arXiv: 2409.13740 (2024)
arXiv 2024
-
[7]
Markus J. Buehler. “MechGPT, a Language-Based Strategy for Mechanics and Materials Modeling That Connects Knowledge Across Scales, Disciplines, and Modalities”. In: Applied Mechanics Reviews 76.2 (Jan. 2024). ISSN : 2379-0407. DOI: 10.1115/1.4063843 . URL: http://dx.doi.org/10.1115/1.4063843
-
[8]
Can Large Language Models Empower Molecular Property Prediction?
Chen Qian et al. “Can Large Language Models Empower Molecular Property Prediction?” In: arXiv preprint arXiv: 2307.07443 (2023)
arXiv 2023
Show all 89 references
-
[9]
Leveraging large language models for predictive chemistry
Kevin Maik Jablonka et al. “Leveraging large language models for predictive chemistry”. In: Nature Machine Intelligence 6.2 (2024), pp. 161–169
2024
-
[10]
Benchmarking Large Language Models for Molecule Prediction Tasks
Zhiqiang Zhong, Kuangyu Zhou, and Davide Mottin. “Benchmarking Large Language Models for Molecule Prediction Tasks”. In: arXiv preprint arXiv: 2403.05075 (2024)
2024 arXiv
-
[11]
LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions
Andre Niyongabo Rubungo et al. “LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions”. In: arXiv preprint arXiv: 2310.14029 (2023). 10
2023 arXiv
-
[12]
MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction
Yuyan Liu et al. “MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction”. In: arXiv preprint arXiv: 2406.12950 (2024)
2024 arXiv
-
[13]
Crystal structure generation with autoregressive large language modeling
Luis M. Antunes, Keith T. Butler, and Ricardo Grau-Crespo. “Crystal structure generation with autoregressive large language modeling”. In: Nature Communications 15.1 (Dec. 2024). ISSN : 2041-1723. DOI: 10.1038/s41467-024-54639-7 . URL: http://dx.doi.org/10. 1038/s41467-024-54639-7
2024 doi
-
[14]
Large Language Models Are Innate Crystal Structure Generators
Jingru Gan et al. “Large Language Models Are Innate Crystal Structure Generators”. In: arXiv preprint arXiv: 2502.20933 (2025)
2025
-
[15]
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Nate Gruver et al. “Fine-Tuned Language Models Generate Stable Inorganic Materials as Text”. In: arXiv preprint arXiv: 2402.04379 (2024)
2024 arXiv
-
[16]
Fleming: An AI Agent for Antibiotic Discovery in Mycobacterium tubercu- losis
Ziming Wei et al. “Fleming: An AI Agent for Antibiotic Discovery in Mycobacterium tubercu- losis”. In: (Apr. 2025). DOI: 10.1101/2025.04.01.646719. URL: http://dx.doi.org/ 10.1101/2025.04.01.646719
2025 doi
-
[17]
Augmenting large language models with chemistry tools
Andres M. Bran et al. “Augmenting large language models with chemistry tools”. In: Nature Machine Intelligence 6.5 (May 2024), pp. 525–535. ISSN : 2522-5839. DOI: 10.1038/s42256- 024-00832-8. URL: http://dx.doi.org/10.1038/s42256-024-00832-8
2024 doi
-
[18]
ORGANA: A robotic assistant for automated chemistry experimen- tation and characterization
Kourosh Darvish et al. “ORGANA: A robotic assistant for automated chemistry experimen- tation and characterization”. In: Matter 8.2 (Feb. 2025), p. 101897. ISSN : 2590-2385. DOI: 10.1016/j.matt.2024.10.015 . URL: http://dx.doi.org/10.1016/j.matt.2024. 10.015
2025 doi
-
[19]
Autonomous chemical research with large language models
Daniil A Boiko et al. “Autonomous chemical research with large language models”. In:Nature 624.7992 (2023), pp. 570–578
2023
-
[20]
How should the advancement of large language models affect the practice of science?
Marcel Binz et al. “How should the advancement of large language models affect the practice of science?” In: Proceedings of the National Academy of Sciences 122.5 (Jan. 2025). ISSN : 1091-6490. DOI: 10.1073/pnas.2401227121 . URL: http://dx.doi.org/10.1073/ pnas.2401227121
2025 doi
-
[21]
Generative Retrieval-Augmented Ontologic Graph and Multiagent Strate- gies for Interpretive Large Language Model-Based Materials Design
Markus J. Buehler. “Generative Retrieval-Augmented Ontologic Graph and Multiagent Strate- gies for Interpretive Large Language Model-Based Materials Design”. In: ACS Engineering Au 4.2 (Jan. 2024), pp. 241–277. ISSN : 2694-2488. DOI: 10 . 1021 / acsengineeringau . 3c00058. URL...
2024 doi
-
[22]
An Interdisciplinary Outlook on Large Language Models for Scientific Research
James Boyko et al. “An Interdisciplinary Outlook on Large Language Models for Scientific Research”. In: arXiv preprint arXiv: 2311.04929 (2023)
2023 arXiv
-
[23]
SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules
David Weininger. “SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules”. In: Journal of chemical information and computer sciences 28.1 (1988), pp. 31–36
1988
-
[24]
Probing the limitations of multimodal language models for chemistry and materials research
Nawaf Alampara et al. “Probing the limitations of multimodal language models for chemistry and materials research”. In: arXiv preprint arXiv: 2411.16955 (2024)
2024 arXiv
-
[25]
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learn- ing Tasks
Tuan Dinh et al. “LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learn- ing Tasks”. In: arXiv preprint arXiv: 2206.06565 (2022)
2022 arXiv
-
[26]
At Which Training Stage Does Code Data Help LLMs Reasoning?
Yingwei Ma et al. “At Which Training Stage Does Code Data Help LLMs Reasoning?” In:The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL: https://openreview.net/forum?id= KIPJKST4gw
2024
-
[27]
A Survey on Data Selection for Language Models
Alon Albalak et al. “A Survey on Data Selection for Language Models”. In: arXiv preprint arXiv: 2402.16827 (2024)
2024 arXiv
-
[28]
The pile: An 800gb dataset of diverse text for language modeling
Leo Gao et al. “The pile: An 800gb dataset of diverse text for language modeling”. In: arXiv preprint arXiv:2101.00027 (2020)
2020 arXiv
-
[29]
Exploring the limits of transfer learning with a unified text-to-text trans- former
Colin Raffel et al. “Exploring the limits of transfer learning with a unified text-to-text trans- former”. In: Journal of machine learning research21.140 (2020), pp. 1–67
2020
-
[30]
Training Compute-Optimal Large Language Models
Jordan Hoffmann et al. “Training Compute-Optimal Large Language Models”. In: arXiv preprint arXiv: 2203.15556 (2022)
2022 arXiv
-
[31]
Scaling laws for neural language models
Jared Kaplan et al. “Scaling laws for neural language models”. In: arXiv preprint arXiv:2001.08361 (2020)
2020 arXiv
-
[32]
When less is more: Investigating data pruning for pretraining llms at scale
Max Marion et al. “When less is more: Investigating data pruning for pretraining llms at scale”. In: arXiv preprint arXiv:2309.04564 (2023). 11
2023 arXiv
-
[33]
Textbooks are all you need
Suriya Gunasekar et al. “Textbooks are all you need”. In: arXiv preprint arXiv:2306.11644 (2023)
2023 arXiv
-
[34]
Datacomp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre et al. “Datacomp: In search of the next generation of multimodal datasets”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 27092–27112
2023
-
[35]
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Guilherme Penedo et al. “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15...
2024
-
[36]
Dolma: An open corpus of three trillion tokens for language model pretraining research
Luca Soldaini et al. “Dolma: An open corpus of three trillion tokens for language model pretraining research”. In: arXiv preprint arXiv:2402.00159 (2024)
2024 arXiv
-
[37]
Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two- Phase Pretraining
Steven Feng et al. “Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two- Phase Pretraining”. In: arXiv preprint arXiv: 2412.15285 (2024)
2024 arXiv
-
[38]
LAION-5B: An Open Large-Scale Dataset for Training next Generation Image-Text Models
Christoph Schuhmann et al. “LAION-5B: An Open Large-Scale Dataset for Training next Generation Image-Text Models”. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo et al. V ol. 35. Curran Associates, Inc., 2022, pp. 25278–25294
2022
-
[39]
MoleculeNet: a benchmark for molecular machine learning
Zhenqin Wu et al. “MoleculeNet: a benchmark for molecular machine learning”. In: Chemical science 9.2 (2018), pp. 513–530
2018
-
[40]
Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development
Kexin Huang et al. “Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development”. In: Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtu...
2021
-
[41]
PubChem Substance and Compound databases
Sunghwan Kim et al. “PubChem Substance and Compound databases”. In: Nucleic Acids Research 44.D1 (Sept. 2015), pp. D1202–D1213. ISSN : 1362-4962. DOI: 10.1093/nar/ gkv951. URL: http://dx.doi.org/10.1093/nar/gkv951
2015 doi
-
[42]
UniProt: a worldwide hub of protein knowledge
UniProt Consortium. “UniProt: a worldwide hub of protein knowledge”. In: Nucleic acids research 47.D1 (2019), pp. D506–D515
2019
-
[43]
Evaluating Chemistry Prompts for Large-Language Model Fine- Tuning
Carmelo Gonzales et al. “Evaluating Chemistry Prompts for Large-Language Model Fine- Tuning”. In: AI for Accelerated Materials Design-NeurIPS. 2024
2024
-
[44]
Quantum chemistry structures and properties of 134 kilo molecules
Raghunathan Ramakrishnan et al. “Quantum chemistry structures and properties of 134 kilo molecules”. In: Scientific Data 1.1 (Aug. 2014). ISSN : 2052-4463. DOI: 10.1038/sdata. 2014.22. URL: http://dx.doi.org/10.1038/sdata.2014.22
2014 doi
-
[45]
Chemical reactions from US patents (1976-Sep2016)
Daniel Lowe. “Chemical reactions from US patents (1976-Sep2016)”. In: (June 2017). DOI: 10 . 6084 / m9 . figshare . 5104873 . v1. URL: https : / / figshare . com / articles / dataset/Chemical_reactions_from_US_patents_1976-Sep2016_/5104873
1976
-
[46]
Big Data from Pharmaceutical Patents: A Computational Analysis of Medicinal Chemists’ Bread and Butter
Nadine Schneider et al. “Big Data from Pharmaceutical Patents: A Computational Analysis of Medicinal Chemists’ Bread and Butter”. In: Journal of Medicinal Chemistry 59.9 (Apr. 2016), pp. 4385–4402. ISSN : 1520-4804. DOI: 10.1021/acs.jmedchem.6b00153 . URL: http://dx.doi.org/10...
2016 doi
-
[47]
Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis
Xiwen Jia et al. “Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis”. In: Nature 573.7773 (Sept. 2019), pp. 251–255. ISSN : 1476-4687. DOI: 10.1038/ s41586-019-1540-5 . URL: http://dx.doi.org/10.1038/s41586-019-1540-5
2019 doi
-
[48]
Machine-learning-assisted materials discovery using failed experiments
Paul Raccuglia et al. “Machine-learning-assisted materials discovery using failed experiments”. In: Nature 533.7601 (May 2016), pp. 73–76. ISSN : 1476-4687. DOI: 10.1038/nature17439. URL: http://dx.doi.org/10.1038/nature17439
2016 doi
-
[49]
Data-Driven Matching of Experimental Crystal Structures and Gas Adsorption Isotherms of Metal–Organic Frameworks
Daniele Ongari et al. “Data-Driven Matching of Experimental Crystal Structures and Gas Adsorption Isotherms of Metal–Organic Frameworks”. In: Journal of Chemical & Engi- neering Data 67.7 (Feb. 2022), pp. 1743–1756. ISSN : 1520-5134. DOI: 10.1021/acs.jced. 1c00958. URL: ht...
2022 doi
-
[50]
SELFIES and the future of molecular string representations
Mario Krenn et al. “SELFIES and the future of molecular string representations”. In: Patterns 3.10 (Oct. 2022), p. 100588. ISSN : 2666-3899. DOI: 10.1016/j.patter.2022.100588 . URL: http://dx.doi.org/10.1016/j.patter.2022.100588. 12
2022
-
[51]
Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models
Yin Fang et al. “Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models”. In: International Conference on Learning Representations (2023). DOI: 10.48550/arXiv.2306.08018
-
[52]
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Bing Su et al. “A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language”. In: arXiv preprint arXiv: 2209.05481 (2022)
2022 arXiv
-
[53]
Multi-modal molecule structure–text model for text-based retrieval and editing
Shengchao Liu et al. “Multi-modal molecule structure–text model for text-based retrieval and editing”. In: Nature Machine Intelligence 5.12 (Dec. 2023), pp. 1447–1457. ISSN : 2522-5839. DOI: 10.1038/s42256-023-00759-6 . URL: http://dx.doi.org/10.1038/s42256- 023-00759-6
2023 doi
-
[54]
Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning
Gang Liu et al. “Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning”. In: arXiv preprint arXiv: 2410.04223 (2024)
2024 arXiv
-
[55]
Multimodal Learning for Materials
Viggo Moro et al. “Multimodal Learning for Materials”. In: arXiv preprint arXiv:2312.00111 (2023)
2023 arXiv
-
[56]
ChemDFM: A Large Language Foundation Model for Chemistry
Zihan Zhao et al. “ChemDFM: A Large Language Foundation Model for Chemistry”. In:arXiv preprint arXiv:2401.14818 (2024)
2024 arXiv
-
[57]
Wudaocorpora: A super large-scale chinese corpora for pre-training language models
Sha Yuan et al. “Wudaocorpora: A super large-scale chinese corpora for pre-training language models”. In: AI Open 2 (2021), pp. 65–68
2021
-
[58]
LlaSMol: Advancing Large Language Models for Chemistry with a Large- Scale, Comprehensive, High-Quality Instruction Tuning Dataset
Botao Yu et al. “LlaSMol: Advancing Large Language Models for Chemistry with a Large- Scale, Comprehensive, High-Quality Instruction Tuning Dataset”. In: arXiv preprint arXiv: 2402.09391 (2024)
2024 arXiv
-
[59]
Enhancing Chemical Reaction and Retrosynthesis Prediction with Large Language Model and Dual-task Learning
Xuan Lin et al. “Enhancing Chemical Reaction and Retrosynthesis Prediction with Large Language Model and Dual-task Learning”. In: arXiv preprint arXiv: 2505.02639 (2025)
2025 arXiv
-
[60]
BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction
Yifei Yang et al. “BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction”. In: arXiv preprint arXiv: 2408.10285 (2024)
2024 arXiv
-
[61]
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Renqian Luo et al. “BioGPT: generative pre-trained transformer for biomedical text generation and mining”. In: Briefings in bioinformatics 23.6 (2022), bbac409
2022
-
[62]
Neural scaling of deep chemical models
Nathan C Frey et al. “Neural scaling of deep chemical models”. In:Nature Machine Intelligence 5.11 (2023), pp. 1297–1305
2023
-
[63]
Darwin 1.5: Large language models as materials science adapted learners
Tong Xie et al. “Darwin 1.5: Large language models as materials science adapted learners”. In: arXiv preprint arXiv:2412.11970 (2024)
2024 arXiv
-
[64]
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Amanpreet Singh et al. “SciRepEval: A Multi-Format Benchmark for Scientific Document Representations”. In: Conference on Empirical Methods in Natural Language Processing
-
[65]
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”. In: arXiv preprint arXiv: 2112.11446 (2021)
2021 arXiv
-
[66]
Data mixing laws: Optimizing data mixtures by predicting language modeling performance
Jiasheng Ye et al. “Data mixing laws: Optimizing data mixtures by predicting language modeling performance”. In: arXiv preprint arXiv:2403.16952 (2024)
2024 arXiv
-
[67]
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation
Mario Krenn et al. “Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation”. In: Machine Learning: Science and Technology1.4 (2020), p. 045024
2020
-
[68]
InChI, the IUPAC international chemical identifier
Stephen R Heller et al. “InChI, the IUPAC international chemical identifier”. In:Journal of cheminformatics 7 (2015), pp. 1–34
2015
-
[69]
RanDepict: Random chemical structure depiction generator
Henning Otto Brinkhaus et al. “RanDepict: Random chemical structure depiction generator”. In: Journal of cheminformatics 14.1 (2022), p. 31
2022
-
[70]
MatText: Do language models need more than text & scale for materials modeling?
Nawaf Alampara, Santiago Miret, and Kevin Maik Jablonka. “MatText: Do language models need more than text & scale for materials modeling?” In: arXiv preprint arXiv:2406.17295 (2024)
2024 arXiv
-
[71]
ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. “ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction”. In: arXiv preprint arXiv: 2010.09885 (2020)
2020 arXiv
-
[72]
Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?
Dávid Bajusz, Anita Rácz, and Károly Héberger. “Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?” In: Journal of cheminformatics 7 (2015), pp. 1–13
2015
-
[73]
tiktoken: A fast BPE tokenizer for use with OpenAI’s models
OpenAI. tiktoken: A fast BPE tokenizer for use with OpenAI’s models . https://github. com/openai/tiktoken. Accessed: 2025-05-14. 2022
2025
-
[74]
Europe PMC in 2023
Summer Rosonovski et al. “Europe PMC in 2023”. In: Nucleic Acids Research 52.D1 (2024), pp. D1668–D1676. 13
2024
-
[75]
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society
Guohao Li et al. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. 2023. arXiv: 2303.17760 [cs.AI]
2023 arXiv
-
[76]
SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model
Loubna Ben Allal et al. SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model. 2025. arXiv: 2502.02737 [cs.CL] . URL: https://arxiv.org/abs/ 2502.02737
2025 arXiv
-
[77]
Trends in Deep Learning for Property-driven Drug Design
Jannis Born and Matteo Manica. “Trends in Deep Learning for Property-driven Drug Design”. In: Current Medicinal Chemistry 28.38 (2021), pp. 7862–7886
2021
-
[79]
Chemical name to structure: OPSIN, an open source solution
Daniel M Lowe et al. Chemical name to structure: OPSIN, an open source solution. 2011
2011
-
[80]
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Viraat Aryabumi et al. “To Code, or Not To Code? Exploring Impact of Code in Pre-training”. In: arXiv preprint arXiv: 2408.10914 (2024)
2024 arXiv
-
[81]
StarCoder: may the source be with you!
Raymond Li et al. “StarCoder: may the source be with you!” In: Trans. Mach. Learn. Res. 2023 (2023). URL: https://openreview.net/forum?id=KoFOg41haE
2023
- [82]
-
[83]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning
DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning”. In: arXiv preprint arXiv: 2501.12948 (2025)
2025 arXiv
-
[84]
The properties of known drugs. 1. Molecular frameworks
Guy W Bemis and Mark A Murcko. “The properties of known drugs. 1. Molecular frameworks”. In: Journal of medicinal chemistry 39.15 (1996), pp. 2887–2893
1996
-
[85]
Nougat: Neural optical understanding for academic documents
Lukas Blecher et al. “Nougat: Neural optical understanding for academic documents”. In: arXiv preprint arXiv:2308.13418 (2023)
2023 arXiv
-
[86]
Neural Machine Translation of Rare Words with Subword Units
Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural Machine Translation of Rare Words with Subword Units”. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Katrin Erk and Noah A. Smith. Berlin, Ge...
2016 doi
-
[87]
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Taku Kudo. “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”. In: Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Pa...
2018
-
[89]
Open-ended questions, requiring free-form generation of the correct SMILES string
-
[90]
jablonkagroup/chempile-caption
Multiple-choice questions (MCQs), where the model selected the correct answer from structural isomers of the target compound. For the multi-spectra dataset, we maintained consistency by employing the same SMILES formatting protocol and evaluation methodology as in the single-s...
-
[2022]
URL: https://api.semanticscholar.org/CorpusID:254018137
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.