Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read ChemPile gathers over 75 billion tokens of curated chemical data into an open corpus built for training chemical foundation models.

desk verdict ChemPile is a genuinely large, openly released chemical corpus that the field will likely use, but the quality of its largest paper subset rests on a classifier validated on 150 examples from a different domain; the paper should be published only after the authors either tighten that validation or soften the curation claim. read the letter →

arxiv 2505.12534 v1 pith:37ZIMQEU submitted 2025-05-18 cs.LG

classification cs.LG
keywords chemicalfoundationmodelslargelanguagedatasetcurationmultimodallearningSMILESSELFIESIUPACnamesmolecularrepresentations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ChemPile, an open dataset of about 255 GB and 76.7 billion tokens assembled for training and evaluating general-purpose foundation models in chemistry. It combines seven complementary subsets — educational texts, chemistry-filtered research papers, language-interfaced tabular data in text and image form, chemistry-related code, reasoning traces, and image-caption pairs — so that a model can learn chemistry the way a chemist does, from foundational concepts to specialized reasoning and multimodal problem solving. The authors claim this is the largest openly available curated chemical corpus and the only one at a scale suited to foundation-model pretraining. If the corpus is as clean and diverse as reported, it would let the community study data mixing, molecular representation choice, and scaling behavior in chemistry on reproducible, permissively licensed data with consistent train/validation/test splits.

What carries the argument

The load-bearing object is ChemPile itself, built as reproducible infrastructure rather than a single static file. Three mechanisms carry the argument: a transformer-based text classifier that filters a large literature corpus down to chemistry-related papers; a sampling engine that converts tabular chemistry datasets into natural-language questions by filling 1,636 expert-written templates with randomized synonyms, enumeration schemes, and multiple-choice options, then expands every entry into SMILES, SELFIES, InChI, IUPAC names, and images; and a split protocol that assigns every molecule to a global scaffold-based train/validation/test partition so the same molecular core never appears in both training and evaluation.

What would settle it

Draw a random sample of about 1,000 documents from the Paper subset and have two independent chemists label each as chemistry research or not; if the measured precision falls well short of the approximately 0.77 F1 reported, or if a large fraction of the corpus's biggest text subsets is clearly off-topic, the curated-chemical-corpus claim is contradicted.

Watch

Extended reading notes

Core claim

ChemPile's central claim is that a single, openly released corpus can supply the volume, diversity, and quality that chemical foundation models have been missing. The dataset spans seven subsets built from very different sources: textbooks and lecture transcripts; research articles filtered from large abstract and full-text corpora; tabular chemistry datasets converted into natural language through 1,636 hand-written templates; those same tabular data expanded into multiple molecular representations and rendered molecular images; code filtered from large permissively licensed code corpora; community question-answer data; and synthetic reasoning traces for interpreting molecular spectra. Totaling about 255 GB, 76.7 billion tokens, and 260 million documents, the corpus is, the paper states, larger than the 34-billion-token corpus behind the largest previously reported chemical foundation model and orders of magnitude larger than released chemical instruction datasets. The paper also provides scaffold-based splits designed to keep the same molecular scaffold out of both training and test partitions, and it reports that embeddings of IUPAC names track molecular similarity more closely (r=0.722) than embeddings of SMILES strings (r=0.521), evidence that representation choice matters for how well the corpus can teach chemistry.

Load-bearing premise

The entire 'curated chemical corpus' claim rests on the assumption that the classifier used to select the paper subset is accurate enough, but it was validated on only about 150 manually labeled examples with an F1 near 0.77, so a modest drop in precision would admit substantial non-chemical text into the 14.1-billion-token Paper subset.

Editorial extensions

If this is right

  • Chemical foundation models can be pretrained entirely on open data at a scale (roughly 76.7 billion tokens) that was previously available only to much smaller or closed chemical datasets.
  • Because the same molecules appear in SMILES, SELFIES, IUPAC, InChI, and rendered images, researchers can directly test which representation or combination transfers best to property prediction and inverse design.
  • The scaffold-based splits make benchmark results comparable across labs by preventing the same molecular scaffold from appearing in both training and test partitions.
  • The modular subsets support data-mixing studies, including how much code, reasoning-trace, or image-caption data improves chemical reasoning and multimodal understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the scale claim holds, chemical benchmarks should show scaling-law-like log-linear gains in downstream accuracy as ChemPile token counts grow; the paper does not itself present such scaling curves.
  • The reported IUPAC-embedding advantage suggests a single-representation ablation could show that IUPAC-heavy pretraining beats SMILES-only training for property prediction, but that model comparison is not in the paper.
  • Because much of the reasoning subset was generated by LLMs, models trained on ChemPile may inherit model-specific spectral-assignment errors; comparing downstream reasoning accuracy with and without the synthetic traces would separate distillation gains from distillation noise.
  • The template-sampling engine is domain-agnostic in design, so the same curation recipe could convert tabular data from other sciences into language-interfaced instruction data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. ChemPile is an open, multimodal chemical corpus reported as 255 GB of compressed Parquet data (76.7B GPT-2 tokens, 260M documents) assembled from seven subsets: ChemPile-Education (LibreTexts textbooks, MIT OCW materials, YouTube lecture transcripts, US Olympiad problems), ChemPile-Paper (chemistry-filtered EuroPMC literature, ChemRxiv/BioRxiv/MedRxiv preprints, arXiv materials-science and physical-chemistry categories, and material safety data sheets), ChemPile-LIFT and ChemPile-mLIFT (template-generated language-interfaced tabular data with SMILES/SELFIES/InChI/IUPAC representations and, for mLIFT, molecular images), ChemPile-Code (keyword-filtered StarCoder and CodeParrot), ChemPile-Reasoning (Stack Exchange Q&A plus LLM-distilled spectral-elucidation traces), and ChemPile-Caption (100K image-caption pairs from LibreTexts). The paper claims that ChemPile is the largest open curated chemical corpus at a scale suited to foundation-model pretraining, with expert-reviewed curation, consistent HuggingFace interfaces, and leakage-controlled splits. The headline arithmetic is internally consistent (Table 1 sums to roughly 255 GB and 76.7B tokens). The principal weakness is the quality control of the chemistry-filtered Paper subset (14.1B tokens), which rests on a classifier validated on only 150 out-of-domain annotations; this is the load-bearing issue examined below.

Significance. If the quality claims hold, ChemPile is a substantial community resource: no openly released chemistry corpus at this scale (about 75B tokens) exists, and the comparison with ChemDFM's unreleased 34B-token corpus makes that gap concrete. The paper ships reproducible infrastructure: curation scripts on GitHub, a documented sampling engine with 1,636 expert-reviewed templates, OPSIN-validated SMILES-to-IUPAC conversion, RDKit-graph-based verification of distilled reasoning traces, and a consistent HuggingFace API. The internal accounting is consistent across Table 1 and the appendix, and the multiple-representation design for identical molecules is a real contribution for representation studies. The main uncertainty is the curation quality of the Paper subset, which is the largest component of naturally occurring scientific prose and is filtered by a modestly validated classifier; the significance of the whole corpus is conditional on substantiating that filter on the target distribution. The diversity evidence in Figure 2b is suggestive rather than quantitative.

major comments (3)
  1. [§4.7, Appendix K, datasheets 'Data Splits'] The central 'curated chemical data' claim is load-bearing on the ChemPile-Paper subset (14.1B tokens, 11.7M documents), whose EuroPMC portion is selected by a BERT multilabel classifier trained on CAMEL data and validated on about 150 manually annotated entries from FineWebMath (F1 approximately 0.77). Three points make this validation insufficient for the claim. First, the validation set comes from a different distribution than the target corpus, so the reported F1 does not estimate precision on EuroPMC; with N = 150 the uncertainty is large (a 95% confidence interval on F1 = 0.77 spans roughly plus or minus 0.07), and F1 alone does not reveal the contamination rate, so precision and recall should be reported separately. Second, the manuscript does not state whether the ChemRxiv/BioRxiv/MedRxiv preprints that feed the same 14.1B-token subset pass through this or any chemistry filter; since BioRxiv and MedRxiv are broad biomedical servers, an unfiltered inclusion would be a large off-topic source. Third, the Appendix E datasheet itself concedes that 'some of the articles in the dataset might not include chemical research and only be related to chemistry.' The manuscript also reports '3.3 billion tokens' for the EuroPMC chemistry-filtered content while Table 1 lists 14.1B tokens for the whole Paper subset, so a per-source breakdown (documents and tokens for EuroPMC, each preprint server, arXiv, and MSDS) is needed to audit the total. I request: (i) classifier precision and recall on a held-out EuroPMC sample with confidence intervals; (ii) an explicit statement of the filtering applied to each Paper source; (iii) per-source size statistics; and (iv) release of the classifier and the 150-entry annotation set.
  2. [§4.3, Appendix P] The split documentation is internally inconsistent and does not support the leakage-prevention claim. Every datasheet reports train/validation/test ratios of 0.9, 0.1, and 0.1, which sum to 1.1; taken literally this is not a valid partition. In the Appendix K pseudocode, with train_fraction = 0.9 and val_fraction = 0.1, the test set (all_molecules_list[train_size + val_size:end]) is empty, and the fallback random-assignment branch can never assign rows to the test class because train_fraction + val_fraction = 1.0. In addition, Section 4.7 states that SMILES-based datasets are split by RDKit Murcko scaffold, but the provided 'scaffold splitting' pseudocode performs a plain shuffle-and-cut of a molecule list with no scaffold computation anywhere in the algorithm. As written, the protocol is not reproducible, and a user cannot tell whether the shipped splits are scaffold-based (leakage-controlled, as claimed) or random. Please provide corrected pseudocode that matches the released implementation, state the actual split fractions, and describe how scaffolds are assigned across the global molecule list.
  3. [§4.3, Appendix P] Appendix P reports approximately 91% accuracy for the SMILES-to-IUPAC model, but the manuscript does not specify the acceptance criterion used in the OPSIN-based 'automatic verification' of generated IUPAC names, nor the fraction of entries that were accepted or discarded. If verification only checks that OPSIN can parse the generated name (syntactic validity), names that parse to a molecule different from the intended one will pass; if it checks round-trip equivalence with the input SMILES, the residual error rate is much lower. Since ChemPile-mLIFT is the largest subset by size (155 GB), the correctness of its IUPAC field materially affects the corpus-wide quality claim. Please report the verification criterion (syntactic parse versus canonical equivalence to the source SMILES), the acceptance rate, and the post-verification error rate on a held-out sample.
minor comments (7)
  1. [§2.3, §3, Figure 2b] The diversity comparison does not specify which datasets are embedded, how many samples per dataset were used, or whether sampling was balanced; the conclusion that ChemPile 'spans a larger space' is qualitative, so please report the full dataset list, sample counts, and, ideally, a quantitative coverage or volume metric.
  2. [§1, §3, Appendix J] The text describes ChemPile as released under a 'permissive license,' but the actual licenses include CC BY-NC-SA 4.0 (Education, LIFT, mLIFT, Caption) and CC BY-NC-ND 4.0 (Paper), which are non-commercial and, for Paper, no-derivatives; this wording is misleading for potential commercial pretraining use and should be corrected.
  3. [§3, Figure 2a] Token counts are computed with different tokenizers across corpora (GPT-2 tiktoken for ChemPile versus undisclosed tokenizers for ChemDFM and BioGPT), so a caveat that the comparison is tokenizer-dependent would keep the 'more than 50% larger' statement within its uncertainty.
  4. [§N.4] The statement that 'no assessment was conducted regarding the model's adherence to instructed output formatting guidelines' should be reconciled with the claim that reasoning traces were parsed via the [START_REASONING]/[END_REASONING] tags; the appendix should also report the fraction of LLM generations discarded for failing parsing or correctness checks.
  5. [§4.2, reference [77]] The PaperScraper tool is cited to a drug-design review (Born and Manica, Current Medicinal Chemistry 2021); a direct citation of the PaperScraper repository or its accompanying paper would help readers locate the tool.
  6. [Appendix M] The representation-embedding correlation analysis uses OpenAI's text-embedding-3-large, which is not reproducible and was not trained on chemical text; the conclusion that IUPAC embeddings are closest to established similarity measures is model-dependent and should be flagged as such.
  7. [Appendix F (ChemPile-Code datasheet)] The Known Limitations entry refers to 'the accuracy of the classifier used to select the code,' but Section 4.4 describes regular-expression-based keyword filtering; the terminology should be aligned across the paper and the datasheet.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ChemPile is an assembly/curation paper whose central claims rest on external sources, documented construction pipelines, and independent validations.

full rationale

ChemPile is a dataset-construction paper, not a derivation of a predictive law. The central claims — token counts, scale comparisons, diversity, and curation quality — are supported by construction artifacts and external comparators. The LIFT and mLIFT subsets are generated by filling manually curated templates with external property datasets, and the SMILES-to-IUPAC conversion is checked against the independent OPSIN tool, so no output is equivalent to its input by construction. The only fitted model is the BERT classifier used to filter EuroPMC; its performance is reported as F1 ≈ 0.77 on roughly 150 manually annotated entries (Section 4.2 and Appendix N.3.1). That is a validation/quality concern — and the datasheet explicitly concedes that 'Some of the articles in the dataset might not include chemical research and only be related to chemistry' — but it is not circular: the classifier is not used to define the claimed result, and no quantity is predicted from a parameter fitted to that same quantity. Several self-citations appear (e.g., [3], [5], [24]), but only as contextual background or prompt-format precedent; none is load-bearing for the scale, diversity, or curation claims. No circular step can be exhibited from the paper's own text, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No fitted constants enter a predictive equation; the curation thresholds are methodological choices, not free parameters. The axioms are the background beliefs the paper relies on to make the dataset's value claim: that scale and diversity improve models, that the trained classifier filters correctly, that LLM-generated content is factually acceptable, that the split strategy prevents leakage, and that Tanimoto similarity is a valid reference for representation quality. No new physical or conceptual entities are introduced; everything is derived from existing data or tools.

free parameters (4)
  • EuroPMC document chunk size = 512 tokens, 50-token overlap
    Chosen by hand in Appendix N.3.1 for the BERT classifier; affects which documents are classified as chemistry and therefore the composition of the 14.1B-token Paper subset.
  • Minimum caption length for ChemPile-Caption = 200 characters
    Chosen by hand in Section 4.6 to exclude images with little text; determines the size and quality of the caption subset.
  • Minimum solution length for ChemPile-Education Olympiad = 250 characters
    Chosen by hand in Section 4.1 to filter short Olympiad solutions; determines part of the Education subset content.
  • Sampling engine MCQ option count = 2 to 5 options
    Hand-chosen randomization range in Appendix L for multiple-choice generation; shapes the LIFT/mLIFT task distribution.
assumptions (5)
  • domain assumption Larger and more diverse pretraining corpora improve foundation model capabilities (scaling laws and data mixing studies).
    Invoked in Sections 2.1 and 3 to justify why ChemPile's scale and diversity matter; the paper does not demonstrate this for chemistry.
  • domain assumption The CAMEL-trained BERT classifier provides valid chemistry labels for filtering EuroPMC content.
    Section 4.2 and Appendix N.3.1; the classifier is treated as a reliable filter despite F1≈0.77 on 150 examples.
  • domain assumption LLM rewriting and synthetic reasoning generation preserve scientific accuracy.
    Appendix N.1.1 (GPT-4.1 rewrites YouTube transcripts) and Appendix N.4 (Claude-3.5-Sonnet and Deepseek-R1 generate spectral reasoning traces); only token-format and SMILES validity are checked, not factual content.
  • domain assumption Random splitting of amino-acid sequences and non-SMILES datasets prevents data leakage across ChemPile subsets.
    Section 4.7 and Appendix K; scaffold splitting is applied to SMILES, but random splitting is used for proteins and other data, which may still allow leakage between training and test.
  • standard math Tanimoto similarity on molecular fingerprints is an appropriate ground truth for evaluating chemical representation embeddings.
    Figure 3 and Appendix M use Tanimoto similarity as the reference to argue that IUPAC embeddings better reflect molecular similarity; this is a domain convention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models." pith.science (2026). https://pith.science/paper/37ZIMQEU

@misc{pith2026250512534,
  author       = {Pith},
  title        = {Pith review of: ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37ZIMQEU}},
  note         = {Machine review of arXiv:2505.12534}
}
read the original abstract

Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality datasets that reflect the field's multifaceted nature. We present the ChemPile, an open dataset containing over 75 billion tokens of curated chemical data, specifically built for training and evaluating general-purpose models in the chemical sciences. The dataset mirrors the human learning journey through chemistry -- from educational foundations to specialized expertise -- spanning multiple modalities and content types including structured data in diverse chemical representations (SMILES, SELFIES, IUPAC names, InChI, molecular renderings), scientific and educational text, executable code, and chemical images. ChemPile integrates foundational knowledge (textbooks, lecture notes), specialized expertise (scientific articles and language-interfaced data), visual understanding (molecular structures, diagrams), and advanced reasoning (problem-solving traces and code) -- mirroring how human chemists develop expertise through diverse learning materials and experiences. Constructed through hundreds of hours of expert curation, the ChemPile captures both foundational concepts and domain-specific complexity. We provide standardized training, validation, and test splits, enabling robust benchmarking. ChemPile is openly released via HuggingFace with a consistent API, permissive license, and detailed documentation. We hope the ChemPile will serve as a catalyst for chemical AI, enabling the development of the next generation of chemical foundation models.

Figures

Figures reproduced from arXiv: 2505.12534 by the authors.

Figure 1
Figure 1. Overview of the ChemPile and its curation process. The figure illustrates the dataset creation process. Education and Caption consist of gathering resources from online resources. Code and (m)LIFT are based on dataset content, for the first filtering from general datasets, while for the second, by filling templates with the data. For ChemPile-Paper, the content is collected by filtering and processing published open… view at source ↗
Figure 2
Figure 2. (a): Token count comparison between the ChemPile dataset and other domain-specific large datasets used to train foundation models. ChemDFM [56] is a foundation model for chemistry trained on 34B tokens in chemistry-related papers and textbooks augmented with general text (49M tokens), BatGPT [60] is a foundation model for chemical engineering, BioGPT [61] for biology, and ChemGPT [62] is a foundation model trained o… view at source ↗
Figure 3
Figure 3. Correlation between the Tanimoto similarity and the cosine similarity (θsim) of the embeddings for most common chemical representations. The correlation is shown for four representation embeddings: SMILES (top left), IUPAC name (top right), SELFIES (bottom left), and InChI (bottom right). For the four subplots, we show the Pearson correlation r in the top left corner of all subplots. This feature of the ChemPile is … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example of how our sampling engine operates. The sampling depends on two a metadata file, and a raw data file containing all the correct columns as described in the metadata. The colors match what elements of the text templates is replaced in the final text with natura…
Figure 5
Figure 5. Figure 5: ChemPile-Education covers different kinds of educational data. Textbook data contains foundational knowledge, but also worked examples. N.1.1 Sources LibreTexts Chemistry We systematically extracted and processed the primary textual content from LibreTexts Chemistry HT…
Figure 6
Figure 6. Figure 6: Keyword distribution by category. The plot shows the dataset distribution based on keywords identified in the entry. Here we have considered all the keywords from different categories irrespective of the domain (chemistry, materials science, and biology). Keywords for …
Figure 7
Figure 7. Figure 7: Additional embedding dimensionality reduction visualizations for Figure 2b We use the umap-learn package for UMAP and the scikit-learn package for TSNE. Default settings are used. P SMILES-IUPAC translation Source SMILES sequences are tokenized using Byte-Level Byte Pa…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A ModernBERT-based encoder trained with masked language modeling on SMILES-annotated scientific documents plus a contrastive stage yields embeddings that are competitive on both molecular property prediction and scien...

  2. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    cs.SE 2026-07 conditional novelty 6.0 of 10

    SciCodePile contributes a 128GB scientific code corpus and a 200-task executable benchmark showing that current LLMs still fall far short of reliable scientific code generation.

  3. Molecular Machine Learning in Chemical Process Design

    physics.chem-ph 2025-08 accept novelty 3.0 of 10

    This paper argues that integrating molecular machine learning into chemical process design could accelerate discovery of novel molecules and processes, but requires better data, benchmarks, and industry collaboration.

Reference graph

Works this paper leans on

89 extracted references · 25 canonical work pages · cited by 3 Pith papers

  1. [1]

    The future of chemistry is language

    Andrew D White. “The future of chemistry is language”. In: Nature Reviews Chemistry 7.7 (2023), pp. 457–458

  2. [2]

    A review of large language models and autonomous agents in chemistry

    Mayk Caldas Ramos, Christopher J. Collison, and Andrew D. White. “A review of large language models and autonomous agents in chemistry”. In: Chemical Science 16.6 (2025), pp. 2514–2572. ISSN : 2041-6539. DOI: 10.1039/d4sc03921a. URL: http://dx.doi.org/ 10.1039/D4SC03921A

  3. [3]

    14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon

    Kevin Maik Jablonka et al. “14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon”. In: Digital Discovery 2.5 (2023), pp. 1233–1250. ISSN : 2635-098X. DOI: 10 . 1039 / d3dd00113j. URL: http : //dx.doi.org/10.1039/D3DD00113J

  4. [4]

    Machine learning for a sustainable energy future

    Zhenpeng Yao et al. “Machine learning for a sustainable energy future”. In:Nature Reviews Materials 8.3 (Oct. 2022), pp. 202–215. ISSN : 2058-8437. DOI: 10.1038/s41578- 022- 00490-5. URL: http://dx.doi.org/10.1038/s41578-022-00490-5

  5. [5]

    Are large language models superhuman chemists?

    Adrian Mirza et al. “Are large language models superhuman chemists?” In: arXiv preprint arXiv: 2404.01475 (2024)

  6. [6]

    Language agents achieve superhuman synthesis of scientific knowledge

    Michael D. Skarlinski et al. “Language agents achieve superhuman synthesis of scientific knowledge”. In: arXiv preprint arXiv: 2409.13740 (2024)

  7. [7]

    MechGPT, a Language-Based Strategy for Mechanics and Materials Modeling That Connects Knowledge Across Scales, Disciplines, and Modalities

    Markus J. Buehler. “MechGPT, a Language-Based Strategy for Mechanics and Materials Modeling That Connects Knowledge Across Scales, Disciplines, and Modalities”. In: Applied Mechanics Reviews 76.2 (Jan. 2024). ISSN : 2379-0407. DOI: 10.1115/1.4063843 . URL: http://dx.doi.org/10.1115/1.4063843

  8. [8]

    Can Large Language Models Empower Molecular Property Prediction?

    Chen Qian et al. “Can Large Language Models Empower Molecular Property Prediction?” In: arXiv preprint arXiv: 2307.07443 (2023)

Show all 89 references
  1. [9]

    Leveraging large language models for predictive chemistry

    Kevin Maik Jablonka et al. “Leveraging large language models for predictive chemistry”. In: Nature Machine Intelligence 6.2 (2024), pp. 161–169

  2. [10]

    Benchmarking Large Language Models for Molecule Prediction Tasks

    Zhiqiang Zhong, Kuangyu Zhou, and Davide Mottin. “Benchmarking Large Language Models for Molecule Prediction Tasks”. In: arXiv preprint arXiv: 2403.05075 (2024)

  3. [11]

    LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions

    Andre Niyongabo Rubungo et al. “LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions”. In: arXiv preprint arXiv: 2310.14029 (2023). 10

  4. [12]

    MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction

    Yuyan Liu et al. “MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction”. In: arXiv preprint arXiv: 2406.12950 (2024)

  5. [13]

    Crystal structure generation with autoregressive large language modeling

    Luis M. Antunes, Keith T. Butler, and Ricardo Grau-Crespo. “Crystal structure generation with autoregressive large language modeling”. In: Nature Communications 15.1 (Dec. 2024). ISSN : 2041-1723. DOI: 10.1038/s41467-024-54639-7 . URL: http://dx.doi.org/10. 1038/s41467-024-54639-7

  6. [14]

    Large Language Models Are Innate Crystal Structure Generators

    Jingru Gan et al. “Large Language Models Are Innate Crystal Structure Generators”. In: arXiv preprint arXiv: 2502.20933 (2025)

  7. [15]

    Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

    Nate Gruver et al. “Fine-Tuned Language Models Generate Stable Inorganic Materials as Text”. In: arXiv preprint arXiv: 2402.04379 (2024)

  8. [16]

    Fleming: An AI Agent for Antibiotic Discovery in Mycobacterium tubercu- losis

    Ziming Wei et al. “Fleming: An AI Agent for Antibiotic Discovery in Mycobacterium tubercu- losis”. In: (Apr. 2025). DOI: 10.1101/2025.04.01.646719. URL: http://dx.doi.org/ 10.1101/2025.04.01.646719

  9. [17]

    Augmenting large language models with chemistry tools

    Andres M. Bran et al. “Augmenting large language models with chemistry tools”. In: Nature Machine Intelligence 6.5 (May 2024), pp. 525–535. ISSN : 2522-5839. DOI: 10.1038/s42256- 024-00832-8. URL: http://dx.doi.org/10.1038/s42256-024-00832-8

  10. [18]

    ORGANA: A robotic assistant for automated chemistry experimen- tation and characterization

    Kourosh Darvish et al. “ORGANA: A robotic assistant for automated chemistry experimen- tation and characterization”. In: Matter 8.2 (Feb. 2025), p. 101897. ISSN : 2590-2385. DOI: 10.1016/j.matt.2024.10.015 . URL: http://dx.doi.org/10.1016/j.matt.2024. 10.015

  11. [19]

    Autonomous chemical research with large language models

    Daniil A Boiko et al. “Autonomous chemical research with large language models”. In:Nature 624.7992 (2023), pp. 570–578

  12. [20]

    How should the advancement of large language models affect the practice of science?

    Marcel Binz et al. “How should the advancement of large language models affect the practice of science?” In: Proceedings of the National Academy of Sciences 122.5 (Jan. 2025). ISSN : 1091-6490. DOI: 10.1073/pnas.2401227121 . URL: http://dx.doi.org/10.1073/ pnas.2401227121

  13. [21]

    Generative Retrieval-Augmented Ontologic Graph and Multiagent Strate- gies for Interpretive Large Language Model-Based Materials Design

    Markus J. Buehler. “Generative Retrieval-Augmented Ontologic Graph and Multiagent Strate- gies for Interpretive Large Language Model-Based Materials Design”. In: ACS Engineering Au 4.2 (Jan. 2024), pp. 241–277. ISSN : 2694-2488. DOI: 10 . 1021 / acsengineeringau . 3c00058. URL...

  14. [22]

    An Interdisciplinary Outlook on Large Language Models for Scientific Research

    James Boyko et al. “An Interdisciplinary Outlook on Large Language Models for Scientific Research”. In: arXiv preprint arXiv: 2311.04929 (2023)

  15. [23]

    SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

    David Weininger. “SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules”. In: Journal of chemical information and computer sciences 28.1 (1988), pp. 31–36

  16. [24]

    Probing the limitations of multimodal language models for chemistry and materials research

    Nawaf Alampara et al. “Probing the limitations of multimodal language models for chemistry and materials research”. In: arXiv preprint arXiv: 2411.16955 (2024)

  17. [25]

    LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learn- ing Tasks

    Tuan Dinh et al. “LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learn- ing Tasks”. In: arXiv preprint arXiv: 2206.06565 (2022)

  18. [26]

    At Which Training Stage Does Code Data Help LLMs Reasoning?

    Yingwei Ma et al. “At Which Training Stage Does Code Data Help LLMs Reasoning?” In:The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL: https://openreview.net/forum?id= KIPJKST4gw

  19. [27]

    A Survey on Data Selection for Language Models

    Alon Albalak et al. “A Survey on Data Selection for Language Models”. In: arXiv preprint arXiv: 2402.16827 (2024)

  20. [28]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao et al. “The pile: An 800gb dataset of diverse text for language modeling”. In: arXiv preprint arXiv:2101.00027 (2020)

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text trans- former

    Colin Raffel et al. “Exploring the limits of transfer learning with a unified text-to-text trans- former”. In: Journal of machine learning research21.140 (2020), pp. 1–67

  22. [30]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann et al. “Training Compute-Optimal Large Language Models”. In: arXiv preprint arXiv: 2203.15556 (2022)

  23. [31]

    Scaling laws for neural language models

    Jared Kaplan et al. “Scaling laws for neural language models”. In: arXiv preprint arXiv:2001.08361 (2020)

  24. [32]

    When less is more: Investigating data pruning for pretraining llms at scale

    Max Marion et al. “When less is more: Investigating data pruning for pretraining llms at scale”. In: arXiv preprint arXiv:2309.04564 (2023). 11

  25. [33]

    Textbooks are all you need

    Suriya Gunasekar et al. “Textbooks are all you need”. In: arXiv preprint arXiv:2306.11644 (2023)

  26. [34]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre et al. “Datacomp: In search of the next generation of multimodal datasets”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 27092–27112

  27. [35]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo et al. “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15...

  28. [36]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini et al. “Dolma: An open corpus of three trillion tokens for language model pretraining research”. In: arXiv preprint arXiv:2402.00159 (2024)

  29. [37]

    Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two- Phase Pretraining

    Steven Feng et al. “Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two- Phase Pretraining”. In: arXiv preprint arXiv: 2412.15285 (2024)

  30. [38]

    LAION-5B: An Open Large-Scale Dataset for Training next Generation Image-Text Models

    Christoph Schuhmann et al. “LAION-5B: An Open Large-Scale Dataset for Training next Generation Image-Text Models”. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo et al. V ol. 35. Curran Associates, Inc., 2022, pp. 25278–25294

  31. [39]

    MoleculeNet: a benchmark for molecular machine learning

    Zhenqin Wu et al. “MoleculeNet: a benchmark for molecular machine learning”. In: Chemical science 9.2 (2018), pp. 513–530

  32. [40]

    Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development

    Kexin Huang et al. “Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development”. In: Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtu...

  33. [41]

    PubChem Substance and Compound databases

    Sunghwan Kim et al. “PubChem Substance and Compound databases”. In: Nucleic Acids Research 44.D1 (Sept. 2015), pp. D1202–D1213. ISSN : 1362-4962. DOI: 10.1093/nar/ gkv951. URL: http://dx.doi.org/10.1093/nar/gkv951

  34. [42]

    UniProt: a worldwide hub of protein knowledge

    UniProt Consortium. “UniProt: a worldwide hub of protein knowledge”. In: Nucleic acids research 47.D1 (2019), pp. D506–D515

  35. [43]

    Evaluating Chemistry Prompts for Large-Language Model Fine- Tuning

    Carmelo Gonzales et al. “Evaluating Chemistry Prompts for Large-Language Model Fine- Tuning”. In: AI for Accelerated Materials Design-NeurIPS. 2024

  36. [44]

    Quantum chemistry structures and properties of 134 kilo molecules

    Raghunathan Ramakrishnan et al. “Quantum chemistry structures and properties of 134 kilo molecules”. In: Scientific Data 1.1 (Aug. 2014). ISSN : 2052-4463. DOI: 10.1038/sdata. 2014.22. URL: http://dx.doi.org/10.1038/sdata.2014.22

  37. [45]

    Chemical reactions from US patents (1976-Sep2016)

    Daniel Lowe. “Chemical reactions from US patents (1976-Sep2016)”. In: (June 2017). DOI: 10 . 6084 / m9 . figshare . 5104873 . v1. URL: https : / / figshare . com / articles / dataset/Chemical_reactions_from_US_patents_1976-Sep2016_/5104873

  38. [46]

    Big Data from Pharmaceutical Patents: A Computational Analysis of Medicinal Chemists’ Bread and Butter

    Nadine Schneider et al. “Big Data from Pharmaceutical Patents: A Computational Analysis of Medicinal Chemists’ Bread and Butter”. In: Journal of Medicinal Chemistry 59.9 (Apr. 2016), pp. 4385–4402. ISSN : 1520-4804. DOI: 10.1021/acs.jmedchem.6b00153 . URL: http://dx.doi.org/10...

  39. [47]

    Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis

    Xiwen Jia et al. “Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis”. In: Nature 573.7773 (Sept. 2019), pp. 251–255. ISSN : 1476-4687. DOI: 10.1038/ s41586-019-1540-5 . URL: http://dx.doi.org/10.1038/s41586-019-1540-5

  40. [48]

    Machine-learning-assisted materials discovery using failed experiments

    Paul Raccuglia et al. “Machine-learning-assisted materials discovery using failed experiments”. In: Nature 533.7601 (May 2016), pp. 73–76. ISSN : 1476-4687. DOI: 10.1038/nature17439. URL: http://dx.doi.org/10.1038/nature17439

  41. [49]

    Data-Driven Matching of Experimental Crystal Structures and Gas Adsorption Isotherms of Metal–Organic Frameworks

    Daniele Ongari et al. “Data-Driven Matching of Experimental Crystal Structures and Gas Adsorption Isotherms of Metal–Organic Frameworks”. In: Journal of Chemical & Engi- neering Data 67.7 (Feb. 2022), pp. 1743–1756. ISSN : 1520-5134. DOI: 10.1021/acs.jced. 1c00958. URL: ht...

  42. [50]

    SELFIES and the future of molecular string representations

    Mario Krenn et al. “SELFIES and the future of molecular string representations”. In: Patterns 3.10 (Oct. 2022), p. 100588. ISSN : 2666-3899. DOI: 10.1016/j.patter.2022.100588 . URL: http://dx.doi.org/10.1016/j.patter.2022.100588. 12

  43. [51]

    Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models

    Yin Fang et al. “Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models”. In: International Conference on Learning Representations (2023). DOI: 10.48550/arXiv.2306.08018

  44. [52]

    A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

    Bing Su et al. “A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language”. In: arXiv preprint arXiv: 2209.05481 (2022)

  45. [53]

    Multi-modal molecule structure–text model for text-based retrieval and editing

    Shengchao Liu et al. “Multi-modal molecule structure–text model for text-based retrieval and editing”. In: Nature Machine Intelligence 5.12 (Dec. 2023), pp. 1447–1457. ISSN : 2522-5839. DOI: 10.1038/s42256-023-00759-6 . URL: http://dx.doi.org/10.1038/s42256- 023-00759-6

  46. [54]

    Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning

    Gang Liu et al. “Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning”. In: arXiv preprint arXiv: 2410.04223 (2024)

  47. [55]

    Multimodal Learning for Materials

    Viggo Moro et al. “Multimodal Learning for Materials”. In: arXiv preprint arXiv:2312.00111 (2023)

  48. [56]

    ChemDFM: A Large Language Foundation Model for Chemistry

    Zihan Zhao et al. “ChemDFM: A Large Language Foundation Model for Chemistry”. In:arXiv preprint arXiv:2401.14818 (2024)

  49. [57]

    Wudaocorpora: A super large-scale chinese corpora for pre-training language models

    Sha Yuan et al. “Wudaocorpora: A super large-scale chinese corpora for pre-training language models”. In: AI Open 2 (2021), pp. 65–68

  50. [58]

    LlaSMol: Advancing Large Language Models for Chemistry with a Large- Scale, Comprehensive, High-Quality Instruction Tuning Dataset

    Botao Yu et al. “LlaSMol: Advancing Large Language Models for Chemistry with a Large- Scale, Comprehensive, High-Quality Instruction Tuning Dataset”. In: arXiv preprint arXiv: 2402.09391 (2024)

  51. [59]

    Enhancing Chemical Reaction and Retrosynthesis Prediction with Large Language Model and Dual-task Learning

    Xuan Lin et al. “Enhancing Chemical Reaction and Retrosynthesis Prediction with Large Language Model and Dual-task Learning”. In: arXiv preprint arXiv: 2505.02639 (2025)

  52. [60]

    BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction

    Yifei Yang et al. “BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction”. In: arXiv preprint arXiv: 2408.10285 (2024)

  53. [61]

    BioGPT: generative pre-trained transformer for biomedical text generation and mining

    Renqian Luo et al. “BioGPT: generative pre-trained transformer for biomedical text generation and mining”. In: Briefings in bioinformatics 23.6 (2022), bbac409

  54. [62]

    Neural scaling of deep chemical models

    Nathan C Frey et al. “Neural scaling of deep chemical models”. In:Nature Machine Intelligence 5.11 (2023), pp. 1297–1305

  55. [63]

    Darwin 1.5: Large language models as materials science adapted learners

    Tong Xie et al. “Darwin 1.5: Large language models as materials science adapted learners”. In: arXiv preprint arXiv:2412.11970 (2024)

  56. [64]

    SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

    Amanpreet Singh et al. “SciRepEval: A Multi-Format Benchmark for Scientific Document Representations”. In: Conference on Empirical Methods in Natural Language Processing

  57. [65]

    Scaling Language Models: Methods, Analysis & Insights from Training Gopher

    Jack W. Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”. In: arXiv preprint arXiv: 2112.11446 (2021)

  58. [66]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Jiasheng Ye et al. “Data mixing laws: Optimizing data mixtures by predicting language modeling performance”. In: arXiv preprint arXiv:2403.16952 (2024)

  59. [67]

    Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation

    Mario Krenn et al. “Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation”. In: Machine Learning: Science and Technology1.4 (2020), p. 045024

  60. [68]

    InChI, the IUPAC international chemical identifier

    Stephen R Heller et al. “InChI, the IUPAC international chemical identifier”. In:Journal of cheminformatics 7 (2015), pp. 1–34

  61. [69]

    RanDepict: Random chemical structure depiction generator

    Henning Otto Brinkhaus et al. “RanDepict: Random chemical structure depiction generator”. In: Journal of cheminformatics 14.1 (2022), p. 31

  62. [70]

    MatText: Do language models need more than text & scale for materials modeling?

    Nawaf Alampara, Santiago Miret, and Kevin Maik Jablonka. “MatText: Do language models need more than text & scale for materials modeling?” In: arXiv preprint arXiv:2406.17295 (2024)

  63. [71]

    ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

    Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. “ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction”. In: arXiv preprint arXiv: 2010.09885 (2020)

  64. [72]

    Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?

    Dávid Bajusz, Anita Rácz, and Károly Héberger. “Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?” In: Journal of cheminformatics 7 (2015), pp. 1–13

  65. [73]

    tiktoken: A fast BPE tokenizer for use with OpenAI’s models

    OpenAI. tiktoken: A fast BPE tokenizer for use with OpenAI’s models . https://github. com/openai/tiktoken. Accessed: 2025-05-14. 2022

  66. [74]

    Europe PMC in 2023

    Summer Rosonovski et al. “Europe PMC in 2023”. In: Nucleic Acids Research 52.D1 (2024), pp. D1668–D1676. 13

  67. [75]

    CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society

    Guohao Li et al. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. 2023. arXiv: 2303.17760 [cs.AI]

  68. [76]

    SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model

    Loubna Ben Allal et al. SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model. 2025. arXiv: 2502.02737 [cs.CL] . URL: https://arxiv.org/abs/ 2502.02737

  69. [77]

    Trends in Deep Learning for Property-driven Drug Design

    Jannis Born and Matteo Manica. “Trends in Deep Learning for Property-driven Drug Design”. In: Current Medicinal Chemistry 28.38 (2021), pp. 7862–7886

  70. [79]

    Chemical name to structure: OPSIN, an open source solution

    Daniel M Lowe et al. Chemical name to structure: OPSIN, an open source solution. 2011

  71. [80]

    To Code, or Not To Code? Exploring Impact of Code in Pre-training

    Viraat Aryabumi et al. “To Code, or Not To Code? Exploring Impact of Code in Pre-training”. In: arXiv preprint arXiv: 2408.10914 (2024)

  72. [81]

    StarCoder: may the source be with you!

    Raymond Li et al. “StarCoder: may the source be with you!” In: Trans. Mach. Learn. Res. 2023 (2023). URL: https://openreview.net/forum?id=KoFOg41haE

  73. [82]

    The Stack: 3 TB of permissively licensed source code

    Denis Kocetkov et al. “The Stack: 3 TB of permissively licensed source code”. In: Trans. Mach. Learn. Res. (2022). DOI: 10.48550/arXiv.2211.15533

  74. [83]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning

    DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning”. In: arXiv preprint arXiv: 2501.12948 (2025)

  75. [84]

    The properties of known drugs. 1. Molecular frameworks

    Guy W Bemis and Mark A Murcko. “The properties of known drugs. 1. Molecular frameworks”. In: Journal of medicinal chemistry 39.15 (1996), pp. 2887–2893

  76. [85]

    Nougat: Neural optical understanding for academic documents

    Lukas Blecher et al. “Nougat: Neural optical understanding for academic documents”. In: arXiv preprint arXiv:2308.13418 (2023)

  77. [86]

    Neural Machine Translation of Rare Words with Subword Units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural Machine Translation of Rare Words with Subword Units”. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Katrin Erk and Noah A. Smith. Berlin, Ge...

  78. [87]

    Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

    Taku Kudo. “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”. In: Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Pa...

  79. [89]

    Open-ended questions, requiring free-form generation of the correct SMILES string

  80. [90]

    jablonkagroup/chempile-caption

    Multiple-choice questions (MCQs), where the model selected the correct answer from structural isomers of the target compound. For the multi-spectra dataset, we maintained consistency by employing the same SMILES formatting protocol and evaluation methodology as in the single-s...

  81. [2022]

    URL: https://api.semanticscholar.org/CorpusID:254018137

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.