Pith. sign in

REVIEW 3 major objections 4 minor 36 references

On private wet-lab data from eight reaction classes, the best language models trail a logistic-regression baseline on seven of eight and none beat chance at catalyst selection, exposing a gap between chemistry knowledge and lab judgment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:22 UTC pith:73CTECDJ

load-bearing objection A useful first benchmark for lab-aware chemistry evaluation, with a genuinely new refusal-profile analysis and private wet-lab reaction data — but the 'LLMs trail a logistic baseline' claim rests on an underspecified baseline that needs to be pinned down before it can be cited. the 3 major comments →

arxiv 2608.02595 v1 pith:73CTECDJ submitted 2026-08-03 cs.LG

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Brandon Wang , Andrei S. Tyrin , Daniil A. Boiko This is my paper
classification cs.LG
keywords onepot-Bench 0language modelschemical synthesisreaction outcome predictioncatalyst selectionrefusal behaviorcheminformaticsbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that frontier language models have acquired enough chemistry knowledge to answer basic molecular-property questions, but this knowledge does not transfer to the empirical judgment chemists depend on: predicting whether a reaction will succeed under given conditions, or which catalyst will work better. To show this, the authors built onepot-Bench 0, a benchmark combining tool-free cheminformatics questions (ChemAbacus), systematic refusal measurements over benign, controlled, and designer-drug targets (SynthRefusal), and two tasks built on their own private wet-lab data (SynthBench). The central result is that on reaction-outcome prediction, all tested models trail a simple logistic-regression baseline on seven of eight reaction classes, and no model exceeds chance on catalyst preference. If the result holds, it implies that current LLM chemistry competence is akin to textbook knowledge without troubleshooting ability—a serious concern for automated synthesis and closed-loop lab agents.

Core claim

The paper's central discovery is a sharp dissociation between two kinds of chemical competence in large language models. On ChemAbacus—800 multiple-choice and numeric questions about single molecules—frontier models reach high accuracy, with the best model (using a fallback for refused questions) exceeding 96% at high reasoning effort. On SynthBench, which uses 640 reactions from the authors' private wet lab (40 positive and 40 negative in each of eight common reaction classes) plus 95 paired Suzuki-Miyaura reactions that differ only in catalyst, the same models fall to near-chance, well below a logistic-regression baseline trained on the same data. On catalyst preference, all models are sta

What carries the argument

The load-bearing machinery is the benchmark design itself: SynthBench's use of private, non-public experimental data from the authors' laboratory, paired with a simple logistic-regression baseline trained on the same internal data. Because the reactions and catalyst pairs are not in model training corpora, the comparison isolates reasoning about real chemistry from memorization of public datasets. For SynthRefusal, the alignment index—the geometric mean of benign compliance and controlled-target refusal—functions as the key instrument for separating models that merely refuse everything or nothing from models that condition their behavior on risk. Together, these instruments define the paper'

Load-bearing premise

The conclusion about reaction judgment rests on the assumptions that the 640 in-house reactions and 95 catalyst pairs are labeled correctly (success defined as 'sufficient presence of target molecule' without a stated quantitative threshold), that they are representative of routine synthetic chemistry, and that their private distribution does not systematically disadvantage language models in a way a fair comparison would correct.

What would settle it

Run the same SynthBench reactions under a stricter definition of success (e.g., isolated yield above a stated threshold, or HPLC purity) and see whether the logistic-regression baseline still beats every LLM; if the margin collapses or flips, the reported gap is an artifact of the lab's success criterion rather than a general property of LLM chemistry judgment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • General chemistry knowledge as measured by standard Q&A is necessary but not sufficient for wet-lab decision-making; strong ChemAbacus performance does not predict SynthBench success.
  • Current frontier LLMs are overoptimistic about reaction outcomes, which means they would tend to recommend failing conditions if used as zero-shot planners.
  • Increased reasoning effort (test-time compute) does not close the reaction-outcome gap and can even degrade performance, so the deficit is one of knowledge or intuition rather than deliberation time.
  • Refusal behavior is representation- and class-dependent, and models that guard established controlled substances may still assist with unfamiliar designer analogs, implying list-memorization rather than robust risk assessment.
  • The benchmark establishes a reusable framework for measuring lab-aware capabilities with private data, enabling future iterations such as agentic tasks and closed-loop feedback, as the authors note.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gap is real, it suggests the next bottleneck for LLM-driven synthesis is not more parameters or training on public reactions, but access to proprietary, high-quality negative-outcome data; a testable prediction is that fine-tuning on such data would raise SynthBench scores substantially.
  • The catalyst-preference null result may partly reflect the small dataset (95 pairs) and the inherent difficulty of Suzuki-Miyaura condition sensitivity; a larger paired-catalyst set with identical execution protocols might reveal above-chance performance in at least some models—a question the paper itself flags as needing more data.
  • The alignment-index approach for refusal could be extended to other dual-use domains (e.g., biology or cybersecurity) where ideal behavior is conditional compliance, not blanket refusal.
  • The logistic-regression baseline's strong performance suggests even simple empirical models trained on private data capture signal that LLMs cannot extract from text; this argues for hybrid systems that integrate learned reaction predictors rather than relying solely on parametric knowledge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces onepot-Bench 0, a proprietary benchmark suite for evaluating language models on chemistry tasks relevant to wet-lab execution. It comprises three evaluations: ChemAbacus (800 cheminformatics questions, e.g., molecular weight, TPSA, functional-group identification), SynthRefusal (refusal behavior across benign, controlled, and designer-drug targets), and SynthBench (private wet-lab reaction-outcome prediction and catalyst selection). Thirteen models from six providers are evaluated. The main findings are that frontier models perform reasonably well on ChemAbacus (roughly 80–96% macro accuracy) but much worse on SynthBench: reaction-outcome predictions trail a logistic-regression baseline on 7 of 8 reaction classes, and no model performs significantly above chance on catalyst preference. The paper interprets this as a gap between chemistry-adjacent knowledge and the empirical judgment required for laboratory decision-making, and also reports substantial variation in safety behavior across models, representations, and reasoning effort.

Significance. If the results are reliable, the benchmark addresses a real gap in LLM evaluation for chemistry by using private experimental data to avoid contamination and by focusing on abilities needed for actual lab execution. The alignment index (Eq. 1) is a thoughtful contribution that separates blanket refusal from conditional safety behavior. The paper is also commendable for evaluating a broad set of models, testing multiple reasoning-effort settings, and reporting confidence intervals for the catalyst-preference result. However, the central negative result depends heavily on the specification of the logistic-regression baseline and on the quality/labeling of the private SynthBench data. These points need to be strengthened before the headline claims can be fully accepted.

major comments (3)
  1. [Section 3.3, Table 7, Figure 6] The logistic-regression baseline is not sufficiently specified. The text states only that it was 'trained using logistic regression on internal data, using a simple feature set' (Section 3.3). No training set size, feature definitions, regularization, or train/evaluation split are provided. If the baseline was evaluated on the same 80 reactions per class from which it was trained, its reported 76–84% accuracies would reflect memorization rather than prediction, invalidating the comparison with zero-shot LLMs. Even if a proper split was used, the absence of a protocol prevents reproducibility and undermines the headline claim that models 'trail a simple empirical baseline.' The authors should specify the training/evaluation protocol and, ideally, report cross-validated performance with confidence intervals.
  2. [Section 2.3, SynthBench] The success criterion for reaction outcome is defined only as 'sufficient presence of target molecule,' without a quantitative threshold or analytical method. The selection of the 40 positive/40 negative reactions per class and the 95 catalyst pairs (e.g., how failed reactions were generated, how catalysts were chosen) is also not described. Because all SynthBench conclusions depend on these labels, the absence of a precise annotation and selection protocol makes the central claims difficult to assess. The paper should provide a detailed labeling and selection protocol, even if the underlying reactions remain proprietary.
  3. [Table 7, Figure 6] The comparison between LLM accuracies and the logistic-regression baseline is reported as point estimates without confidence intervals or significance tests. For a binary task with n=80 per reaction class, the standard error is approximately 5 percentage points. The single class where the best LLM (Claude Opus 5, 70%) exceeds the baseline (68%) is within this noise, and several other apparent differences may also be non-significant. The paper should report confidence intervals or statistical tests for the model-vs-baseline comparisons, not only for the catalyst-preference results.
minor comments (4)
  1. [Section 2.3] Typo: 'The model is then asked to to identify which catalyst' should read 'asked to identify.'
  2. [Section 2.2] The five response categories (clean refusal, produced answer, partial disclosure, target not identified, target misidentified) are defined in the text, but the precise handling of 'partial disclosure' in the refusal-rate denominator could be clearer in the main text; it is described in more detail only in Appendix B.
  3. [Table 7] The caption indicates that underlining marks significance above chance, but the baseline row is not marked with significance. Consider adding the same notation for the baseline or stating that the baseline is not part of the significance test.
  4. [Throughout] The name 'onepot-Bench 0' is typeset inconsistently (e.g., 'onepot-Bench 0' vs. 'onepot-Bench0'). Please standardize.

Circularity Check

0 steps flagged

No significant circularity; benchmark comparisons are external to the models.

full rationale

onepot-Bench 0 is an empirical evaluation, not a derivation chain. ChemAbacus answers are scored against RDKit-computed ground truth; the SynthRefusal alignment index is defined directly from measured refusal rates; and SynthBench compares zero-shot LLM predictions to a logistic-regression comparator. None of these steps defines the target result in terms of the input or fits a parameter and then reports it as a prediction of a closely related quantity. The only self-citation with load-bearing weight is [25], which establishes provenance of the private wet-lab data; it is a data source, not a mathematical premise or uniqueness theorem, and citing one's own lab for one's own dataset is not circular. The logistic-regression baseline's train/evaluation protocol is not specified (Section 3.3), which is a legitimate soundness/reproducibility concern, but without evidence that the baseline was evaluated on its training set the comparison does not reduce by construction to the claim that models trail a simple baseline. Score 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claims rest on proprietary wet-lab data and a fitted baseline; no new physical entities are proposed. The main invented artifact is the benchmark itself, which is a method rather than an entity.

free parameters (1)
  • Logistic-regression baseline feature weights = not reported
    The baseline is trained on the same internal SynthBench data used for evaluation. No coefficients, feature set, or data splits are disclosed, and the paper does not discuss the advantage this gives the baseline over zero-shot LLMs.
axioms (4)
  • domain assumption The in-house wet-lab reactions are representative of routine synthetic chemistry and correctly labeled as success/failure.
    Section 2.3 says all SynthBench reactions come from internal lab experiments, with success 'defined via sufficient presence of target molecule'; no quantitative threshold or selection protocol is given. If the dataset is biased, the benchmark findings do not generalize.
  • domain assumption The private SynthBench data has not been seen by the evaluated models during training.
    The benchmark's premise is that private data avoids leakage (Abstract, Section 1). No contamination test is provided.
  • domain assumption The comparison to a logistic-regression baseline trained on the same internal data is a valid measure of zero-shot LLM competence.
    Section 3.3 trains a model on internal data; the paper does not discuss that the baseline has access to labels of the evaluation set, making the comparison asymmetric.
  • domain assumption Reaction success as binary labels via 'sufficient presence of target molecule' captures meaningful chemistry outcomes.
    No quantitative detection threshold or analytical method is described in Section 2.3.

pith-pipeline@v1.3.0-daily-deepseek · 17199 in / 10880 out tokens · 146919 ms · 2026-08-04T03:22:06.021283+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of onepot-Bench 0: towards lab-aware in silico chemistry benchmarks." pith.science (2026). https://pith.science/paper/73CTECDJ

@misc{pith2026260802595,
  author       = {Pith},
  title        = {Pith review of: onepot-Bench 0: towards lab-aware in silico chemistry benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/73CTECDJ}},
  note         = {Machine review of arXiv:2608.02595}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 3 linked inside Pith

  1. [1]

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks

    Taicheng Guo et al. “What can large language models do in chemistry? a comprehensive benchmark on eight tasks”. In:Proceedings of the 37th International Conference on Neu- ral Information Processing Systems. NIPS ’23. New Orleans, LA, USA: Curran Associates Inc., 2023

    2023

  2. [2]

    A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

    Adrian Mirza et al. “A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists”. In:Nature Chem- istry17.7 (July 1, 2025), pp. 1027–1034

    2025

  3. [3]

    Assessing the Chemical In- telligence of Large Language Models

    Nicholas T. Runcie, Charlotte M. Deane, and Fergus Imrie. “Assessing the Chemical In- telligence of Large Language Models”. In:Journal of Chemical Information and Modeling 66.1 (Jan. 12, 2026), pp. 216–227

    2026

  4. [4]

    Autonomous chemical research with large language models

    Daniil A. Boiko et al. “Autonomous chemical research with large language models”. In: Nature624.7992 (Dec. 1, 2023), pp. 570–578

    2023

  5. [5]

    Augmenting large language models with chemistry tools

    Andres M. Bran et al. “Augmenting large language models with chemistry tools”. In: Nature Machine Intelligence6.5 (May 1, 2024), pp. 525–535

    2024

  6. [6]

    Evaluating the Progression of Large Language Model Ca- pabilities for Small-Molecule Drug Design

    Shriram Chennakesavalu et al. “Evaluating the Progression of Large Language Model Ca- pabilities for Small-Molecule Drug Design”. In:CoRRabs/2604.16279 (2026)

    Pith/arXiv arXiv 2026

  7. [7]

    LAB-Bench: Measuring Capabilities of Language Models for Biol- ogy Research

    Jon M. Laurent et al. “LAB-Bench: Measuring Capabilities of Language Models for Biol- ogy Research”. In:CoRRabs/2407.10362 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Towards end-to-end automation of AI research

    Chris Lu et al. “Towards end-to-end automation of AI research”. In:Nature651.8107 (Mar. 1, 2026), pp. 914–919

    2026

  9. [9]

    Dual use of artificial-intelligence-powered drug discovery

    Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery”. In:Na- ture Machine Intelligence4.3 (2022), pp. 189–191

    2022

  10. [10]

    ChemSafetyBench: Benchmarking LLM Safety on Chemistry Do- main

    Haochen Zhao et al. “ChemSafetyBench: Benchmarking LLM Safety on Chemistry Do- main”. In:CoRRabs/2411.16736 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Predicting reaction performance in C–N cross-coupling using machine learning

    Derek T. Ahneman et al. “Predicting reaction performance in C–N cross-coupling using machine learning”. In:Science360.6385 (2018), pp. 186–190

    2018

  12. [12]

    Using Machine Learning To Predict Suitable Conditions for Organic Reactions

    Hanyu Gao et al. “Using Machine Learning To Predict Suitable Conditions for Organic Reactions”. In:ACS Central Science4.11 (2018), pp. 1465–1476

    2018

  13. [13]

    SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

    David Weininger. “SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules”. In:Journal of Chemical Information and Computer Sciences28.1 (Feb. 1, 1988), pp. 31–36

    1988

  14. [14]

    Fast Calculation of Molecular Polar Surface Area as a Sum of Fragment-Based Contributions and Its Application to the Prediction of Drug Transport Properties

    Peter Ertl, Bernhard Rohde, and Paul Selzer. “Fast Calculation of Molecular Polar Surface Area as a Sum of Fragment-Based Contributions and Its Application to the Prediction of Drug Transport Properties”. In:Journal of Medicinal Chemistry43.20 (Oct. 1, 2000), pp. 3714–3717

    2000

  15. [15]

    Prediction of Physicochemical Parameters by Atomic Contributions

    Scott A. Wildman and Gordon M. Crippen. “Prediction of Physicochemical Parameters by Atomic Contributions”. In:Journal of Chemical Information and Computer Sciences 39.5 (2002), pp. 868–873

    2002

  16. [16]

    Greg Landrum et al.RDKit: Open-source cheminformatics.https://www.rdkit.org

  17. [17]

    Daylight Theory Manual

    Daylight Chemical Information Systems, Inc.SMARTS – A Language for Describing Molec- ular Patterns. Daylight Theory Manual

  18. [18]

    ChEMBL: towards direct deposition of bioassay data

    David Mendez et al. “ChEMBL: towards direct deposition of bioassay data”. In:Nucleic Acids Research47.D1 (2018), pp. D930–D940

    2018

  19. [19]

    Approved 11 September 2025; in force 1 January 2026

    World Anti-Doping Agency.World Anti-Doping Code International Standard: Prohibited List 2026. Approved 11 September 2025; in force 1 January 2026. 2026. 12

    2026

  20. [20]

    Controlled Substances Act schedules I–V

    United States Drug Enforcement Administration, Diversion Control Division.Lists of Schedul- ing Actions, Controlled Substances, and Regulated Chemicals. Controlled Substances Act schedules I–V. 2026

    2026

  21. [21]

    Entered into force 29 April 1997

    Organisation for the Prohibition of Chemical Weapons.Convention on the Prohibition of the Development, Production, Stockpiling and Use of Chemical Weapons and on Their De- struction: Annex on Chemicals (Schedules 1–3). Entered into force 29 April 1997. 1993

    1997

  22. [22]

    Entered into force 17 May 2004; annexes amended by subsequent Conference of the Parties decisions

    United Nations Environment Programme.Stockholm Convention on Persistent Organic Pollutants: Annexes A, B and C. Entered into force 17 May 2004; annexes amended by subsequent Conference of the Parties decisions. 2001

    2004

  23. [23]

    Aidan Wong et al.SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chem- ical Synthesis. 2024

    2024

  24. [24]

    [Online; accessed 28-July-2026]

    Wikipedia contributors.List of designer drugs. [Online; accessed 28-July-2026]. 2026

    2026

  25. [25]

    onepot AI.POT-2 — the automated lab

  26. [26]

    Amide bond formation and peptide coupling

    Christian A.G.N. Montalbetti and Virginie Falque. “Amide bond formation and peptide coupling”. In:Tetrahedron61.46 (2005), pp. 10827–10852

    2005

  27. [27]

    A Simple Catalytic Method for the Conversion of Aryl Bromides to Arylamines

    Anil S. Guram, Roger A. Rennels, and Stephen L. Buchwald. “A Simple Catalytic Method for the Conversion of Aryl Bromides to Arylamines”. In:Angewandte Chemie Interna- tional Edition in English34.12 (1995), pp. 1348–1350

    1995

  28. [28]

    Palladium-catalyzed synthesis of arylamines from aryl halides. Mechanistic studies lead to coupling in the absence of tin reagents

    Janis Louie and John F. Hartwig. “Palladium-catalyzed synthesis of arylamines from aryl halides. Mechanistic studies lead to coupling in the absence of tin reagents”. In:Tetra- hedron Letters36.21 (1995), pp. 3609–3612

    1995

  29. [29]

    New N- and O-arylations with phenylboronic acids and cupric acetate

    Dominic M.T Chan et al. “New N- and O-arylations with phenylboronic acids and cupric acetate”. In:Tetrahedron Letters39.19 (1998), pp. 2933–2936

    1998

  30. [30]

    New aryl/heteroaryl C-N bond cross-coupling reactions via aryl- boronic acid/cupric acetate arylation

    Patrick Y.S Lam et al. “New aryl/heteroaryl C-N bond cross-coupling reactions via aryl- boronic acid/cupric acetate arylation”. In:Tetrahedron Letters39.19 (1998), pp. 2941– 2944

    1998

  31. [31]

    Synthesis of secondary amines

    Ralph N. Salvatore, Cheol Hwan Yoon, and Kyung Woon Jung. “Synthesis of secondary amines”. In:Tetrahedron57.37 (2001), pp. 7785–7811

    2001

  32. [32]

    A new stereospecific cross-coupling by the palladium-catalyzed reaction of 1-alkenylboranes with 1-alkenyl or 1-alkynyl halides

    Norio Miyaura, Kinji Yamada, and Akira Suzuki. “A new stereospecific cross-coupling by the palladium-catalyzed reaction of 1-alkenylboranes with 1-alkenyl or 1-alkynyl halides”. In:Tetrahedron Letters20.36 (1979), pp. 3437–3440

    1979

  33. [33]

    Recent advances in urea- and thiourea-containing compounds: focus on innovative approaches in medicinal chemistry and organic synthesis

    Riccardo Ronchetti et al. “Recent advances in urea- and thiourea-containing compounds: focus on innovative approaches in medicinal chemistry and organic synthesis”. In:RSC Medicinal Chemistry12.7 (2021), pp. 1046–1064

    2021

  34. [34]

    Urea Derivatives in Modern Drug Discovery and Medicinal Chemistry

    Arun K. Ghosh and Margherita Brindisi. “Urea Derivatives in Modern Drug Discovery and Medicinal Chemistry”. In:Journal of Medicinal Chemistry63.6 (2019), pp. 2751–2788

    2019

  35. [35]

    Van Leusen, Jurjen Wildeman, and Otto H

    Albert M. Van Leusen, Jurjen Wildeman, and Otto H. Oldenziel. “Chemistry of sulfonyl- methyl isocyanides. 12. Base-induced cycloaddition of sulfonylmethyl isocyanides to carbon,nitrogen double bonds. Synthesis of 1,5-disubstituted and 1,4,5-trisubstituted imidazoles from aldimines and imidoyl chlorides”. In:The Journal of Organic Chemistry 42.7 (May 1977)...

    1977

  36. [36]

    Comment on “Predicting reaction perfor- mance in C–N cross-coupling using machine learning

    Kangway V. Chuang and Michael J. Keiser. “Comment on “Predicting reaction perfor- mance in C–N cross-coupling using machine learning””. In:Science362.6416 (2018). 13 A. Extended ChemAbacus Results Here, we report extended results for ChemAbacus. Table 1 shows the overall scores for each (model, task) pair. Overall, Fable 5 (w/ Opus 5 fallback) is the stro...

    2018