Pith. sign in

REVIEW 3 major objections 3 minor 14 references

Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that SciGym is the first open-ended dry-lab benchmark to score LLMs on the full cycle of scientific experimentation and reports that, on 137 simulated systems, all six frontier models degrade sharply as network complexity…

desk verdict SciGym is a genuinely useful dry-lab benchmark with honest limitations, but the identifiability gap means its headline claim about scientific capability is conditional until per-system audits are done. read the letter →

arxiv 2507.02083 v2 pith:TWAZLWRK submitted 2025-07-02 cs.AI

Haonan Duan , Stephen Zhewen Lu , Caitlin Fiona Harrigan , Nishkrit Desai , Jiarui Lu , Micha{l} Koziarski , Leonardo Cotta , Chris J. Maddison This is my paper · ORCID
classification cs.AI
keywords LLMscientificdiscoverysystemsbiologySBMLdrylabbenchmarkexperimentdesignreactionnetworkinferenceagenticreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM's scientific ability can be measured by making it run a complete discovery loop in a simulated biology laboratory: propose a perturbation, read the resulting time-series data, reason with code, and finally submit a mechanistic model of the hidden system. To do this it introduces SciGym, built from 350 curated SBML models of biochemical reaction networks, and evaluates six frontier LLMs on the 137 smallest systems. The results show the benchmark separates model tiers—pro versions clearly beat their mini versions—but also that every model's reaction recovery drops and trajectory error rises as the number of reactions grows. The paper reads this as evidence that current agents learn something from experiments, yet overfit their proposed mechanisms to the data they saw and nearly fail to recover modifier-mediated regulation. If the central claim is right, the field gains a cheap, reusable, open-ended testbed for agentic scientific discovery and a clear description of where frontier models fall short.

What carries the argument

The central object is the SBML model—a machine-readable XML standard for biochemical reaction networks, where each reaction lists reactants, products, optional modifiers, and a kinetic law in MathML, and the whole system simulates as a system of ordinary differential equations. SciGym uses a curated SBML model as the hidden reference, gives the agent a partial version with all reactions removed, and simulates any requested initial-concentration perturbation to return dense time-series data. The agent iterates as a thought-action-observation loop with a Python shell, and the submitted model is scored by Network Topology Score (pairwise species-edge F1), Reaction Matching Score (F1 over reactions, optionally requiring matching modifiers), and Simulation Trajectory Error (SMAPE between predicted and true trajectories).

What would settle it

Run an identifiability audit on the released tasks: enumerate or sample all reaction graphs that reproduce the simulated time-series under the allowed initial-concentration perturbations, and check whether any system admits more than one network; if one does, a low agent score on that system cannot be interpreted purely as weak scientific capability.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that SciGym is the first benchmark to evaluate LLMs on the full cycle of scientific experimentation rather than on isolated skills or static datasets. The task is to recover the reaction topology and dynamics of a hidden SBML system using only simulated observations: the agent receives the model with all reactions removed, can change initial concentrations of species and run Python analysis, and must submit a new SBML model. Across 137 small systems, every evaluated model improved over a zero-shot baseline, pro-tier models outperformed mini-tier models, and all models degraded as system complexity grew. The authors conclude that current LLM agents can extract signal from experiments but their proposed mechanisms are overfit to the specific trajectories observed and are particularly weak at identifying modifier relationships.

Load-bearing premise

The whole exercise reads as a measure of scientific reasoning only if the hidden reaction network of each system is identifiable from the experiments the agent is permitted to run, so that low scores are failures of intelligence rather than failures of the task to be solvable.

Editorial extensions

If this is right

  • If the central claim holds, benchmark designers no longer need wet-lab costs to measure iterative scientific behavior: an SBML dry lab can generate experiment data in the loop, so agentic scientific discovery becomes cheaply and reproducibly testable.
  • Model rankings in this setting are diagnostic: across the Gemini, Claude, and GPT-4 families, the pro-tier models consistently beat their mini counterparts on both trajectory error and reaction recovery, so the benchmark separates capability levels.
  • Complexity scaling is the key headroom signal: every model's error rises sharply as systems grow from 2 to 10 reactions, which means evaluations on small toy networks will overstate real scientific competence.
  • Overfitting is measurable: when initial conditions are perturbed with Gaussian noise, proposed models degrade more than the true system, so the benchmark can quantify generalization rather than just fit.
  • Modifier discovery is a distinct bottleneck: reactant-product edges are recovered at several times the rate of modifier-involved edges, pointing to a specific capability gap in inferring regulatory interactions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence of the de-identification design is that keeping original species names may let prior biological knowledge substitute for experimentation; rerunning the benchmark with fully anonymized names (or measuring name-based priors) would separate memorized biochemistry from discovered mechanisms.
  • An identifiability audit of each task would settle whether low scores reflect reasoning deficits or tasks that are underdetermined by the allowed initial-condition perturbations, a possibility the paper itself flags.
  • The same harness could be extended to map experimental affordances: adding noise, sparse sampling, knockouts, or hidden species would show which capabilities most improve discovery, and the logged agent trajectories could seed training data for better scientific agents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces SCIGYM, an agentic benchmark that turns curated BioModels SBML systems into a dry lab for evaluating LLM-driven scientific discovery. An LLM receives an incomplete SBML model containing species but no reactions and may, over up to 20 iterations, write Python code, request experiments (changing initial concentrations of species), and submit a final SBML model. Performance is measured by three metrics: Network Topology Score, Reaction Matching Score, and Simulation Trajectory Error. Six frontier LLMs are evaluated on 137 small systems, with results showing that pro variants outperform mini variants, that performance degrades as system complexity increases, that modifier relationships are recovered at near-zero rates, and that proposed models degrade under perturbed initial conditions. The authors release code, data, and evaluation artifacts.

Significance. If the identified gaps are addressed, SCIGYM is a valuable, reusable instrument: it requires no fitted parameters, provides a full per-system result appendix, includes a zero-shot baseline, and ships open code and data. The headline interpretation, however, is conditional on showing that low scores reflect scientific reasoning rather than task underdetermination or prior-knowledge shortcuts. The paper's candid limitation statements and released artifacts make these concerns concretely testable, which is a genuine strength.

major comments (3)
  1. [§5 (Perturbations), Fig. 6, Table 1] The central claim that SCIGYM scores diagnose LLM scientific capability requires that each hidden reaction network be recoverable in principle from the only allowed perturbation, changing initial concentrations. Section 5 explicitly concedes that "our perturbations cannot guarantee complete system recovery of every SBML model," citing Villaverde et al. (2016), yet no per-system structural identifiability analysis is reported. If an unknown fraction of the 137 systems admit multiple distinct reaction networks with identical input-output behavior under every allowed experiment, then the complexity trend in Figure 6 and the overfitting interpretation of Figure 7 could be driven by a rising share of underdetermined tasks among larger networks rather than by declining reasoning ability. I request a concrete audit: for each system, verify whether the permitted experiments uniquely determine the ground-truth reaction topology, and re-run the headline analyses restricted to the identifiable subset.
  2. [§3.1, Appendix C.2 (De-identifying SBML files)] The de-identification pipeline keeps original species names "such that biological entities in the system remain identifiable." Names like ATP or Glucose carry strong prior information about plausible reaction networks, so an agent can substitute memorized biochemical knowledge for experiment-driven discovery. This is a confound for the zero-shot versus agent comparison in Figure 5 and for the pro-versus-mini ordering in Table 1, since more capable models may exploit prior knowledge more effectively. I request a control that randomizes or anonymizes species names (or, alternatively, a quantitative analysis of how much performance changes when names are removed), and a discussion of the trade-off between biological realism and measurement validity.
  3. [§5 (Experiments), Tables 1 and 2, Fig. 6] All results appear to come from a single run per model per system, with no seeds, repeated sampling, or variance estimates reported. Claims such as "pro variants consistently outperform their mini counterparts" and "performance declined significantly as system complexity increased" are therefore not accompanied by any measure of uncertainty; stochastic decoding alone could change some of the reported orderings. I request that the authors report the number of runs, the sampling procedure, and confidence intervals or per-system variance, and that the complexity and modifier-recovery claims be supported by appropriate statistical tests.
minor comments (3)
  1. [Appendix E.2/E.3] Sections E.2 and E.3 have the same title, "Detailed Results by BioMD," and Tables S3 and S4 are identical; this appears to be a duplication error that should be corrected.
  2. [§2 (Background)] The sentence "Going forward, we use texttt when referring to specific SBML tag types" appears to be an unresolved formatting instruction rather than a sentence about the manuscript's content; it should be rephrased or removed.
  3. [Abstract and §5.1] The abstract's phrase "performance declined significantly" and the analogous claim in §5.1 are not backed by any statistical test or confidence interval; the authors should either add the test or soften the wording to match the evidence presented.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark measures what it defines, and the acknowledged identifiability limitation is a validity risk, not a circular step.

full rationale

The paper's derivation chain is a benchmark construction, not a fitted prediction. SBML models from BioModels are filtered, de-identified, and stripped of reactions; agents propose experiments via initial-concentration changes, analyze returned time series, and submit SBML models; evaluation compares topology (NTS), reaction structure (RMS), and trajectory error (STE) against the held-out reference. No parameter is fitted to the target data and then renamed as a prediction, and no scientific capability score is derived from the very quantity it is supposed to predict. The complexity-decline and modifier-recovery results are empirical observations from the agent runs, not consequences of the metric definitions. The paper contains no load-bearing self-citation: the only substantive external citations are to BioModels for model provenance, to SBML/simulation tools, and to Villaverde et al. for structural identifiability background; none of these are by the present authors, and none are invoked to forbid alternative interpretations. The candid Section 5 statement that initial-concentration perturbations 'cannot guarantee complete system recovery of every SBML model' weakens the interpretability of low agent scores if some tasks are underdetermined, but this is an external validity and identifiability concern, explicitly acknowledged by the authors, not a case where the claimed output is equivalent to its input by construction. Similarly, Appendix C.2's decision to keep species names ('so that biological entities remain identifiable') could leak prior knowledge into the task, but that is an information-leakage issue, not a circular derivation. No step in the paper reduces Eq. X to Eq. Y by definition, and the central benchmark claim remains independently checkable against held-out reference models.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper contributes no fitted constants and no new scientific entities; its central claims rest on domain assumptions about the simulators, the perturbation class, the de-identification, and transfer to real biology, most of which the authors partially acknowledge in Section 6. The only hand-set quantities are analysis constants such as the Gaussian noise scale in the robustness check and the simulation step/cap in curation, neither of which is fit to the models. The only non-standard construction is the species-knockout perturbation used in the Appendix E.1 ablation, which the authors explicitly call 'our custom design rather than a standard SBML operation'; it is an experimental tool inside the testbed, not a postulated mechanism or scientific entity.

free parameters (2)
  • Gaussian noise scale for initial conditions in robustness analysis (Figure 7) = unspecified; variance proportional to original concentration
    Chosen by hand and not reported numerically; the qualitative overfitting trend is consistent across models, but the exact degradation magnitudes are not reproducible from the text.
  • Simulation duration pipeline constants (0.05 s step, 10,000 s cap, 1e-6 steady-state rate threshold) = 0.05 s, 10000 s, 1e-6
    Hand-chosen in Appendix C.3 to standardize curation; they determine which dynamics are visible to agents and thus what the benchmark measures, but they are not fit to data.
assumptions (5)
  • domain assumption SBML ODE simulation in Tellurium faithfully realizes the reference system, and the initial-concentration perturbation protocol implements the intended experiment.
    Section 3.2; the benchmark's data-generating process is taken as ground truth without validation against wet-lab behavior.
  • domain assumption The hidden reaction network is identifiable from the allowed perturbation class (initial concentration changes with full observation of all species).
    Section 5 and Appendix E.1; the paper itself cites Villaverde et al. (2016) stating the perturbations cannot guarantee complete recovery.
  • domain assumption Retained species names in de-identified models do not leak the hidden structure to the agent.
    Appendix C.2: names are kept 'such that biological entities remain identifiable'; names such as ATP or Glucose carry strong prior network information, so the assumption is questionable.
  • domain assumption Agent performance on noise-free deterministic simulations transfers to real scientific capability.
    Section 6 lists this as the Simulation to Reality Gap limitation; the paper's motivating claim depends on this transfer.
  • standard math Standard numerical integration (CVODE) and steady-state solving (NLEQ2) are reliable for the curated models.
    Appendix C.3; these are unproved background tools relied on for data generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab." pith.science (2026). https://pith.science/paper/TWAZLWRK

@misc{pith2026250702083,
  author       = {Pith},
  title        = {Pith review of: Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWAZLWRK}},
  note         = {Machine review of arXiv:2507.02083}
}
read the original abstract

Designing experiments and result interpretations are core scientific competencies, particularly in biology, where researchers perturb complex systems to uncover the underlying systems. Recent efforts to evaluate the scientific capabilities of large language models (LLMs) fail to test these competencies because wet-lab experimentation is prohibitively expensive: in expertise, time and equipment. We introduce SciGym, a first-in-class benchmark that assesses LLMs' iterative experiment design and analysis abilities in open-ended scientific discovery tasks. SciGym overcomes the challenge of wet-lab costs by running a dry lab of biological systems. These models, encoded in Systems Biology Markup Language, are efficient for generating simulated data, making them ideal testbeds for experimentation on realistically complex systems. We evaluated six frontier LLMs on 137 small systems, and released a total of 350 systems. Our evaluation shows that while more capable models demonstrated superior performance, all models' performance declined significantly as system complexity increased, suggesting substantial room for improvement in the scientific capabilities of LLM agents.

Figures

Figures reproduced from arXiv: 2507.02083 by the authors.

Figure 1
Figure 1. SCIGYM simulates scientific discovery. Real-world scientists (left) iteratively refine their hypotheses by designing experiments, collecting lab data, and analyzing results. In SCIGYM (right), agents request experimental actions, receive the observation data simulated from the perturbed reference system, and then perform analyses in a Python shell to refine their hypotheses. setup in which experiments are expensive … view at source ↗
Figure 2
Figure 2. Biological process in The Michaelis–Menten enzymatic process. The substrate S binds to the enzyme E (left) which catalyzes the formation of the product P (right). Reduced SBML Representation. For our purposes, an SBML is a 4-tuple (S, Θ, R, T ) ∈ S consisting of its listOfSpecies S := {Sj} n j=1, listOfParameters Θ, listOfReactions R := {Ri} m i=1, and all other tags T . Each reaction Ri := (Ri , Pi , Mi , θi , ri ,… view at source ↗
Figure 3
Figure 3. Open-ended scientific iteration Agents must discover all components of missing reactions and can update the SBML models using Python code (left). SBML models are encoded as structured XML files, with reactions consisting of five components: reactants, products, modifiers, kinetic laws and parameters (right). 3 SCIGYM Benchmark 3.1 Data Curation The full curation pipeline is shown in Figure S1, and we describe briefl… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: SBML models serve as data simulators for vari￾ous experimental perturbations. The agent proposes pertur￾bations to the system, such as changing its initial conditions and can receive the observed data of the perturbed system. Action Space. In each iteration, the agent …
Figure 5
Figure 5. Figure 5: All agents outperform their respective zero-shot baselines. We compare each agent’s performance with zero-shot prompting. Each marker represents performance on one system, with circles representing pro models and squares representing mini models. Blue shows zero-shot p…
Figure 6
Figure 6. Figure 6: Agents’ performance decline as the under￾lying system’s complexity grows. Do models learn from experiments? As an initial validation, we implemented a baseline direct prompting of the models. For fair comparison, we employed similar ReAct prompts across conditions, exc…
Figure 7
Figure 7. Figure 7: Agents’ proposed systems are sensitive to perturbing initial conditions. How robust are the proposed sys￾tems? We conducted analyses to determine whether agents’ proposed SBML models generalize to different initial conditions or overfit to the specific experimental dat…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages

  1. [1]

    You can also choose to perturb the system and see how the system responds

    Request E x p e r i m e n t s : You can request e x p e r i m e n t s to gather data from the true b i o l o g i c a l system you are studying . You can also choose to perturb the system and see how the system responds . This will help you better u n d e r s t a n d the me ch an ism of the system

    work page

  2. [2]

    You can use several s c i e n t i f i c com pu ti ng li br ar ie s and c u s t o m i z e d f un ct io ns

    Write Code : You have access to a Python e n v i r o n m e n t to run analysis . You can use several s c i e n t i f i c com pu ti ng li br ar ie s and c u s t o m i z e d f un ct io ns . Your code is executed as provided , so ensure your syntax is correct

    work page

  3. [3]

    r ea ct io n1

    Submit the model : You can choose to submit the model and end the process if you think your h y p o t h e s i s c o m p l e t e l y explains the me ch an is m . <! - - BEGIN E X P E R I M E N T A L _ A C T I O N S --> <! - - END E X P E R I M E N T A L _ A C T I O N S --> ## Code Ex ec ut io n For your code , print the results you want to see , and we wil...

    work page

  4. [5]

    Can not be parsed by libsbml (49)

    work page

  5. [6]

    Can not be simulated with Tellurium (19)

    work page

  6. [7]

    Have no reactions (106)

    work page

  7. [8]

    Have rules (422) This resulted in 350 SBML models out of the original 1,096 manually curated instances. 0 200 400 600 800 Number of Species 100 101 102 Frequency 0 100 200 300 400 Number of Reactions 100 101 102 Frequency 0 2 4 6 Wall-Clock Time (s) 10 2 10 1 100 Density Figure S2: Number of species (left), reactions (middle), and simulation wall-clock ti...

    work page

  8. [9]

    We remove optional metadata fields from the SBML entities

    Strip metadata. We remove optional metadata fields from the SBML entities. These include metaid, notes, annotations, model history, control vocabulary, dates, author information, and names*

    work page

Show all 14 references
  1. [10]

    We shuffle the order of parameters, reactions, species, and compartments

    Shuffle components. We shuffle the order of parameters, reactions, species, and compartments

  2. [11]

    We rename the ids of all components in the model to a unique 4 character alphanumeric identifier

    Renaming ids. We rename the ids of all components in the model to a unique 4 character alphanumeric identifier. *Species names are kept such that biological entities in the system remain identifiable. 21 C.3 Determining Simulation Timescale Each SBML instance in the BioModels ...

  3. [12]

    This step may fail for some models due to numerical issues or the absence of a steady state

    Steady-state analysis: For each model, we first attempt to solve for the steady state of the ODE system (when every species has a rate of change less than 10−6) using the NLEQ2 algorithm. This step may fail for some models due to numerical issues or the absence of a steady state

  4. [13]

    We use a fixed step size of t = 0.05 seconds

    Time-course simulation: We then simulate the ODE system by integrating over time with the following specifications: We use the integrator defined in the original SED-ML file, defaulting to CVODE if missing. We use a fixed step size of t = 0.05 seconds. We simulate until at lea...

  5. [14]

    1.0" encoding=

    Final duration determination: We select the final simulation duration by taking the maximum value among the time required to reach steady state (if successfully solved in step 1), the end time of the time-course simulation, and the original end time specified in the SED-ML fil...

    2019

  6. [2014]

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M

    URL https://api.semanticscholar.org/CorpusID:15367821. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi....

    2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.