REVIEW 4 major objections 5 minor 48 references
MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read MDArena, a 50-task benchmark of real molecular-dynamics workflows, finds that even the strongest coding agent passes full checks on only 24 of 50 tasks (48%), exposing a gap between plausible partial progress and reproducible, fully…
desk verdict A solid, reproducible MD agent benchmark with a mostly credible central finding; the unvalidated verifier is a real but fixable weakness. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the two-part verifier that defines success. Deterministic correctness checks confirm that expected output artifacts are present, valid, and satisfy numerical, structural, or software-level requirements, and the correctness reward is the fraction of criteria met. The process verifier is an LLM-as-judge that reads the agent's full execution trajectory and scores it against task-specific, author-defined criteria covering scientific decision-making and result interpretation, with a median of four criteria per task. Strict-Pass@1, the benchmark's binary success metric, requires every correctness and process check to pass. Reference solutions, executable scripts supplied for 31 of 50 tasks, are required to pass all checks, which is intended to confirm that each task is solvable and that the verifier does not penalize correct workflows. This verifier design is what lets the paper interpret the gap between high partial rewards and low strict success as agent failure rather than benchmark miscalibration.
What would settle it
Have experienced molecular-dynamics researchers independently grade a sample of agent trajectories, especially ones that failed the automated checks, without seeing the automated scores; if a substantial share of failed workflows is judged scientifically valid and reproducible, the verifier is not controlling false positives and Strict-Pass@1 would need renormalization. Conversely, a single trajectory that passes all checks but yields an internally inconsistent simulation, for example a topology whose atom ordering does not match its coordinates, would show the verifier is under-sensitive.
Extended reading notes
Core claim
On its own terms, the paper establishes that the limiting factor in agentic molecular-dynamics automation is not the ability to make plausible progress but the reliability to satisfy every required detail. Agents frequently produce intermediate artifacts that look reasonable yet fail checks designed to catch subtle, compounding errors, such as mismatched residue numbering between file formats or missing include files referenced by a topology. The evaluation's two-part verifier makes this dissociation visible: deterministic artifact checks score the outputs, and an LLM judge scores the execution trajectory against 243 author-defined process criteria; Strict-Pass@1 requires all checks to pass. Because correctness and process rewards are far higher than strict success, the paper concludes that agents routinely 'almost solve' tasks, and that converting partial progress into full success is the frontier. It also reports that harness choice and reasoning effort matter: the extra-high-effort configuration outperforms the medium-effort version mostly on process checks, while operational failures such as timeouts and context-limit errors are often a coupling between model and harness rather than a pure model weakness.
Load-bearing premise
The load-bearing premise is that the two-part verifier reliably separates scientifically valid workflows from invalid ones; the only validation offered is that 31 reference solutions pass all checks, which rules out over-strict false negatives but does not rule out false positives, so if the deterministic checks or the LLM rubric are miscalibrated, both the reported pass rates and the interpretation of the correctness/process gap would change.
Editorial extensions
If this is right
- Even the strongest evaluated configuration fails roughly half of realistic tasks, so for now agent outputs should be treated as drafts to be reviewed by a human rather than as finished simulation protocols.
- Because open-weight models trail sharply (12% and 2% strict success), closing the gap for accessible models will require more than better tooling; the paper argues raw model capability must improve.
- Hard, long-horizon workflows with multiple valid routes, such as membrane-protein system preparation, alchemical free-energy setup, and enhanced-sampling protocols, are the unsolved frontier and are the categories future versions of the benchmark will track.
- Raising reasoning effort changes the outcome: the extra-high-effort configuration achieves nearly the same correctness reward as the medium-effort one but a higher process reward, indicating that deliberation on edge cases is what converts near-misses into strict passes.
- Because each configuration was evaluated once per task, the reported pass rates are practical single-run baselines; repeated-run reliability and stochastic variation remain uncharacterized.
Reading between the lines
- A natural next experiment is to take the near-miss trajectories and feed the failed checks back to the agent as targeted hints, measuring how much of the failure rate is recoverable without any new model capability.
- The same two-part verification scheme could be deployed as a continuous-integration check inside real molecular-dynamics projects, flagging pipelines that produce scientifically questionable artifacts before they are used downstream.
- Because the process score comes from an LLM judge, an expert audit of a random sample of scored trajectories would help establish whether process rewards track scientific validity; if the judge rewards plausible narration over file-level correctness, the interpretation of the correctness/process gap would need revision.
- The single-run protocol means the ordering of configurations near the middle of the table may not be stable; repeated runs with confidence intervals would show which headline differences are real.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MDArena, a benchmark of 50 containerized molecular-dynamics tasks drawn from active biomolecular simulation projects, together with a two-part verifier: deterministic artifact checks for correctness and an LLM-as-judge protocol for process criteria. Six model/harness configurations are evaluated under a common Harbor-based protocol. The best configuration, Codex GPT-5.5 at extra-high reasoning effort, achieves 24/50 Strict-Pass@1 (48%), followed by GPT-5.5 Medium (21/50) and Gemini Flash 3.5 (20/50); open-weight models trail substantially. The paper reports average correctness and process rewards that are much higher than strict pass rates, interprets this as evidence that agents make meaningful partial progress but fail on fine-grained details, and identifies membrane-protein preparation and alchemical free-energy setup as largely unsolved. The benchmark, code, verifier implementations, and reference solutions are publicly released.
Significance. If the verifier is trustworthy, MDArena is a valuable and unusually transparent benchmark: it is grounded in real research workflows, spans diverse systems and methods, ships containerized tasks and reference solutions, and reports standard errors on the headline pass rates. The central qualitative pattern — high partial credit versus low strict success, with hard tasks largely unsolved — is consistent across all six configurations and is robust to several plausible perturbations. The paper also gives credit where due: it acknowledges single-run evaluation, task-selection bias, LLM-judge dependence, and the single-group provenance of tasks. The main risk is that the verifier has been validated only against positive reference solutions, so false positives are uncharacterized; this directly affects the absolute pass rates and the correctness/process gap interpretation. That gap is fixable and does not undermine the benchmark's usefulness as a comparative platform, but it does need to be addressed before the central claims can be accepted as stated.
major comments (4)
- [Sections 3.2 and 3.3] The verifier is validated only by requiring the 31 reference solutions to pass all checks, which guards against false negatives but not false positives. There are no negative-control tasks, no adversarial examples of flawed workflows that the verifier should reject, no inter-judge reliability numbers, and no human-agreement study for the 243 LLM-judged process criteria. Because Strict-Pass@1 rates in Section 5.1 and the correctness/process gap are absolute claims about scientific validity, this validation gap leaves the central numbers unanchored. I would like to see, at minimum, a set of deliberately broken workflows that the verifier must reject, along with a report of judge agreement on a sample of trajectories.
- [Sections 4.1 and 6] Each model-harness configuration is evaluated once per task, so the reported pass rates, including the 24/50 versus 21/50 ordering between GPT-5.5 XHigh and Medium, are single draws. The paper reports binomial standard errors across tasks, but this does not characterize run-to-run stochasticity of the agents themselves. The ordering between GPT-5.5 XHigh and Medium, and between Gemini Flash and Pro, could change under repetition. Please provide repeated-run estimates for at least a subset of tasks, or otherwise temper the ranking claims.
- [Section 5.1] The gap between Strict-Pass@1 and the average correctness and process rewards is, to a nontrivial degree, a compositional consequence of defining strict success as all checks passing. With a median of four process checks per task and additional correctness checks, an agent that fails a single check on many tasks will show high average rewards but low Strict-Pass@1. The paper interprets this gap as evidence that agents 'make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows'; that interpretation needs support from the distribution of per-task check failures (for example, how many failed tasks were near-misses on one check versus complete failures), rather than from averaged rewards alone.
- [Sections 3.1 and 6] The task pool was deliberately curated to expose known agent failure modes, and the paper acknowledges that the resulting success rates 'should not be interpreted as estimates of performance over all routine MD tasks.' This is appropriate, but the abstract and introduction phrase the result as a general statement about 'realistic molecular dynamics workflows' and 'reliability as autonomous MD researchers.' Please make sure the framing in the abstract and Section 1 does not contradict the acknowledged selection bias, so that the 48% figure is not read as a baseline over routine MD practice.
minor comments (5)
- [Abstract and Section 4.1] Model names are inconsistent: the abstract says 'Gemini Flash 3.5' while Section 4.1 and Table 1 use 'Gemini-3.5-Flash' (and similarly for Gemini Pro 3.1). Please unify the naming.
- [Section 5.4.3] The sentence beginning 'Qwen exhibits this failure mode heavily on easy tasks...' contains a comma splice that makes the contrast hard to parse; split it into two sentences.
- [References] Several reference entries are missing a space after 'URL' (for example, the Communications Biology and Mey et al. entries), which renders them as 'URLhttps://...'. Please fix the formatting.
- [Figure 2 caption] The caption says error bars show standard errors across tasks, 'using binomial standard errors for pass/fail success rates,' but it is unclear whether the correctness and process rewards use the same error formula. Please clarify the error-bar definition for those panels.
- [Table A1] Table A1 is very dense. Consider moving it to an online supplement or reformatting it for readability, since it contains important metadata but is hard to scan in its current form.
Circularity Check
No circular derivation: MDArena is an empirical benchmark with disclosed selection bias and unquantified verifier limitations, not a predictive or fitted derivation.
full rationale
MDArena makes no derived quantitative predictions and fits no parameters; its central claims are direct measurements of agent pass rates under a two-part verifier. Every load-bearing element (task construction, verifier design, evaluation protocol) is independently implemented: tasks are containerized artifacts with reference solutions, deterministic correctness checks are programmatic, and the LLM-as-judge process criteria are author-defined rubrics. No equation reduces a claimed result to an input definition. The benchmark's known limitations are disclosed in Section 6: single-run evaluation, LLM-judge model dependence, single-group task provenance, and deliberate selection of tasks expected to expose agent failure modes. That last disclosure means aggregate success rates should not be read as estimates over all routine MD tasks, but it does not make the measured 24/50 Strict-Pass@1 or the correctness/process gap circular: the checks and rubrics would still record a full pass if an agent satisfied them, and the gap between average rewards and binary success is contingent on actual partial progress, not forced by construction. The verifier validation in Section 3.3 (31 reference solutions passing all checks) is a sanity check against false negatives, and the absence of negative controls is a validity threat, not circularity. Self-citations to Hsu et al. and Joshi et al. merely document task provenance and methodology; none carries the argument. Verdict: no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The deterministic checks and process rubrics in MDArena are a valid operationalization of scientifically correct and reproducible MD workflows.
- domain assumption Tasks drawn from a single research group and filtered for difficulty are informative about agent capability on realistic MD workflows.
- domain assumption The agent timeout of 900 seconds, with extensions for selected tasks, does not systematically disadvantage certain configurations.
- standard math Binomial standard errors across tasks are an appropriate uncertainty model for strict-pass rates.
Cite this review
Pith. "Pith review of MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows." pith.science (2026). https://pith.science/paper/QLENAFDI
@misc{pith2026260802642,
author = {Pith},
title = {Pith review of: MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows},
year = {2026},
howpublished = {\url{https://pith.science/paper/QLENAFDI}},
note = {Machine review of arXiv:2608.02642}
}
read the original abstract
Accelerating scientific discovery is among the most consequential applications of AI, and computational biomolecular simulation stands out as a particularly promising target within this broader effort. Coding agents promise to automate significant portions of this workflow, yet their reliability on realistic molecular dynamics (MD) tasks remains poorly characterized. To address this issue, we introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects, spanning 29 molecular systems and 14 broad research protocols, including trajectory analysis, complex system preparation, free-energy protocols, and enhanced sampling. We evaluate six model/harness configurations spanning Codex and OpenCode. Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50. Average correctness and process rewards are substantially higher than strict success rates across all configurations, indicating that agents frequently make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows. Hard tasks remain largely unsolved, particularly membrane-protein system preparation and alchemical free-energy setup, both unsolved or near-unsolved by every evaluated configuration. MDArena thus exposes a substantial gap between the usefulness of coding agents as supervised assistants and their reliability as autonomous MD researchers, while providing a reproducible and extensible platform for tracking progress toward closing it.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
doi:10.5281/zenodo.20953922 , url =
Harbor: A framework for evaluating and optimizing agents and models in container environments , year =. doi:10.5281/zenodo.20953922 , url =
-
[3]
doi:10.48550/arXiv.2505.09388 , url=
Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and others , year=. doi:10.48550/arXiv.2505.09388 , url=. 2505.09388 , archivePrefix=
-
[4]
Computer Physics Communications , volume=
LAMMPS-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales , author=. Computer Physics Communications , volume=. 2022 , publisher=
2022
-
[5]
Journal of Computational Chemistry , volume=
CHARMM-GUI: a web-based graphical user interface for CHARMM , author=. Journal of Computational Chemistry , volume=. 2008 , publisher=
2008
-
[6]
doi:10.48550/arXiv.2312.11805 , url=
Team, Gemini and Anil, Rohan and Borgeaud, Sebastian and Alayrac, Jean-Baptiste and Yu, Jiahui and Soricut, Radu and Schalkwyk, Johan and Dai, Andrew M and Hauth, Anja and Millican, Katie and others , year=. doi:10.48550/arXiv.2312.11805 , url=. 2312.11805 , archivePrefix=
-
[7]
Best practices for alchemical free energy calculations [article v1. 0] , author=. Living Journal of Computational Molecular Science , volume=
-
[8]
ACPYPE-Antechamber python parser interface , author=. BMC Research Notes , volume=. 2012 , publisher=
work page 2012
Show all 48 references
-
[9]
0: the sage small molecule force field , author=
Development and benchmarking of open force field 2.0. 0: the sage small molecule force field , author=. Journal of Chemical Theory and Computation , volume=. 2023 , publisher=
2023
-
[10]
Release , volume=
Rdkit documentation , author=. Release , volume=
-
[11]
Journal of Cheminformatics , volume=
Open Babel: An open chemical toolbox , author=. Journal of Cheminformatics , volume=. 2011 , publisher=
2011
-
[12]
International Conference on Machine Learning , pages=
Openfe: Automated feature generation with expert-level performance , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[13]
Journal of Open Source Software , volume=
alchemlyb: The simple alchemistry library , author=. Journal of Open Source Software , volume=
-
[14]
Computer Physics Communications , volume=
PLUMED 2: New feathers for an old bird , author=. Computer Physics Communications , volume=. 2014 , publisher=
2014
-
[15]
The Journal of Chemical Physics , volume=
The ORCA quantum chemistry program package , author=. The Journal of Chemical Physics , volume=. 2020 , publisher=
2020
-
[16]
PLOS ONE , volume=
Testing for physical validity in molecular simulations , author=. PLOS ONE , volume=. 2018 , publisher=
2018
-
[17]
Journal of Open Source Software , volume=
physical\_validation: A Python package to assess the physical validity of molecular simulation results , author=. Journal of Open Source Software , volume=
-
[18]
Journal of Chemical Information and Modeling , volume=
AmberTools , author=. Journal of Chemical Information and Modeling , volume=. 2023 , publisher=
2023
- [19]
- [20]
-
[21]
and Tanikanti, Aditya and Ke
Pham, Thang D. and Tanikanti, Aditya and Ke. Communications Chemistry , volume =. 2026 , doi =
2026
- [22]
-
[23]
2026 , eprint =
Shi, Zhuofan and A, Hubao and Shao, Yufei and Huang, Dongliang and An, Hongxu and Xin, Chunxiao and Shen, Haiyang and Wang, Zhenyu and Na, Yunshan and Huang, Gang and Jing, Xiang , title =. 2026 , eprint =. doi:10.48550/arXiv.2601.02075 , url =
2026 doi
- [24]
- [25]
- [26]
-
[27]
and Coveney, Peter V
Wan, Shunzhou and Sinclair, Robert C. and Coveney, Peter V. , title =. Philosophical Transactions of the Royal Society A , volume =. 2021 , doi =
2021
-
[28]
Journal of Chemical Theory and Computation , volume=
Alchemical metadynamics: Adding alchemical variables to metadynamics to enhance sampling in free energy calculations , author=. Journal of Chemical Theory and Computation , volume=. 2023 , publisher=
2023
-
[29]
Journal of Chemical Theory and Computation , volume=
Replica exchange of expanded ensembles: A generalized ensemble approach with enhanced flexibility and parallelizability , author=. Journal of Chemical Theory and Computation , volume=. 2024 , publisher=
2024
-
[30]
Wiley Interdisciplinary Reviews: Computational Molecular Science , volume=
Biomolecular simulations: From dynamics and mechanisms to computational assays of biological activity , author=. Wiley Interdisciplinary Reviews: Computational Molecular Science , volume=. 2019 , publisher=
2019
- [31]
-
[32]
ACM Transactions on Software Engineering and Methodology , year=
Large language model-based agents for software engineering: A survey , author=. ACM Transactions on Software Engineering and Methodology , year=
-
[33]
2023 , doi =
Reliability and Reproducibility Checklist for Molecular Dynamics Simulations , journal =. 2023 , doi =
2023
-
[34]
SoftwareX , volume=
GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers , author=. SoftwareX , volume=. 2015 , publisher=
2015
-
[35]
The Journal of Physical Chemistry B , volume=
OpenMM 8: molecular dynamics simulation with machine learning potentials , author=. The Journal of Physical Chemistry B , volume=. 2023 , publisher=
2023
-
[36]
Nature , volume=
Scientific discovery in the age of artificial intelligence , author=. Nature , volume=. 2023 , publisher=
2023
-
[37]
Journal of Computational Chemistry , volume=
MDAnalysis: a toolkit for the analysis of molecular dynamics simulations , author=. Journal of Computational Chemistry , volume=. 2011 , publisher=
2011
-
[38]
Neuron , volume=
Molecular dynamics simulation for all , author=. Neuron , volume=. 2018 , publisher=
2018
-
[39]
2025 , month = apr, url =
2025
-
[40]
Harbor Documentation , year =
-
[41]
Nature Reviews Methods Primers , volume =
Moreau, David and Wiebels, Kristina and Boettiger, Carl , title =. Nature Reviews Methods Primers , volume =. 2023 , doi =
2023
-
[42]
and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R
Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , title =. Advances in Neural Information Processing Systems , year =
-
[43]
Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang and Li, Fuqiang and Ma, Ren and Zheng, Mingzhang and Qian, Bill and Shao, Daniel and Muennighoff...
- [44]
-
[45]
2026 , eprint =
Lin, Minhua and Wu, Juncheng and Wang, Zijun and Shi, Zhan and Sang, Yisi and He, Bing and Liu, Zewen and Wei, Tianxin and Wu, Zongyu and Zhang, Zhiwei and Wang, Dakuo and Zhang, Xiang and Dumoulin, Benoit and Xie, Cihang and Zhou, Yuyin and Wang, Suhang and Lu, Hanqing , titl...
- [46]
-
[47]
Journal of Computer-Aided Molecular Design , volume=
Identifying signatures of proteolytic stability and monomeric propensity in O-glycosylated insulin using molecular simulation , author=. Journal of Computer-Aided Molecular Design , volume=. 2022 , publisher=
2022
-
[48]
Nature Communications , volume=
Structural basis for prostaglandin and drug transport via SLCO2A1 , author=. Nature Communications , volume=. 2026 , publisher=
2026
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.