REVIEW 3 major objections 4 minor 50 references
VASP Agent: An Agentic Framework for Autonomous First-principles Calculations
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read An LLM agent that wraps VASP calculations in expert-defined workflows and guardrails completes 98% of automation tasks, up from 66% for a standalone GPT-4o, and raises result accuracy by more than 27 points.
desk verdict A useful, reusable benchmark and an honest failure analysis are the real contributions; the accuracy metric partly measures consistency with the authors' own reference calculations, and a promised pymatgen comparison is missing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the workflow library, which formalizes expert procedures as sequences of modular components (file I/O, command execution, data parsing, and GetLLMAnswer). GetLLMAnswer generates VASP parameters by filling hierarchical prompt templates that embed domain background, intermediate outputs, and strict output-format constraints. This forces the LLM to produce executable, internally consistent inputs and to respect interdependencies across steps (e.g., matching IBRION with POTIM, keeping cell-relaxation settings fixed across NEB states), preventing the tag-initialization and context-loss failures observed in standalone use.
What would settle it
Hold out a set of materials whose band gaps, adsorption energies, and barriers are known from experiment or from an independent DFT implementation using different parameters; if the agent's accuracy on this blind set drops substantially, the claimed gains are in part self-consistency rather than general scientific competence.
Extended reading notes
Core claim
By embedding domain expertise into an agentic loop rather than relying on prompt engineering alone, the system claims to make multi-step first-principles calculations reliable enough for autonomous execution. It selects a workflow from a library of expert-defined procedures, executes each step through reusable components—file I/O, command execution, parsing, and an LLM-driven parameter generator—and uses runtime evidence to enforce convergent, well-posed settings. Across six LLMs and four task types (structural relaxation, band structure, adsorption energy, transition states), the agent lifts completion and accuracy on every model; GPT-4o completion rises from 66.46% to 97.92% and accuracy f
Load-bearing premise
The benchmark's 'ground truth' values are not fully external: band gaps without experimental data were determined by the authors with hybrid functionals, and adsorption/transition-state energies were computed with the authors' own VASP settings, so the reported accuracy gains may partly measure agreement with the benchmark's own conventions rather than physical correctness.
Editorial extensions
If this is right
- Open-source models integrated with the agent reach completion rates above 90%, narrowing the gap with proprietary models and making secure local deployment feasible for sensitive data.
- Reasoning-oriented models benefit disproportionately, suggesting that long-range planning consistency is a key bottleneck that the agent helps address.
- The agent's failure recovery means tasks that would crash a fixed pipeline can be continued, reducing wasted computation in high-throughput settings.
- The new benchmark offers a reproducible suite of 80 tasks with reference data for evaluating future agents dedicated to computational materials science.
Reading between the lines
- Because the accuracy metric is ratio-based and the references were partly generated with the same VASP conventions the agent is coached to reproduce, a blind external test is needed to distinguish self-consistency from physical accuracy.
- The framework depends on a hand-built workflow library, so novel tasks outside that library would require human extension; true open-ended discovery would demand the agent to construct new workflows, which it does not do.
- The same decomposition pattern (workflow + reusable components + runtime evidence) could transfer to other simulation codes by swapping the parameter templates, turning this into a general recipe for autonomous computational experiments.
- The persistent low transition-state accuracy indicates that for the hardest tasks the agent acts as a parameter-tuner rather than a physics reasoner; an agent that models convergence behavior could be a next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents VASP Agent, an LLM-driven agentic framework for executing VASP first-principles calculations. The framework combines a predefined workflow library, modular components (file I/O, command execution, parsing), hierarchical prompts with domain background, and deterministic runtime tools to generate INCAR parameters, run calculations, and parse outputs. The authors construct a benchmark of 80 tasks across four categories — structural relaxation (40 tasks), band structure (24), adsorption energy (10), and transition state (6) — and evaluate six LLMs with and without the agent using completion rate and a ratio-based accuracy metric. They report substantial improvements (e.g., GPT-4o completion rate rising from 66.46% to 97.92% and accuracy from 45.74% to 73.07%). The paper also analyzes failure modes and compares open-source vs proprietary and reasoning vs standard models.
Significance. If the reported gains are robust, the paper makes a useful contribution to automating routine DFT workflows. Its assets include a concrete multi-task benchmark, public code/data links, evaluation across six LLMs, and candid acknowledgment that transition-state accuracy remains low. The framework's design — deterministic tools plus domain-specialized prompts and guardrails — is sensible and likely to improve pipeline reliability. However, the evaluation methodology has two load-bearing weaknesses: no repeated trials despite LLM stochasticity, and a benchmark whose reference values are partly generated with the same parameter conventions the agent is coached to follow. These weaken the broader accuracy and physical-coherence claims. With additional validation, the work could become an important resource for the AI-for-materials community.
major comments (3)
- [Methods §4.2 and Results §2.3 (Figures 2–6)] All reported completion and accuracy scores are point estimates; the manuscript does not state the number of independent runs or report error bars or confidence intervals. LLM outputs are stochastic, and per-task counts are small (40 SR, 24 BS, 10 AE, 6 TS), so a single failed run can shift a completion rate by 2.5–16.7 percentage points. The claim of 'robust and consistent gains' therefore lacks statistical support. Please provide repeated runs (at least five per condition), report mean ± standard deviation or confidence intervals, and state the temperature/sampling settings.
- [Methods §4.1 (reference generation)] Band-gap references are taken from experiment/literature where available, but otherwise 'determined using hybrid functional calculations' by the authors; all adsorption and transition-state references are computed with the authors' chosen VASP settings (PBE, DFT-D3, 450 eV cutoff, 4×4×1 k-mesh, CI-NEB force convergence <0.05 eV/Å). Since the agent's hierarchical prompts and workflow library embed the same domain conventions, the accuracy metric (min/max ratio in §4.2.2) can be satisfied by reproducing the benchmark's own parameter choices. This does not establish physical correctness or an independent 'high-fidelity' ground truth. Please (i) report how many references are self-computed, (ii) add an independent validation set with experimental or otherwise externally determined values, or (iii) show that accuracy is insensitive to plausible variations in the reference settings.
- [Results §2.3 and Methods §4.2.1 (completion vs correctness)] The completion metric awards full credit for execution-level success: for band structure it requires only that the calculation 'completes without errors and produces electronic eigenvalues along specified paths'; for transition states it awards 6 of 10 points for NEB convergence regardless of barrier correctness. The paper's own results show transition-state accuracy remains 'persistently low' despite near-full completion. The abstract's phrase 'reliable end-to-end computational execution' is therefore overstated: high completion rates demonstrate pipeline robustness, not scientifically valid results. Please temper the claims or revise the completion metric to include a physical-validity check (e.g., verifying the band-gap value or barrier against a tolerance).
minor comments (4)
- [Discussion] The Discussion states 'benchmark dataset of over 100 materials' whereas §2.2 says '80 practical application computing scenarios'; the task counts in Table 1 sum to 80. Please reconcile the numbers.
- [Methods §4.2.2 (Transition State)] The text says a score is received only if the predicted value falls within a 10% relative error threshold, but the equations give a continuous min/max ratio with no threshold. Please specify which behavior is intended, and if the threshold is used, implement it in the scoring equations.
- [Figures 3–6] The bar charts aggregate many tasks but the manuscript does not provide a per-task, per-model numerical table. Adding such a table would support the reported percentages and improve reproducibility.
- [Methods §4.1 (Adsorption Energy)] Minor wording: DFT-D3 is a dispersion correction; 'further correct intermolecular forces' should be rephrased as 'account for dispersion interactions' or similar.
Circularity Check
No significant circularity: the agent's outputs are not derived from the benchmark references by construction.
full rationale
The paper's central claim is that an LLM-agent framework improves the completion rate and accuracy of VASP calculations relative to standalone LLMs. This claim is backed by an empirical benchmark, not by a derivation in which a predicted quantity is defined in terms of the reference. The accuracy metrics compare agent-produced values to independently tabulated reference values (e.g., Materials Project settings, experimental/literature band gaps, or author-computed hybrid-functional results). No equation in the paper makes the prediction equal to the reference by construction, and no model parameter is fitted to the benchmark labels. The fact that some references were generated by the authors using VASP is a benchmark-construction choice; it may affect external validity or fairness, but it does not make the evaluation circular because the agent does not receive the reference values as inputs and must still correctly generate INCAR tags, run VASP, and parse outputs. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in solely via citation. The paper is self-contained as an empirical study of an engineering system, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- SOAP descriptor hyperparameters =
cutoff=5.0 Å, nmax=8, lmax=6, sigma=0.5 Å
- TS relative-error threshold =
10%
- SR reference INCAR parameters =
Materials Project recommended settings
- k-point density and cutoff for adsorption/TS references =
1/25 Å^-1 spacing; 8x8x8 bulk and 4x4x1 slab grids; 450 eV cutoff
assumptions (4)
- domain assumption DFT/PBE/hybrid calculations in VASP are valid ground truth for band gaps, adsorption energies, and transition-state barriers.
- ad hoc to paper The predefined workflow library correctly encodes best-practice calculation procedures.
- domain assumption Single-run LLM outputs without repeated sampling are representative of model performance.
- domain assumption SOAP cosine similarity is a valid measure of structural accuracy.
Cite this review
Pith. "Pith review of VASP Agent: An Agentic Framework for Autonomous First-principles Calculations." pith.science (2026). https://pith.science/paper/HE3IGIEA
@misc{pith2026251219458,
author = {Pith},
title = {Pith review of: VASP Agent: An Agentic Framework for Autonomous First-principles Calculations},
year = {2026},
howpublished = {\url{https://pith.science/paper/HE3IGIEA}},
note = {Machine review of arXiv:2512.19458}
}
read the original abstract
Large Language Models (LLMs) are increasingly embedded in agentic frameworks for scientific discovery. First-principles materials computation imposes a demanding standard for autonomy: successful execution depends on internally consistent inputs, supervision of long-running calculations, and verified outputs. Here we present VASP Agent, a coding-agent-centered system that combines reusable domain skills, deterministic tools, workspace-state inspection, runtime evidence, and scientific guardrails to execute multi-step VASP calculations. The system is evaluated across multiple tasks including structural relaxation, bandgap calculation, equilibrium lattice constant determination, and CO/Pt(111) adsorption. VASP Agent completes all evaluated cases, and its computed numerical results are compared with those obtained using pymatgen and other agentic tools. When large deviations occur, the calculation parameters produced by VASP Agent are more appropriate than those produced by LLM-based workflows. Failure analysis shows that errors that terminate fixed pipelines can be diagnosed and recovered under agentic control.
Reference graph
Works this paper leans on
-
[1]
npj Artificial Intelligence1(1), 14 (2025)
Zhang, Y., Khan, S.A.,et al.: Exploring the role of large language models in the scientific method: from hypothesis to discovery. npj Artificial Intelligence1(1), 14 (2025)
2025
-
[2]
arXiv preprint arXiv:2406.10833 (2024)
Zhang, Y., Chen, X., et al.: A comprehensive survey of scientific large lan- guage models and their applications in scientific discovery. arXiv preprint arXiv:2406.10833 (2024)
arXiv 2024
-
[3]
arXiv preprint arXiv:2505.13259 (2025)
Zheng, T., Deng, Z., et al.: From automation to autonomy: A survey on large language models in scientific discovery. arXiv preprint arXiv:2505.13259 (2025)
arXiv 2025
-
[4]
arXiv preprint arXiv:2311.07361 (2023)
AI4Science,M.R.,Quantum,M.A.:Theimpactoflargelanguagemodelsonscien- tific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361 (2023)
arXiv 2023
-
[5]
https://arxiv.org/abs/2310.07984
Zheng, Y., Koh, H.Y., et al.: Large Language Models for Scientific Synthesis, Inference and Explanation (2023). https://arxiv.org/abs/2310.07984
arXiv 2023
-
[6]
Chemical science (2025)
Ramos, M.C., Collison, C.J., White, A.D.: A review of large language models and autonomous agents in chemistry. Chemical science (2025)
2025
-
[7]
arXiv preprint arXiv:2312.07559 (2023)
Lála, J., O’Donoghue, O., et al.: Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559 (2023)
arXiv 2023
-
[8]
https://arxiv.org/abs/2404.10981
Huang, Y., Huang, J.: A Survey on Retrieval-Augmented Text Generation for Large Language Models (2024). https://arxiv.org/abs/2404.10981
arXiv 2024
Show all 50 references
-
[9]
https://arxiv.org/abs/2402.19473
Zhao, P., Zhang, H., et al.: Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024). https://arxiv.org/abs/2402.19473
2024 arXiv
-
[10]
npj Computational Materials10(1), 251 (2024)
Prince, M.H., Chan, H.,et al.: Opportunities for retrieval and tool augmented large language models in scientific facilities. npj Computational Materials10(1), 251 (2024)
2024
-
[11]
arXiv preprint arXiv:2504.14891 (2025) 17
Gan, A., Yu, H., et al.: Retrieval augmented generation evaluation in the era of large language models: A comprehensive survey. arXiv preprint arXiv:2504.14891 (2025) 17
2025 arXiv
-
[12]
Bran, A.M., Cox, S., et al.: Chemcrow: Augmenting large-language models with chemistry tools (2023) arXiv:2304.05376 [physics.chem-ph]
2023 arXiv
-
[13]
arXiv preprint arXiv:2409.00135 (2024)
Zhang, H., Song, Y., et al.: Honeycomb: A flexible llm-based agent system for materials science. arXiv preprint arXiv:2409.00135 (2024)
2024 arXiv
-
[14]
https://arxiv.org/abs/2402.06852
Zhang, D., Liu, W., et al.: ChemLLM: A Chemical Large Language Model (2024). https://arxiv.org/abs/2402.06852
2024 arXiv
-
[15]
https://arxiv.org/abs/2406.03949
Zhang, K., Zeng, S., et al.: UltraMedical: Building Specialized Generalists in Biomedicine (2024). https://arxiv.org/abs/2406.03949
2024 arXiv
-
[16]
Nature communications15(1), 1418 (2024)
Dagdelen, J., Dunn, A.,et al.: Structured information extraction from scientific text with large language models. Nature communications15(1), 1418 (2024)
2024
-
[17]
npj Computational Materials11(1), 194 (2025)
Odobesku, R., Romanova, K.,et al.: Agent-based multimodal information extraction for nanomaterials. npj Computational Materials11(1), 194 (2025)
2025
-
[18]
https://arxiv.org/abs/2508.20729
Cheng, A., Zhang, L., He, G.: Re4: Scientific Computing Agent with Rewriting, Resolution, Review and Revision (2025). https://arxiv.org/abs/2508.20729
2025
-
[19]
https://arxiv.org/abs/2407.13168
Tian, M., Gao, L., et al.: SciCode: A Research Coding Benchmark Curated by Scientists (2024). https://arxiv.org/abs/2407.13168
2024 arXiv
-
[20]
https://arxiv.org/abs/2410
Yang, Z., Liu, W., et al.: MOOSE-Chem: Large Language Models for Rediscover- ing Unseen Chemistry Scientific Hypotheses (2025). https://arxiv.org/abs/2410. 07076
2025
-
[21]
https://arxiv.org/ abs/2404.07738
Baek, J., Jauhar, S.K., et al.: ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025). https://arxiv.org/ abs/2404.07738
2025 arXiv
-
[22]
https://arxiv.org/abs/2504.07421
Abaskohi, A., Ramesh, A.V., et al.: AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery (2025). https://arxiv.org/abs/2504.07421
2025
-
[23]
https://arxiv.org/abs/2505.13400
Ghareeb, A.E., Chang, B., et al.: Robin: A multi-agent system for automating scientific discovery (2025). https://arxiv.org/abs/2505.13400
2025 arXiv
-
[24]
Ruan, Y., Lu, C.,et al.: An automatic end-to-end chemical synthesis development platformpoweredbylargelanguagemodels.Naturecommunications15(1),10160 (2024)
2024
-
[25]
arXiv preprint arXiv:2408.06292 (2024)
Lu, C., Lu, C., et al.: The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292 (2024)
2024 arXiv
-
[26]
Proceedings of the National Academy of Sciences122(5), 2401238121 (2025) 18
Musslick, S., Bartlett, L.K.,et al.: Automating the practice of science: Oppor- tunities, challenges, and implications. Proceedings of the National Academy of Sciences122(5), 2401238121 (2025) 18
2025
-
[27]
Oxford University Press (2024)
Ouzounis, C.A.: Biology’s transformation: from observation through experiment to computation. Oxford University Press (2024)
2024
-
[28]
ACS Publications (2016)
Kofke, D.A., Siepmann, J.I., et al.: Molecular modeling and simulation in JCED. ACS Publications (2016)
2016
-
[29]
Progress in Biophysics and Molecular Biology117(1), 99– 106 (2015) https://doi.org/10.1016/j.pbiomolbio.2014.10.001
Cooper, J., Vik, J.O., Waltemath, D.: A call for virtual experiments: Accelerating the scientific process. Progress in Biophysics and Molecular Biology117(1), 99– 106 (2015) https://doi.org/10.1016/j.pbiomolbio.2014.10.001
2015 doi
-
[30]
arXiv preprint arXiv:2504.08621 (2025)
Zhang, T., Liu, Z., et al.: Mooseagent: A llm based multi-agent framework for automating moose simulation. arXiv preprint arXiv:2504.08621 (2025)
2025
-
[31]
55–65 (2024)
Sun, J., Li, A.,et al.: Chatmol copilot: An agent for molecular modeling and computationpoweredbyllms.In:Proceedingsofthe1stWorkshoponLanguage+ Molecules (L+ M 2024), pp. 55–65 (2024)
2024
-
[32]
https://arxiv.org/abs/2508.07035
Liu, J., Zhu, T., et al.: VASPilot: MCP-Facilitated Multi-Agent Intelligence for Autonomous VASP Simulations (2025). https://arxiv.org/abs/2508.07035
2025 arXiv
-
[33]
Preprint / published online Aug 19, 2025 (2025)
Han, X.-Q., Gao, Z.-F., et al.: PhysAgent: A Multi-Agent Approach to the Auto- mated Discovery of Physical Laws. Preprint / published online Aug 19, 2025 (2025). https://doi.org/10.32388/J2MXUW
2025 doi
-
[34]
https://arxiv.org/abs/2311.17869
Li, Y., Zhan, J.: SAIBench: A Structural Interpretation of AI for Science Through Benchmarks (2023). https://arxiv.org/abs/2311.17869
2023 arXiv
-
[35]
https://arxiv.org/abs/ 2503.13503
Qin, C., Chen, X., et al.: SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models (2025). https://arxiv.org/abs/ 2503.13503
2025 arXiv
-
[36]
Physical review B54(16), 11169 (1996)
Kresse, G., Furthmüller, J.: Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Physical review B54(16), 11169 (1996)
1996
-
[37]
https:// arxiv.org/abs/2412.19437
DeepSeek-AI, Liu, A., et al.: DeepSeek-V3 Technical Report (2025). https:// arxiv.org/abs/2412.19437
2025 arXiv
-
[38]
https://arxiv.org/abs/ 2410.21276
OpenAI, Hurst, .A., et al.: GPT-4o System Card (2024). https://arxiv.org/abs/ 2410.21276
2024 arXiv
-
[39]
https://arxiv.org/abs/ 2505.09388
Yang, A., Li, A., et al.: Qwen3 Technical Report (2025). https://arxiv.org/abs/ 2505.09388
2025 arXiv
-
[40]
System card, Ope- nAI (April 2025)
OpenAI: Openai o3 and o4-mini system card. System card, Ope- nAI (April 2025). Version: April 16, 2025. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf
2025
-
[41]
https://arxiv.org/abs/2507.06261
Comanici, G., Bieber, E., et al.: Gemini 2.5: Pushing the Frontier with Advanced 19 Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabil- ities (2025). https://arxiv.org/abs/2507.06261
2025 arXiv
-
[42]
https://www.anthropic.com/news/ claude-3-7-sonnet
Anthropic: Claude 3.7 Sonnet. https://www.anthropic.com/news/ claude-3-7-sonnet. Accessed: 2025-11-05 (2025)
2025
-
[43]
Springer
Springer Nature: SpringerMaterials: The Landolt-Börnstein Database. Springer. Accessed: 2024-11-06 (2024)
2024
-
[44]
Journal of the American Chemical Society121(33), 7644–7652 (1999)
Bleakley, K., Hu, P.: A density functional theory study of the interaction between co and o on a pt surface: Co/pt (111), o/pt (111), and co/o/pt (111). Journal of the American Chemical Society121(33), 7644–7652 (1999)
1999
-
[45]
Journal of the American Chemical Society121(34) (1999)
Zhang, C., Hu, P., Alavi, A.: A general method for co oxidation on close-packed transition metal surfaces. Journal of the American Chemical Society121(34) (1999)
1999
-
[46]
Physical review letters100(13), 136406 (2008)
Perdew, J.P., Ruzsinszky, A.,et al.: Restoring the density-gradient expansion for exchange in solids and surfaces. Physical review letters100(13), 136406 (2008)
2008
-
[47]
The Journal of chemical physics132(15) (2010)
Grimme,S.,Antony,J.,etal.:Aconsistentandaccurateabinitioparametrization of density functional dispersion correction (dft-d) for the 94 elements h-pu. The Journal of chemical physics132(15) (2010)
2010
-
[48]
ACS Catalysis 13(1), 433–444 (2022)
Wang, J., Xu, H.,et al.: Rational design of pdag catalysts for acetylene selective hydrogenation via structural descriptor-based screening strategy. ACS Catalysis 13(1), 433–444 (2022)
2022
-
[49]
ACS Catalysis6(10), 6730–6738 (2016)
Jørgensen, M., Gronbeck, H.: First-principles microkinetic modeling of methane oxidation over pd (100) and pd (111). ACS Catalysis6(10), 6730–6738 (2016)
2016
-
[50]
The Journal of chemical physics113(22), 9901–9904 (2000) 20
Henkelman, G., Uberuaga, B.P., Jónsson, H.: A climbing image nudged elastic band method for finding saddle points and minimum energy paths. The Journal of chemical physics113(22), 9901–9904 (2000) 20
2000
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.