REVIEW 3 major objections 5 minor 28 references
An LLM acting as the optimizer can calibrate a 10-dimensional health simulation model in 16 evaluations, with accuracy matching or exceeding Bayesian optimization and Nelder–Mead.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 21:48 UTC pith:VJ6NWNB6
load-bearing objection Useful, honest empirical study of LLM-based calibration, but the headline sample-efficiency claim is confounded by the extra structured feedback the LLM gets that BAxUS doesn't. the 3 major comments →
Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternative
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a grey-box calibration loop with the LLM as proposer—not as simulator—can beat state-of-the-art black-box optimizers in sample efficiency. On the unconstrained 10-dimensional anal cancer task the agentic method reaches median best error 5.03e-11 with 16 evaluations; BAxUS reaches 1.34e-9 with 110 and Nelder–Mead 2.87e-8 with about 486. Under the clinical monotonicity constraint the agentic method reaches 1.56e-9 in 16 evaluations, within the same order as BAxUS (1.33e-9) and far better than Nelder–Mead, with constraints expressed only as text in the system prompt. The authors claim the protocol makes each proposal auditable and the search explainable, at the cost of
What carries the argument
The central mechanism is the agentic calibration loop: a harness runs the simulation, checks feasibility, and logs every evaluation, while the LLM receives a system prompt containing the model description, a one-time code summary, parameter semantics and bounds, per-group residual breakdowns, and the full history of guesses, and returns the next parameter vector with a written rationale. This makes the LLM act as a soft acquisition function—an implicit surrogate that trades exploration against exploitation using structured residual feedback—while constraints are enforced both by the prompt instruction and by the harness rejecting infeasible proposals. The loop's sample efficiency comes from
Load-bearing premise
The method's advantage rests on the LLM already knowing, from its training, the qualitative direction in which each parameter moves the simulated outputs—and on the code summary faithfully describing the model; if that prior knowledge is missing or wrong, the agentic search loses its edge and may be misdirected.
What would settle it
Take the same 10-dimensional anal cancer calibration, shuffle the semantic labels of the ten regression probabilities so the LLM is told (or the code summary implies) that increasing a parameter increases incidence when it actually decreases it, and rerun the loop with 16 evaluations; if the median best error no longer beats BAxUS's 1.34e-9, the reported gain depends on prior domain knowledge rather than on the loop structure alone.
If this is right
- Calibration evaluation budgets can shrink by an order of magnitude: 16 model evaluations sufficed on a 10D problem, which matters when each simulation takes minutes.
- Adding or changing clinical constraints becomes a prompt edit rather than new surrogate models and constraint functions.
- Each run yields a human-readable iteration log with rationales, enabling post-hoc audit and post-mortem analysis of failed runs.
- The method is most attractive when simulation cost dominates LLM inference; with faster inference or API backends the wall-clock gap narrows.
- The authors' stated extensions—hybrid LLM+GP validation, LLM selecting acquisition functions, or LLM proposing inside trust regions—suggest the approach can be blended with existing Bayesian optimization machinery.
Where Pith is reading between the lines
- The decisive test of the method is whether the LLM's pretrained knowledge of the domain is doing the work; on a synthetic or proprietary domain without such knowledge, performance would presumably degrade to BAxUS levels or worse, so the claim should be read as conditional on knowledge availability.
- If the code-summarisation step misdescribes how a parameter affects outputs, the agent inherits that error; comparing runs with and without the summary would isolate how much of the gain comes from grey-box code access versus parameter semantics.
- The per-group residual feedback design suggests a natural extension to multi-output calibration where targets have heterogeneous units or clinical importance; the same reasoning structure could weight residuals by stratum.
- The auditable trace could double as a specification for reproducing the calibration in a different model or for automated checks that the search obeyed constraints, turning the log into a compliance artifact.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes 'agentic calibration': an LLM-driven optimization loop for grey-box simulation models in cost-effectiveness analysis. At each iteration, the LLM receives a natural-language model description, parameter semantics, a pre-computed summary of the model source code, the calibration target, and a structured breakdown of per-group residuals; it returns a parameter vector and a written rationale. Constraints are specified as a textual prompt section. The method is evaluated on a 10-dimensional Markov model of anal cancer, comparing against Nelder-Mead (NM) and the Bayesian-optimization method BAxUS, under both unconstrained and constrained calibration. The headline results (Table 2) are that in the unconstrained setting the agentic method reaches median best error 5.03e-11 in 16 model evaluations, versus 1.34e-9 in 110 evaluations for BAxUS and 2.87e-8 in ~486 for NM; in the constrained setting, agentic and BAxUS reach comparable errors (~1.5e-9) with 16 vs 110 evaluations, both far outperforming NM. Wall-clock time is 4-7x higher for the agentic method. The paper is transparent about limitations: no convergence guarantees, reproducibility depends on inference backend, and effectiveness depends on the LLM's pretrained domain knowledge.
Significance. If the sample-efficiency claim holds, the method is a potentially practical alternative for settings where simulation calls are very expensive and where an auditable, constraint-flexible calibration is desired. The paper's strengths are the detailed description of the protocol, the inclusion of full prompts and iteration logs in the appendices, and the explicit acknowledgment of key limitations (domain-knowledge dependence, lack of convergence guarantees, inference cost). The central claim, however, rests on a comparison that is not information-controlled: the agentic method receives per-group residual feedback that BAxUS does not. Because of this, the current evidence does not isolate the contribution of the LLM-driven loop from the contribution of enriched feedback. The significance is therefore conditional on additional experiments or a carefully re-framed claim.
major comments (3)
- [§3.1.2, Table 2, Abstract] The headline comparison is not information-controlled. The agentic loop receives per-iteration structured feedback—bias_by_group, rel_bias_by_group, abs_error_by_group, max_abs_error (Appendix C)—which are the components of the scalar error BAxUS receives as a single number. A black-box optimizer can only infer per-group directional information by evaluating nearby points; the agentic harness supplies it for free, effectively giving gradient-like information without extra simulations. The observed '16 evaluations vs. 110 evaluations' therefore conflates two differences: the optimizer (LLM + prompt) and the information content per evaluation. To support the claim that the LLM-driven loop is the cause of sample efficiency, the paper should include an ablation in which the agentic prompt is reduced to scalar error only, and/or a BO baseline that is given the per-group residual vector (e.g.,
- [§5, Appendix A] The evaluation does not isolate the effect of the LLM's pretrained domain knowledge. The calibration target is the age-specific anal cancer incidence from a published meta-analysis (Clifford et al., 2021), and the model is a standard natural-history Markov model; both are likely represented in the LLM's training corpus. The paper acknowledges in §5 that 'the effectiveness of the method depends on the LLM possessing the domain knowledge,' but no experiment measures the contribution of this prior. A synthetic or less-public model whose parameters are not in the pretraining corpus, or a prompt variant that removes the model-description/code-summary sections, would delineate the method's general optimization ability from its reliance on memorized epidemiology. Since the abstract makes a general claim about 'agentic calibration achieving competitive performance with substantially fewer model
- [§4.2, §5] The practical recommendation that the method is 'particularly suitable when simulation time dominates' (Abstract) is not supported by the experiments. In all reported runs, simulation time is negligible relative to LLM inference; the agentic loop is 4-7x slower in wall-clock time (Table 2: 3,981 s vs. 573 s unconstrained). No experiment or calculation varies the ratio of simulation cost to inference cost, so the crossover point where the evaluation-count advantage becomes a real-time advantage is never demonstrated. A simple sensitivity analysis with a synthetic per-evaluation delay, or at least a formal cost model, is needed to substantiate this central practical claim.
minor comments (5)
- [Figure 2] The age-group labels in the top-row panels appear inconsistent with the parameter list. Table 1 and the iteration logs use groups 30–34 through 75–79, while Figure 2 shows labels 25–29 through 70–74. This should be corrected.
- [§3.2.2] The LLM version is given as 'Qwen3.6 27B,' but the cited reference [26] is the 'Qwen3 technical report.' Please clarify the exact model variant and quantization, and note whether the 4-bit quantization affects the reasoning reliability.
- [§4.1] The unconstrained comparison reports medians over ten runs but no significance test or interval. Given the stochasticity of LLM inference, a paired rank-sum test or bootstrap confidence intervals would strengthen the 'substantially lower' claim.
- [Table 3 / Appendix B] The comparison of modeller-facing effort is qualitative. For BAxUS (PoF), the statement that 'the scale of each constraint function must be chosen' is important; please provide the actual constraint-function scales used in the experiments, since this may affect the constrained BO performance.
- [§3.1.3] The code summarisation step is described as producing a 'factual summary,' but the summary is an LLM output and could be incorrect. The appendix provides one example, but no validation of the summariser's faithfulness is given. A brief discussion of how summarisation errors would affect the loop, or a manual check on a few runs, would be useful.
Circularity Check
No significant circularity; the agentic calibration benchmark is self-contained, with only an acknowledged external-knowledge confound.
full rationale
The paper contains no step in which a claimed result is defined in terms of the quantity it is supposed to derive, and no fitted parameter is relabeled as a prediction. The agentic loop (Sections 3.1.1–3.1.4) is an empirical optimization procedure: the LLM proposes parameter vectors, the harness runs the simulation, and the reported error is computed from the simulated outputs against the calibration target. This is fitting by design, not a derivation of one quantity from another by construction. The paper cites the authors' earlier work ([19], [20]) to justify the choice of NM as a baseline and to reference prior BO calibration methodology, but those citations are not load-bearing for the headline comparison; the central benchmark is an independent empirical evaluation. The acknowledged dependence on pretrained domain knowledge (Section 5: "the effectiveness of the method depends on the LLM possessing the domain knowledge required to reason about the model and its parameters") is a real external-information confound, and the structured per-group residual feedback given to the LLM means the comparison against BAxUS is not information-controlled. However, these are experimental-design and data-leakage concerns, not circular reasoning: the objective values are produced by the simulator, not imported from the LLM or from the paper's own prior claims. No self-definitional step, fitted-input-as-prediction, load-bearing self-citation, imported uniqueness theorem, or ansatz-via-citation is present. Therefore the derivation chain is self-contained and the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- LLM sampling temperature =
0.05
- Per-iteration reasoning budget =
2048 tokens
- Evaluation budgets per method =
16 (agentic), 110 (BAxUS), variable (NM)
- LLM inference configuration =
Qwen3.6 27B, 4-bit quantisation, 128k context
axioms (4)
- domain assumption The simulation model and calibration target are correctly specified by the provided description and source code; the grey-box summary accurately reflects parameter-output effects.
- domain assumption The LLM's pretrained knowledge of epidemiology and clinical direction (e.g., lower HSIL regression increases cancer incidence) is correct for this domain.
- domain assumption Ten runs with different seeds are sufficient to support the comparative claim about solution quality and robustness.
- domain assumption The calibration target from Clifford et al. (2021) is a valid external benchmark.
Cite this review
Pith. "Pith review of Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternative." pith.science (2026). https://pith.science/paper/VJ6NWNB6
@misc{pith2026260718308,
author = {Pith},
title = {Pith review of: Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternative},
year = {2026},
howpublished = {\url{https://pith.science/paper/VJ6NWNB6}},
note = {Machine review of arXiv:2607.18308}
}
read the original abstract
Calibration of grey-box simulation models is a constrained optimization problem in which model evaluations are expensive, the parameter space can be high-dimensional, and the search must respect plausibility constraints. Although the simulation code is fully available to the analyst, the joint effect of multiple parameters remains difficult to predict analytically. Classical optimizers such as Nelder--Mead (NM) are simple to deploy but sample-inefficient, particularly under constraints. Modern Bayesian Optimization methods achieve competitive solutions with far fewer evaluations but require non-trivial modeling machinery for constraint handling. We introduce an agentic calibration method in which a large language model acts as the optimizer, with constraints incorporated as a plain-language section of the system prompt. We evaluate the agentic method, NM, and Bayesian Optimization (BO) on an anal cancer simulation model under both unconstrained and clinically constrained calibration. Under unconstrained calibration, the agentic method achieves substantially lower best error than BO and NM, while requiring fewer model evaluations. Under constrained calibration, the agentic method reaches comparable error levels and both outperform NM. These results are obtained at the cost of increased inference time per iteration. Agentic calibration achieves competitive performance with substantially fewer model evaluations, and constraint handling is essentially free at the modeller-facing interface through simple textual specifications rather than additional modelling machinery. The main trade-off lies in increased per-iteration inference cost, making the approach particularly suitable when simulation time dominates. Beyond performance, the per-iteration rationale makes the search auditable and explainable, so its decisions can be scrutinised and justified to third parties.
Figures
Reference graph
Works this paper leans on
-
[1]
Calibration methods used in cancer simulation models and suggested reporting guidelines.Pharmacoeconomics
Stout NK, Knudsen AB, Kong CY, McMahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines.Pharmacoeconomics. 2009;27:533–545
2009
-
[2]
Calibrating models in economic evaluation: a seven-step approach.Pharmacoeconomics
Vanni T, Karnon J, Madan J, White RG, Edmunds WJ, Foss AM, Legood R. Calibrating models in economic evaluation: a seven-step approach.Pharmacoeconomics. 2011;29(1):35– 49
2011
-
[3]
Taking the human out of the loop: a review of Bayesian optimization.Proceedings of the IEEE
Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N. Taking the human out of the loop: a review of Bayesian optimization.Proceedings of the IEEE. 2016;104(1):148–175
2016
-
[4]
Cambridge University Press; 2023
Garnett R.Bayesian Optimization. Cambridge University Press; 2023. 17
2023
-
[5]
Increasing the scope as you learn: adaptive Bayesian optimization in nested subspaces (BAxUS).Advances in Neural Information Processing Sys- tems
Papenmeier L, Nardi L, Poloczek M. Increasing the scope as you learn: adaptive Bayesian optimization in nested subspaces (BAxUS).Advances in Neural Information Processing Sys- tems. 2022;35
2022
-
[6]
A simplex method for function minimization.The Computer Journal
Nelder JA, Mead R. A simplex method for function minimization.The Computer Journal. 1965;7(4):308–313
1965
-
[7]
High-dimensional Bayesian optimization with sparse axis-aligned subspaces (SAASBO)
Eriksson D, Jankowiak M. High-dimensional Bayesian optimization with sparse axis-aligned subspaces (SAASBO). In:Proceedings of UAI 2021. 2021
2021
-
[8]
Bayesian optimization with inequality constraints
Gardner J, Kusner M, Xu Z, Weinberger K, Cunningham J. Bayesian optimization with inequality constraints. In:Proceedings of ICML 2014. 2014:937–945
2014
-
[9]
Large language models as opti- mizers
Yang C, Wang X, Lu Y, Liu H, Le QV, Zhou D, Chen X. Large language models as opti- mizers. In:Proceedings of ICLR 2024. 2024
2024
-
[10]
Large language models to enhance Bayesian optimization.arXiv:2402.03921
Liu T, Astorga N, Seedat N, van der Schaar M. Large language models to enhance Bayesian optimization.arXiv:2402.03921. 2024
Pith/arXiv arXiv 2024
-
[11]
Zhu Z, Tang Y, Tang X, Zhang J, Gao C, Zhang S, Xu H, Guan T. Large language models as calibrationagentsinhydrologicalmodeling: feasibilityandlimitations.Geophysical Research Letters. 2026;53(2). doi:10.1029/2025GL120043
-
[12]
A review of large language models and autonomous agents in chemistry.Chemical Science
Ramos MC, Collison CJ, White AD. A review of large language models and autonomous agents in chemistry.Chemical Science. 2025;16:2514–2572
2025
-
[13]
Position: Leverage foundational models for black-box optimization
Song X, Tian Y, Lange RT, Lee C, Tang Y, Chen Y. Position: Leverage foundational models for black-box optimization. In:Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR. 2024;235:46168–46180
2024
-
[14]
Methods for the Economic Evaluation of Health Care Programmes
Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford University Press; 2015
2015
-
[15]
Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.JAMA
Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine.JAMA. 2016;316(10):1093–1103
2016
-
[16]
Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1.Value in Health
Caro JJ, Briggs AH, Siebert U, Kuntz KM. Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1.Value in Health. 2012;15(6):796–803
2012
-
[17]
State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3.Medical Decision Mak- ing
Siebert U, Alagoz O, Bayoumi AM, et al. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3.Medical Decision Mak- ing. 2012;32(5):690–700
2012
-
[18]
Bayesian optimization in a billion dimensions via random embeddings.Journal of Artificial Intelligence Research
Wang Z, Hutter F, Zoghi M, Matheson D, de Freitas N. Bayesian optimization in a billion dimensions via random embeddings.Journal of Artificial Intelligence Research. 2016;55:361– 387. 18
2016
-
[19]
Bayesian Optimization with Additive Kernels for a Stepwise Calibration of Simulation Models for Cost-Effectiveness Analysis
Gómez-Guillén D, Díaz M, Arcos JL, Cerquides J. Bayesian Optimization with Additive Kernels for a Stepwise Calibration of Simulation Models for Cost-Effectiveness Analysis. International Journal of Computational Intelligence Systems. 2024;17:249
2024
-
[20]
Bayesian optimization with additive kernels for the calibration of simulation models to perform cost-effectiveness analysis
Gómez-Guillén D, Díaz M, Arcos JL, Cerquides J. Bayesian optimization with additive kernels for the calibration of simulation models to perform cost-effectiveness analysis. In: Artificial Intelligence Research and Development(Proceedings of CCIA 2023). Frontiers in Artificial Intelligence and Applications. IOS Press; 2023
2023
-
[21]
Scalable global optimization via local Bayesian optimization (TuRBO).Advances in Neural Information Processing Systems
Eriksson D, Pearce M, Gardner J, Turner RD, Poloczek M. Scalable global optimization via local Bayesian optimization (TuRBO).Advances in Neural Information Processing Systems. 2019;32
2019
-
[22]
Scalable constrained Bayesian optimization (SCBO)
Eriksson D, Poloczek M. Scalable constrained Bayesian optimization (SCBO). In:Proceed- ings of AISTATS 2021. 2021
2021
-
[23]
Attentionisallyouneed.Advances in Neural Information Processing Systems.2017;30:5998– 6008
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attentionisallyouneed.Advances in Neural Information Processing Systems.2017;30:5998– 6008
2017
-
[24]
Language models are few-shot learners.Advances in Neural Information Processing Systems
Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems. 2020;33:1877–1901
2020
-
[25]
An explanation of in-context learning as implicit Bayesian inference
Xie SM, Raghunathan A, Liang P, Ma T. An explanation of in-context learning as implicit Bayesian inference. In:International Conference on Learning Representations (ICLR). 2022
2022
-
[26]
A meta-analysis of anal cancer incidence by risk group: Toward a unified anal cancer risk scale,
Yang A, Li A, Yang B, Zhang B, Hui B, Zheng B, et al. Qwen3 technical report. arXiv:2505.09388. 2025. 19 Appendix This appendix provides the supporting material for the calibration case study. Appendix A describes the anal cancer simulation model used throughout the experiments, including its struc- ture, the calibrated parameter and the calibration targe...
Pith/arXiv arXiv 2025
-
[27]
<reasoning>...</reasoning> -- 1-4 sentences explaining what you observed in the previous results and why this new guess should improve the fit
-
[28]
Model summary
<guess>{...}</guess> -- a single JSON object with every parameter name above as a key and a number within its bounds as the value. The <guess> block must contain ONLY the JSON object (no comments, no prose). Do NOT call any tool. The application runs the simulation for you and sends back the simulated output and error on the next turn. ## Result fields yo...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.