{"id":"a65aedb8-1a6d-400d-826a-c70a5e6cab49","arxiv_id":"2505.18176","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simultaneous calibration of multiple computer models with the iPro-NC neural network improves high-fidelity emulation, but parameter estimates become non-identifiable in high-dimensional settings.","lead":"This paper asks whether calibrating several computer simulation models at once is better than calibrating them one by one. It introduces a neural network, iPro-NC, that improves emulation of high-fidelity data but loses calibration accuracy in high-dimensional problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The emulation gains attributed to simultaneous calibration (Tables 1, 4, 5) are confounded by the network's bias-correction capacity; no ablation isolates the calibration terms.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption stated as the representational/identifiability assumption of the network. My analysis agrees that identifiability is a concern, but I locate the most load-bearing issue one step earlier: the paper's headline empirical result, improved predictive accuracy from using all sources, does not distinguish simultaneous calibration from simple multi-fidelity emulation with bias correction. The reader also noted the absence of baseline comparisons; my concern sharpens that point by specifying a concrete ablation that would resolve it. I do not find the paper internally inconsistent, and the authors are honest about the high-dimensional calibration failure. However, without the ablation, the positive claim is not causally tied to the method's calibration mechanism. Since the authors themselves limit the claim to emulation accuracy, and the concern is testable rather than decisive, the existing CONDITIONAL verdict remains appropriate; my recommendation is UNCHANGED, with the condition now explicitly including the ablation test.","tokens_in":17003,"tokens_out":4874,"duration_ms":52730,"concrete_test":"Run a calibration-ablated variant of iPro-NC on both the analytic (Section 4.1) and engineering (Section 4.2) examples: keep Blocks 0, 2, 3 and the source embedding but replace Block 1 with a fixed, non-trainable map (e.g., zero the theta input), remove Lcal and LKL from Eq. 2, and train using only Lem_NLL and Lem_IS. Use the same data splits, network sizes, learning rate, and number of epochs. Repeat each setting for at least 5 random seeds and report mean and standard deviation of the RRMSE values corresponding to Tables 1, 4, and 5. If the ablated model's RRMSE is statistically indistinguishable from the full model's, the reported gains cannot be attributed to simultaneous calibration, and the paper's central claim reduces to a known benefit of multi-fidelity emulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's positive evidence is that including all sources improves HF emulation and calibrated LF-to-HF emulation (Tables 1, 4, 5). The central claim 'our method can improve predictive accuracy' is supported only if those gains come from simultaneous calibration rather than from the flexible multi-source emulator. This attribution is not established because the loss (Eq. 2) mixes emulation terms (Lem) with calibration terms (Lcal), and the architecture lets Block 3 absorb arbitrary model-form discrepancies. The paper itself admits the network has 'very large learning capacity' and that this produces non-identifiability (Section 4.1, Figure 4c; Section 4.2: 'most of the Voce hardening parameters are estimated to be at the extrema of their sampling ranges'). If a single flexible network can bias-correct any LF prediction toward the HF source, then the reported emulation improvement may be entirely due to multi-fidelity emulation with bias correction, which is already provided by the prior work cited as [37]. No baseline trains the same architecture with calibration disabled—e.g., fixed theta and no Lcal terms—so the unique contribution of this paper, simultaneous calibration, is not isolated. The paper's own conclusion that iPro-NC is 'only justified if maximizing emulation accuracy is the only goal' further suggests the calibration machinery may not be the source of the observed benefit. This is a causal-attribution gap, not an internal inconsistency, but it is load-bearing because it determines whether the title's question is answered affirmatively.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether multiple low-fidelity computer models should be calibrated simultaneously against a single high-fidelity source, and proposes iPro-NC, a multi-block neural-network framework that combines multi-fidelity emulation with per-source probabilistic calibration. The network encodes a source indicator and categorical variables into latent spaces, maps calibration parameters through a masked block, and outputs per-output normal distributions; the loss in Eq. (2) mixes emulation and calibration negative log-likelihood terms, interval scores, and a KL regularization term. The method is tested on an analytic three-source problem with known functional forms and on a finite-element tensile specimen problem with a Holloman high-fidelity model and two Voce low-fidelity models. The reported results show that training on all sources improves HF emulation and calibrated-LF emulation, while calibration inference is accurate in the low-dimensional analytic case for a source without model-form error, but becomes non-identifiable and initialization-dependent in the higher-dimensional engineering problem.","tokens_in":17325,"tokens_out":9125,"duration_ms":88092,"significance":"The strengths of the paper are its use of external ground truth in both examples, its handling of sources with different numbers of calibration parameters and multiple outputs, and its candid discussion of non-identifiability and initialization sensitivity. If the emulation gains are reproducible and robust, the architecture could be a useful contribution to multi-fidelity data fusion. However, the central attribution is not established: the reported gains from 'all sources' are not separated from having more training data or from the network's bias-correction capacity, and no comparison to any existing calibration method is provided. Because the paper's own conclusion restricts justification to 'maximizing emulation accuracy,' the title's question remains effectively open. The significance is therefore moderate and conditional on additional ablation and baseline experiments.","major_comments":[{"comment":"The text in Section 3.1 states that insensitivity of HF emulation to dummy calibration values is achieved 'by adding a term to the loss function,' but Eq. (2) contains no such term, and Section 5 describes a Jacobian-based dummy-insensitivity term as a scheme that was tried and abandoned. The mechanism that makes Table 1's HF-emulation accuracy insensitive to the dummy θ is therefore unspecified; either a term must be added to Eq. (2) and described in Section 3.2, or the text must be revised to explain how masking alone enforces this insensitivity.","section":"Section 3.1 and Eq. (2)"},{"comment":"The main evidence for the abstract's claim that 'our method can improve predictive accuracy' is the comparison of two-source versus all-source training in Tables 1, 4, and 5. This comparison conflates simultaneous calibration with two other factors: the availability of additional LF training data and the bias-correction capacity of Block 3. There is no ablation that trains the same multi-source network with calibration disabled (e.g., fixed θ and Lcal terms removed), nor a baseline that calibrates each LF source separately while keeping the shared multi-source emulator. Without such an ablation, the observed improvement cannot be attributed to the simultaneous-calibration mechanism that is the paper's subject.","section":"Section 4, Tables 1, 4, 5"},{"comment":"The engineering calibration inference is unreliable: the estimated ν is 0.397 versus the true 0.260, the estimated σ0 for s2 is 767 MPa versus the true 650 MPa, the text reports that 'most of the Voce hardening parameters are estimated to be at the extrema of their sampling ranges,' and the estimates vary with initialization. Because the paper's motivating question is about calibration, and the high-dimensional experiment is the realistic test, the current results imply that the method's calibration inference fails exactly where it matters. The positive claims in the abstract and conclusion should be restricted to emulation accuracy, or the method must be modified and re-evaluated before it can be presented as a calibration framework.","section":"Section 4.2, Table 3"},{"comment":"All quantitative results are from a single training run, yet Section 5 states that 'the model is very complex and its performance depends on initialization' and Section 4.2 reports that calibration estimates vary with initialization. Single-run RRMSE values in Tables 1, 4, and 5 cannot support comparative statements such as 'never the worst performing' or 'significantly improves' without seed sensitivity or repeated-run statistics. The authors should report means and standard deviations over multiple initializations, at least for the main tables.","section":"Section 4, Tables 1–5"},{"comment":"The paper does not compare iPro-NC against any existing method, including Kennedy-O'Hagan calibration, Latent Map GP [11], or the probabilistic neural data-fusion framework [37] on which the architecture relies. Even if the contribution is an exploratory framework, at least one baseline comparison is needed to interpret 'improve predictive accuracy' relative to prior work; without it, the reader cannot judge whether simultaneous calibration offers any advantage over calibrated separate emulation.","section":"Section 2"}],"minor_comments":[{"comment":"In the first block of Table 4, the second row is labeled 's0 ands1' but appears intended to be 's0 ands2'; the duplicated label makes the table misleading.","section":"Table 4"},{"comment":"The statement that 'the network is never the worst performing on any output' is a weak claim because in Table 1 all-source training is not best for y1 (0.0825 vs 0.0677 for s0 and s2); the phrasing should quantify differences or avoid suggestive causal language.","section":"Section 4.1, Table 1"},{"comment":"The interval-score terms are written inconsistently: Eqs. (5) and (6) omit the output subscript i that is used in Eqs. (3) and (4), which makes the summation over outputs in Eq. (2) unclear.","section":"Eqs. (2)–(6)"},{"comment":"The footnote 'We fix εp in both the Holloman and Voce models' conflicts with Eqs. (13) and (14), where σy depends on εp as a variable; this needs rewording or clarification.","section":"Section 4.2"},{"comment":"The captions for subfigures (a) and (b) are nearly identical ('Calibration LS for s2'); the reader would benefit from explicitly stating that (a) uses two sources and (b) uses three sources.","section":"Figure 4"},{"comment":"The clause 'We consider the divergence only between the standard deviations' is vague; it should state that LKL in Eq. (9) penalizes only the estimated standard deviations and not the means.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript relies heavily on the authors' own prior work (refs [11] and [37]), and no comparison to outside methods is provided. This is not by itself grounds for rejection, but it makes the claimed improvement difficult to position with respect to prior art. The conclusion's admission that iPro-NC is only justified for emulation accuracy is surprisingly narrow for a calibration paper; the editor may wish to consider whether the framing and title match the demonstrated scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here is my read on arXiv:2505.18176.\n\nThe paper asks whether calibrating several low-fidelity models at once helps. The honest answer it gives: for emulation, yes; for calibration inference, it depends. iPro-NC extends the authors' earlier probabilistic neural data fusion (Mora et al. 2023) to assign each LF source its own probabilistic calibration parameters, allow different parameter counts per source, and handle multi-response outputs. That is a real, if incremental, step.\n\nThe strongest evidence is the emulation tables. In both the analytic example and the finite element tension problem, training on all sources yields better high-fidelity emulation than any two-source subset. The analytic example has known closed-form truth and the FE example treats Holloman as the high-fidelity reference, so the evaluation is external. The authors also report their failures honestly: calibration posteriors concentrate at sampling-range extrema in the engineering problem, the latent space places s1 closer to s0 than s2 (the opposite of the truth), and results depend on initialization.\n\nThe main gap is attribution. There is no ablation that disables the calibration terms (for example, fixed theta) to show that the emulation gains come from simultaneous calibration rather than from a flexible multi-source emulator with bias correction. Because the loss jointly trains emulation and calibration, the gains are confounded. The paper's own conclusion—use iPro-NC only when emulation accuracy is the goal—effectively concedes this. Repeated-run statistics are missing, no existing calibration method is compared, and the code is promised but not yet released. None of these are fatal, but they need addressing.\n\nNet: a solid engineering contribution with an honest limitation statement. The title question is only partially answered, but the method advances the data-fusion toolbox. I would bring it to reading group and cite it for the emulation results. It deserves peer review; a referee should ask for the calibration ablation, error bars, and at least one baseline comparison.","headline":"A useful, honest extension of neural data fusion that improves multi-fidelity emulation, but the calibration gains are not isolated and high-dimensional inference fails; the title question gets only a partial answer.","tokens_in":17825,"tokens_out":2938,"would_cite":true,"duration_ms":29102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simultaneous calibration of multiple computer models is feasible and improves emulation accuracy, but it makes calibration parameters non-identifiable in high-dimensional problems.","keywords":["Model Calibration","Multi-fidelity Modeling","Uncertainty Quantification","Probabilistic Neural Networks","Inverse Problems","Manifold Learning","Data Fusion"],"falsifier":"Take a synthetic problem with a known true parameter vector $\\theta^{\\ast}$ and two LF sources, one that contains the HF model exactly at $\\theta^{\\ast}$ and one with a deliberately misspecified form. If training on both sources moves the posterior mode of the exact source's parameters away from $\\theta^{\\ast}$ while emulation error stays low, the benefit of simultaneity is coming from bias correction, not from better parameter identification. The paper's own analytic example already shows Block 0 can place the misspecified source closer to $s_0$ than the exact one, so re-running that example with the true $\\theta^{\\ast}$ recorded would settle whether the failure is inherent or fixable.","tokens_in":16786,"feed_emoji":"⚙️","tokens_out":9000,"duration_ms":86177,"temperature":0.7,"pith_summary":"Computer models used in engineering often come in several versions with different fidelities, costs, and tuning parameters, and the usual practice is to calibrate each one separately against high-fidelity data. This paper asks whether calibrating all of them at once is better, and it builds a neural-network framework, iPro-NC, to find out. The framework learns a separate probability distribution for every calibration parameter of every model while using a shared latent embedding of the data sources, so it works when models have different numbers of parameters and different outputs. The paper's answer is mixed: simultaneous calibration improves emulation of the high-fidelity source in both test problems, including an engineering case, but in higher-dimensional problems the estimated parameters become non-identifiable and can sit at the edges of their sampling ranges. The authors conclude that the approach is worth using when accurate emulation is the goal, and that domain knowledge must be added when physical parameters matter.","feed_headline":"Calibrating many models at once improves prediction—with a catch","feed_subtitle":"Simultaneous calibration lifts emulation accuracy, but harder problems lose parameter identifiability.","key_machinery":"The load-bearing object is iPro-NC, a multi-block feed-forward network that converts multi-fidelity calibration into a latent-variable problem. A source-indicator variable $t_s$ is one-hot encoded and mapped by Block 0 to a source embedding $z_s$; masked calibration inputs are combined with $z_s$ in Block 1 to form $z_{\\theta}$; categorical inputs go through Block 2; and Block 3 outputs Gaussian means and standard deviations for every response. Each calibration parameter of each LF source has its own learned mean and standard deviation, sampled through the reparameterization trick and clamped to the training domain. The loss combines negative log-likelihood and interval-score terms for both emulation and calibration, plus a KL term on the calibration standard deviations, with per-source normalization so unbalanced data does not dominate. Block 0's embedding is the interpretability device: distances in $z_s$ are meant to show which sources are most similar to the HF source.","core_discovery":"On the paper's own terms, the central claim is that multi-source calibration is feasible and can be strictly better than one-at-a-time calibration, provided the method is built to handle sources with different parameter spaces and outputs. iPro-NC encodes each source into a low-dimensional latent variable, routes calibration parameters through the network, and trains with a loss that separates emulation (reproduce each source's data) from calibration (make each low-fidelity source reproduce the high-fidelity data). In the analytic example, training on all sources made the HF emulator never the worst on any output, and in the engineering problem all-source training reduced RRMSE substantially for both LF-to-HF emulation and calibrated-LF predictions. The same flexibility, however, lets the network consume misspecification: with model-form error present, the source embedding can rank sources incorrectly, and in the high-dimensional hardening-law calibration most parameters are estimated at the extremes of their ranges. So the paper establishes both halves of the tradeoff: simultaneous calibration helps prediction, and it worsens parameter identifiability as input and parameter dimensionality grow.","pith_inferences":["A practical design rule follows that the paper leaves implicit: include extra LF sources when the deliverable is a surrogate or prediction, and restrict to a minimal, well-posed set of models when the deliverable is the parameter posterior itself.","The Block 0 embedding could be repurposed as a model-form-error diagnostic, but only if identifiability is enforced; otherwise the paper's analytic result shows the embedding can rank a misspecified source closer to the truth than the correct source.","Adding a physics-based penalty that encourages the source embedding to respect known fidelity ordering, or fixing a subset of physically meaningful parameters as in the two-step engineering calibration, is a direct testable extension that may recover identifiability in the high-dimensional regime.","The per-source normalized negative log-likelihood and the emulation/calibration split in the loss are general ideas that could transfer to other multi-fidelity surrogates beyond this architecture."],"forward_implications":["When all available sources are used, iPro-NC's HF emulator is never the worst on any output in the analytic example, and all-source training cuts RRMSE in the engineering example by roughly an order of magnitude for calibrated LF predictions.","The framework removes the usual requirement that all models share the same calibration parameters: sources with different parameter counts and different responses are handled by masking and per-source distributions.","For low-dimensional problems, posterior distributions for calibration parameters cover the MSE-optimal values when the LF model has no model-form error, so joint posterior inference is meaningful there.","In higher-dimensional engineering problems the estimated parameters frequently land at the bounds of their sampling ranges, so parameter values from simultaneous calibration should not be read as physical quantities without added constraints.","The authors state directly that using iPro-NC is justified when maximizing emulation accuracy is the only goal, since training cost grows with the number of sources but calibration reliability does not."],"supporting_citations":[{"why":"Defines the standard calibration-plus-discrepancy formulation that this paper generalizes from one low-fidelity model to many.","marker":"[15]"},{"why":"Introduces latent-map multi-source fusion that iPro-NC replaces with neural embeddings and per-source calibration distributions.","marker":"[11]"},{"why":"Supplies the multi-source probabilistic neural architecture and proper-scoring losses that iPro-NC adapts to the calibration setting.","marker":"[37]"},{"why":"Provides the reparameterization trick that lets gradients flow through sampled calibration parameters during training.","marker":"[40]"},{"why":"Demonstrates that multiple responses improve identifiability in model calibration, which motivates the multi-response design used here.","marker":"[22]"},{"why":"Supplies the hardening law used to build the two low-fidelity sources in the engineering case study.","marker":"[42]"}],"fun_headline_variants":["Simultaneous model calibration helps predictions, but hurts identifiability","Multi-model calibration can boost emulation, but at cost of identifiability","Better predictions from multi-model calibration, but identifiability suffers","Simultaneous calibration: prediction gains vs. parameter identifiability loss","Calibrate multiple models at once? Trade accuracy for identifiability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The network's latent representation can capture the true relationship between every low-fidelity source and the high-fidelity source, and the loss can tell apart errors that should be absorbed by calibration parameters from errors that should be absorbed by bias correction.","fun_headline_variants_meta":{"raw":{"variants":["Simultaneous model calibration helps predictions, but hurts identifiability","Multi-model calibration can boost emulation, but at cost of identifiability","Better predictions from multi-model calibration, but identifiability suffers","Simultaneous calibration: prediction gains vs. parameter identifiability loss","Calibrate multiple models at once? Trade accuracy for identifiability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001017,"raw_usage":{"total_tokens":4313,"prompt_tokens":985,"completion_tokens":3328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":3234}},"tokens_in":601,"tokens_out":3328,"duration_ms":22383,"temperature":1.0,"reasoning_tokens":3234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:22:41.241077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic problem with a known true parameter vector $\\theta^{\\ast}$ and two LF sources, one that contains the HF model exactly at $\\theta^{\\ast}$ and one with a deliberately misspecified form. If training on both sources moves the posterior mode of the exact source's parameters away from $\\theta^{\\ast}$ while emulation error stays low, the benefit of simultaneity is coming from bias correction, not from better parameter identification. The paper's own analytic example already shows Block 0 can place the misspecified source closer to $s_0$ than the exact one, so re-running that example with the true $\\theta^{\\ast}$ recorded would settle whether the failure is inherent or fixable.","supporting_citations":[{"cited_title":"Data fusion with latent map gaussian processes","cited_arxiv_id":null,"evidence_quote":"Introduces latent-map multi-source fusion that iPro-NC replaces with neural embeddings and per-source calibration distributions."},{"cited_title":"Probabilistic neural data fusion for learning from an arbitrary number of multi-fidelity data sets","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-source probabilistic neural architecture and proper-scoring losses that iPro-NC adapts to the calibration setting."},{"cited_title":"Improving identifiability in model calibration using multiple responses, 2012","cited_arxiv_id":null,"evidence_quote":"Demonstrates that multiple responses improve identifiability in model calibration, which motivates the multi-response design used here."},{"cited_title":"Determination of hardness-strength and -flow behavior relationships in bulged aluminum alloys and verification by FE analysis on Rockwell hardness test","cited_arxiv_id":null,"evidence_quote":"Supplies the hardening law used to build the two low-fidelity sources in the engineering case study."}],"review_version":1}