{"id":"3095e60a-06f7-4777-93ea-80959738ce01","arxiv_id":"2412.07521","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A custom validation metric is built by regressing expert face-validation ratings on standard time-series error features, with prediction intervals reported for the resulting score.","lead":"This paper trains a regression-based score, calibrated on expert judgments, to decide whether a vehicle simulation is accurate enough relative to measurements. It aims to replace subjective expert eyeballing with a repeatable metric that reports its own uncertainty.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prediction intervals in Eq. (17) are computed after feature selection on the same data; the stated confidence on predictions is not statistically valid because post-selection uncertainty is ignored.","rationale":"The reader's verdict identifies thin evaluation and the linear-separability assumption as the weakest point. I agree that the real-use-case evidence is too sparse. However, the more decisive issue is internal to the statistics: even with the synthetic dataset, the prediction intervals that constitute the 'confidence' part of the central claim are not valid under the paper's own feature-selection workflow. This makes the central claim false as stated, independent of how much data is collected. It is a concrete, fixable problem, and a coverage test can settle it directly. The paper's honest discussion of the simple vs. full interval shows awareness of one form of overconfidence, but not of post-selection overconfidence. Therefore the verdict should remain CONDITIONAL: the method is promising and clearly presented, but the confidence claim needs to be either re-derived with selection taken into account or weakened to a conditional statement about the fixed selected model.","tokens_in":13623,"tokens_out":5117,"duration_ms":52751,"concrete_test":"Use the synthetic dataset from Sec. III-A (225 curve pairs, 10 expert ratings each). Repeat 200 times: randomly split into 80% train / 20% test, apply the paper's exact feature-selection pipeline (drop features with pairwise correlation >0.9, then OLS) on the training set, compute nominal 95% prediction intervals from Eq. (17) for the test points, and record whether each test rating falls in its interval. Report the empirical coverage rate across all repetitions. If the coverage is materially below 95% (e.g., <90%), the confidence claim is not statistically valid and a post-selection correction or validation-set feature selection is required.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim has two parts: predicting sufficiency and giving a confidence on that prediction. The confidence is implemented as the prediction interval in Eq. (17), which is the standard OLS prediction interval for a fixed, pre-specified linear model. In Sec. II-B the authors combine this with feature selection: in the synthetic study they drop features with correlation above 0.9 before fitting, and the LASSO variant performs selection and estimation in one step. Once the model matrix is chosen using the same data that then determines the interval, the assumptions behind Eq. (17) are violated: the selected features are random, LASSO estimates are biased, and the selection procedure is ignored in the interval calculation. The interval therefore does not have the claimed 1-alpha coverage for a new expert rating, and the 'confidence on this prediction' is not a calibrated confidence. The paper does not mention post-selection inference, data splitting, or any correction for selection. Even in the rack example, the wider 'full' interval in Fig. 16b is still conditional on the selected features and can understate uncertainty. This is not a data-size issue; it would persist with more labels unless the inference procedure is changed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for constructing a use-case-specific validation metric by collecting expert ratings of whether simulation and measurement time series agree, computing a set of 16 standard time-series comparison metrics for each pair, and fitting a linear regression from the metrics to the expert ratings. The authors argue that the resulting regression model, called the custom metric, can predict the sufficiency of a new simulation result and provide a confidence interval for that prediction. The method is demonstrated on a synthetic manufactured-universe example with labels generated from a known parametric formula, a literature example involving five radial-velocity curve pairs rated by eleven experts, and a steering-system rack-position example with five curve pairs rated by fourteen experts.","tokens_in":13817,"tokens_out":6572,"duration_ms":62261,"significance":"If the statistical concerns were resolved, the proposed approach would be practically valuable because it offers a systematic way to translate subjective expert judgment into an objective, reproducible validation score with a stated uncertainty, which is often missing in engineering validation metrics. The paper is transparent about the small size of the real datasets and includes a synthetic experiment that allows controlled probing of hyperparameter effects. However, the central claim of predicting sufficiency with a reliable confidence is not currently supported by the evidence, because the prediction intervals are computed in a way that ignores feature selection and the clustering of expert ratings, and the real use cases contain only five distinct curves per domain.","major_comments":[{"comment":"The prediction interval in Eq. (17) is a standard OLS interval for a fixed, pre-specified linear model, but the model is selected on the same data via the correlation threshold in Sec. III-A and the retained-feature choices in Secs. III-B and III-C. After feature selection, the interval does not have the nominal 1-alpha coverage because the selection procedure is ignored; the paper neither mentions post-selection inference nor uses sample splitting. This directly affects the claimed 'confidence on this prediction' and is therefore a load-bearing issue for the central claim.","section":"II-B, Eq. (17)"},{"comment":"In both real applications, each time-series pair is rated by multiple experts, all of whom see the same features for that pair. The regression treats these ratings as independent observations, but the errors for a given curve are correlated because the curve features are shared. The effective sample size is therefore the number of distinct curves (five), not the number of ratings (e.g., 55 or 70). The prediction interval formula in Eq. (17) assumes independent errors, so the intervals shown in Figs. 13 and 16 understate the uncertainty. The paper should either model the clustering (e.g., with a mixed-effects model) or explicitly treat the five curves as the sample size and adjust the degrees of freedom accordingly.","section":"III-B and III-C"},{"comment":"With only five distinct curves in each real use case, a single train/test split with one held-out curve provides almost no evidence that the custom metric generalizes to new simulation results. The paper acknowledges that 'only five labeled data points were available' and that this is 'not sufficient for usage in a real validation strategy,' yet still concludes that 'a fair prediction is possible.' That conclusion is not supported by the empirical evidence; at best, the real examples are illustrative, and the synthetic experiment cannot validate the representation of expert judgment because its labels are generated by the linear formula in Eq. (18), which is not a realistic model of expert cognition.","section":"III-B and III-C"},{"comment":"The method section describes LASSO as an alternative for combined feature selection and regression, but the applications only use correlation-threshold selection followed by ordinary least squares. No LASSO results are reported anywhere in the paper. If LASSO is meant to be part of the proposed approach, its behavior and inference (which would require additional adjustments for selection and shrinkage bias) should be demonstrated; otherwise, the LASSO path should be removed from the method description or clearly marked as future work.","section":"II-B and III-A"}],"minor_comments":[{"comment":"There are numerous typos and spelling errors, including 'Featrue selection' in Fig. 4, 'can than predict' in the Abstract, 'actatuator' in Sec. III-C, 'valdiation' in Ref. [43], 'numerber' and 'Howerver' in Sec. III-A, 'disucssed' in Sec. I-D, and 'rater just a metric' in Sec. III-B.","section":"Throughout"},{"comment":"In Eq. (16), the notation is unclear: it appears to give an interval for a single weight rather than for a new observation, and the use of sqrt(N) is not defined. The text should distinguish between intervals for the mean response and intervals for a new observation.","section":"II-B"},{"comment":"The sentence 'under the assumption of normally distributed weights' should read 'under the assumption of normally distributed errors'; the distributional assumption concerns the noise term, not the regression weights.","section":"II-B"},{"comment":"The phrase 'the EEARTH score of ISO18751' is incorrect: EEARTH is due to Sarin et al. [19] and ISO18571 is a different standard metric; please correct the terminology.","section":"III-B"},{"comment":"The quotation attributed to reference [52] appears to originate from Schwer [29] rather than from Oberkampf and Trucano; please verify the source of the quoted text.","section":"III-B"},{"comment":"In Fig. 7, the caption mentions 95% confidence intervals but does not explain the difference between the red and yellow intervals; a short note in the caption would help readers understand the two formulas.","section":"III-A"}],"recommendation":"major_revision","confidential_remarks":"This is an application-oriented contribution from an industrial research lab; the novelty is modest but the problem is relevant to the validation community. The main obstacle is the statistical validity of the confidence statements, which is a core part of the paper's claim. If the authors can address the post-selection and clustering issues, perhaps by demonstrating coverage on the synthetic data or by using more appropriate inference methods, the paper could become suitable for publication. I would not recommend rejection, as the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you care about simulation validation. The paper proposes a practical way to build a use-case-specific validation metric: collect expert ratings on face-validation plots, compute a library of standard time-series metrics (MAE, Sprague–Geers, ISO18571, EEARTH), then regress the ratings on those features, with LASSO or correlation-based feature selection and a prediction interval on the output. Extending Sarin et al.'s EEARTH idea with systematic feature selection and explicit prediction intervals is a genuine, modest novelty. The writing is clear, and the authors deserve credit for being upfront about how little real data they have and about the need for more labels.\n\nWhat the paper does well: the synthetic experiment in Sec. III-A uses a manufactured universe, which is the right way to test a calibration procedure in isolation. The train/test split is honest there, and the investigation of sensitivity to noise, expert scatter, and feature count is useful. The rack-position example in Sec. III-C nicely demonstrates that a naive prediction interval can overstate confidence, which is a good cautionary story.\n\nSoft spots, in proportion. First, the stress-test note is correct: the prediction interval in Eq. (17) is a standard OLS interval for a fixed model, but the paper computes it after doing feature selection on the same data. That makes the stated confidence statistically invalid. The LASSO variant is worse: selection and estimation are entangled, and the interval ignores the selection uncertainty. This is not a data-size issue; it would persist with more labels. Data splitting or a bootstrap would be a simple fix. Second, the real-world evidence is thin: five labeled curves in each of the two use cases, with a single held-out sample each. The authors acknowledge this, so it is a limitation, not a deception. Third, the linear-separability assumption in Eq. (15) is untested on real expert judgments; the synthetic labels are generated by a linear formula, so they cannot validate it. No code or data is provided, which would have helped a reviewer check the details.\n\nOverall, the central idea is sound and the limitations are honestly stated. The post-selection issue is a real technical flaw, but it is fixable and the paper is transparent about most of its other weaknesses. For a researcher in model validation, this is worth reading and citing as a precursor to a more rigorous treatment. A serious editor should send it to review rather than desk-reject; the reviewers should push for a corrected uncertainty calculation and, ideally, more data or an external validation.\n\nMy recommendation: engage with it, ask for the inference fix, and keep an eye on the follow-up work.","headline":"A careful, honest methods paper on learning validation metrics from expert ratings, with a real gap in how the reported prediction intervals account for feature selection.","tokens_in":14374,"tokens_out":1507,"would_cite":true,"duration_ms":17113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A regression model trained on expert ratings predicts when a simulation is good enough.","keywords":["model validation","custom metric","expert knowledge","face validation","linear regression","prediction interval","time series metrics","sufficiency rating"],"falsifier":"Collect expert ratings on curve pairs engineered so that all base metric features (MAE, Sprague–Geers components, ISO18571, corridor score, EEARTH) are identical while the pairs differ in ways experts can see in the plots, such as which physical regime the deviation occurs in or whether the mismatch is a phase lag versus a spike. If experts systematically assign different sufficiency ratings to these feature-identical pairs, then no linear combination of the listed features can reproduce expert judgment, and the paper's central claim fails.","tokens_in":13383,"feed_emoji":"🎯","tokens_out":6240,"duration_ms":53602,"temperature":0.7,"pith_summary":"Standard validation metrics, such as RMSE, Sprague–Geers, and the ISO18571 rating, compare two curves but cannot answer the engineer's actual question: is this simulation model accurate enough for its intended use? The paper proposes to learn a use-case-specific answer from experts by collecting their continuous ratings of simulation-versus-measurement curve pairs, extracting a battery of standard time-series metrics from those same pairs, and fitting a linear regression that maps the metrics to the ratings. The resulting 'custom metric' predicts the sufficiency of a new simulation result and, through a prediction interval, quantifies how much trust that prediction deserves. Demonstrated on synthetic data, a literature dataset, and a steering-system rack-position use case, the metric reproduces expert opinions where enough labels exist and visibly flags when the available data are too sparse.","feed_headline":"Trained on expert eyes, a metric judges if a simulation is good enough","feed_subtitle":"Fitting common curve metrics to expert ratings yields a use-case-specific sufficiency score with an uncertainty band.","key_machinery":"The load-bearing object is the custom metric itself: a linear regression model $R = \\sum_i w_i f_i(x,y) + \\epsilon$ in which the features are established time-series validation metrics computed on the measurement-simulation pair. It is built on a formalized face-validation step: experts rate curve pairs on a continuous 0-to-1 slider anchored to the ISO18571 grade table (excellent, good, fair, poor), so the labels carry a shared meaning. The regression machinery then selects a sparse set of base metrics (correlation threshold plus OLS or LASSO) and returns both a point prediction of the expert rating and a prediction interval from the residual variance, which is the component the paper argues is missing from existing validation metrics.","core_discovery":"The central claim is that an expert judgment of 'sufficient model quality' can be approximated by a weighted linear combination of ordinary curve-comparison metrics, with the weights learned from labeled examples. Concretely, the paper models the rating as $R = \\sum_{i=1}^N w_i f_i(x,y) + \\epsilon$, where the features $f_i$ are standard quantities like mean absolute error, Sprague–Geers magnitude and phase errors, the ISO18571 combined score, its corridor score, and the EEARTH score. The weights come from ordinary least squares or LASSO after dropping strongly correlated features, and the prediction interval is derived from the standard linear-regression variance formula. The authors show on synthetic PT2 step-response data that the fitted metric tracks artificially generated expert labels well, remains stable as measurement noise increases when experts ignore that noise, and degrades predictably as expert disagreement grows; on the real datasets it reproduces expert rankings where data permit and produces wide prediction intervals that expose when labeling is too sparse to trust the fit.","pith_inferences":["The linear-separability premise is testable outside the paper: if experts systematically rate two pairs with identical values across all base metrics differently because of context they notice in the curves, no linear combination of these features can reproduce their judgments, and the method would need features that encode physical meaning or excitation type.","The prediction interval could be used as an active-learning acquisition function: label only the pairs where the interval is widest, which is a concrete extension the authors do not explore.","The same recipe—expert ratings regressed onto standard discrepancy features—could transfer to other subjective quality judgments beyond vehicle models, such as audio, image, or human-performance model validation, whenever a shared rating scale can be anchored.","In the synthetic study the expert labels are generated from a linear formula in the parameters, so the strong agreement there primarily demonstrates that a linear model can recover a linear labeling rule; it says little about how experts form judgments in real settings."],"forward_implications":["An engineer can score a new simulation-versus-measurement pair with a single number that answers 'is this good enough?' rather than a bare discrepancy value.","The attached prediction interval tells the user when the metric itself is too uncertain to trust, preventing overconfident validation on small labeled datasets.","The metric is transparent: its weights reveal which standard error components (magnitude, phase, slope, corridor, etc.) drive expert opinion in a given use case.","Because the metric inherits the expert panel's context, the same data labeled by a different panel or with different instructions yields a different custom metric targeted to that application.","Within the paper's evidence, even five labeled curve pairs can give a fair prediction, though the interval makes clear that more labels are needed for reliable use."],"supporting_citations":[{"why":"supplies the grade-and-rating scale used to anchor expert labels and the ISO18571 combined metric used as a feature","marker":"[31]"},{"why":"introduces the EEARTH weighted combined metric and the precedent of fitting metric weights to expert face-validation ratings","marker":"[19]"},{"why":"provides the real-world literature dataset of expert ratings of radial-velocity responses used in the demonstration","marker":"[29]"},{"why":"furnishes the LASSO regression method used for simultaneous feature selection and weight estimation","marker":"[45]"},{"why":"supplies the linear-regression prediction-interval formulas used to quantify the custom metric's uncertainty","marker":"[49]"},{"why":"provides the method of manufactured universes used to design the synthetic-data experiment","marker":"[50]"}],"fun_headline_variants":["Expert-trained metric tells if a simulation passes muster","Custom metric learns from expert ratings to judge simulations","Simulation sufficiency score from expert opinions","Regression on common metrics predicts expert sufficiency","Turn expert ratings into a validation metric with confidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that an expert's 'is this good enough?' judgment is fully captured by a linear combination of the listed curve-metric features; if experts rely on something not in those numbers—physical meaning of the deviation, excitation type, or intended use—the trained metric cannot reproduce their ratings no matter how many labels are collected.","fun_headline_variants_meta":{"raw":{"variants":["Expert-trained metric tells if a simulation passes muster","Custom metric learns from expert ratings to judge simulations","Simulation sufficiency score from expert opinions","Regression on common metrics predicts expert sufficiency","Turn expert ratings into a validation metric with confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2847,"prompt_tokens":992,"completion_tokens":1855,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1786}},"tokens_in":608,"tokens_out":1855,"duration_ms":14185,"temperature":1.0,"reasoning_tokens":1786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:45:24.179855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect expert ratings on curve pairs engineered so that all base metric features (MAE, Sprague–Geers components, ISO18571, corridor score, EEARTH) are identical while the pairs differ in ways experts can see in the plots, such as which physical regime the deviation occurs in or whether the mismatch is a phase lag versus a spike. If experts systematically assign different sufficiency ratings to these feature-identical pairs, then no linear combination of the listed features can reproduce expert judgment, and the paper's central claim fails.","supporting_citations":[{"cited_title":"Road vehicles - objective rating metric for non-ambiguous signals,","cited_arxiv_id":null,"evidence_quote":"supplies the grade-and-rating scale used to anchor expert labels and the ISO18571 combined metric used as a feature"},{"cited_title":"A comprehensive metric for comparing time histories in validation of simulation models with emphasiis on vehicle safety applications,","cited_arxiv_id":null,"evidence_quote":"introduces the EEARTH weighted combined metric and the precedent of fitting metric weights to expert face-validation ratings"},{"cited_title":"Validation metrics for response histories: perspectives and case studies,","cited_arxiv_id":null,"evidence_quote":"provides the real-world literature dataset of expert ratings of radial-velocity responses used in the demonstration"},{"cited_title":"Fahrmeier, C","cited_arxiv_id":null,"evidence_quote":"supplies the linear-regression prediction-interval formulas used to quantify the custom metric's uncertainty"},{"cited_title":"The method of manufactured universes for validating uncertainty quantification methods,","cited_arxiv_id":null,"evidence_quote":"provides the method of manufactured universes used to design the synthetic-data experiment"}],"review_version":1}