{"id":"cc2cfa10-afae-4ac4-be89-cc93311fffe6","arxiv_id":"2411.17826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BAMS combines Gaussian-process-based adaptive sampling with low- and high-fidelity simulators to discover rare AV failures and estimate their rate more efficiently than Monte Carlo and importance-sampling baselines.","lead":"This paper introduces BAMS, an adaptive Bayesian sampling method that selects which logged driving scenarios to simulate in order to discover failure cases and estimate the rate of adverse events. On Waymo's WOSAC dataset, it reports finding roughly ten times more issues than Monte Carlo and difficulty-score baselines while producing rate estimates with much lower variance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline '15x narrower variance vs MC' rests on an MC baseline whose reported 100*RV=13.5 is ~3x the theoretical value for the stated K=5pγN samples; if that baseline is miscalibrated, the ratio drops to ~5x.","rationale":"The paper's central claim is an empirical ratio claim, so a mismatch in one baseline's variance is more load-bearing than the multifidelity independence prior identified by the reader. The additive independent-GP assumption (Section 3, Eq. 5) could affect efficiency, but it is a structural modeling choice with plausible partial mitigation through posterior learning, and the synthetic experiments provide some support. The MC variance inconsistency, by contrast, is checkable arithmetic: the reported 13.5 conflicts with the stated sampling budget by a factor of ~3. If the MC RV should be ~4.4, the abstract's '15 times' becomes ~5 times, and the quantitative headline is no longer accurate as stated. The same issue may contaminate the DS comparison if its RV of 5.40 is similarly miscalibrated. I still credit the paper's independent support: the IS estimator is unbiased by construction, recall is measured on ground-truth high-fidelity labels, and the synthetic experiment (Appendix D) shows the acquisition behavior qualitatively as claimed. But the numerical ratios in the abstract and Table 2 need external verification. The reader's CONDITIONAL verdict is therefore appropriate; my concern adds a concrete, prioritized verification step rather than changing the verdict. No ad hominem is intended; this is a baseline-calibration issue that can be settled by rerunning a simple computation.","tokens_in":18870,"tokens_out":8538,"duration_ms":76039,"concrete_test":"Recompute Table 2's MC row from first principles using the stated protocol: draw K=⌈5pγN⌉=2246 run segments uniformly at random from the N=44,911 segments, form the MC estimate of pγ, repeat 200 times, and compute 100·RV = Var(p̂)/pγ². Also compare against the closed-form (1−pγ)/(pγK)≈4.4. If the experimental value is ≈13.5, identify what K or pγ was actually used (e.g., is K=5√(pγN) or is pγ being estimated differently?); if it is ≈4.4, then Table 2's MC column and the abstract's 15x variance-reduction claim must be re-benchmarked against the corrected MC baseline, and the same check should be run for the DS baseline's 5.40 value to verify the 6x claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 2 (Section 4.2), MC reports 100·RV = 13.5 ± 2.86 for its rate estimate, and the abstract uses this to claim BAMS has 'variances 15 ... times narrower than MC.' But for a Bernoulli failure event with pγ=0.01 and K=5pγN=5×449.11≈2246 independent high-fidelity samples, the MC relative variance is (1−pγ)/(pγK) ≈ 0.044, i.e. 100·RV ≈ 4.4. The reported value of 13.5 is what one would get with K≈733 samples or p≈0.03, neither of which matches the stated protocol. Because RV is a ratio, an inflated MC denominator translates directly into an inflated '15x' claim; the corrected ratio would be roughly 5x, although BAMS would still beat MC. This is an internal inconsistency in the central empirical quantity, not a disagreement with community consensus. It should be resolved before the abstract's quantitative claims are accepted; the unbiasedness of the IS estimator and the qualitative recall ordering do not repair an incorrect baseline variance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BAMS (Bayesian adaptive multifidelity sampling) for two coupled tasks in autonomous-vehicle safety evaluation: estimating the rate pγ of adverse events under an empirical distribution over logged run segments, and discovering novel, likely failure scenarios. The method fits a GP to a performance metric, sequentially selects evaluation points by minimizing a cost-normalized upper bound on the expected point variance, exploits a cheap low-fidelity simulator, and uses the resulting GP scores to build an importance-sampling estimator that is unbiased by construction. Experiments on the WOSAC dataset and a synthetic problem compare BAMS with Monte Carlo, importance-sampling, cross-entropy, and ablated baselines, reporting higher recall at fixed retention and lower relative variance. The abstract claims that BAMS 'discovers 10 times as many issues' and produces rate estimates with variances '15 and 6 times narrower' than MC and IS baselines.","tokens_in":19090,"tokens_out":8537,"duration_ms":77488,"significance":"The problem is practically important: efficient failure discovery and rate estimation are both needed in AV development loops, and the paper proposes a plausible black-box combination of GP-based multifidelity active learning with importance sampling. The unbiasedness of the IS rate estimator is a genuine methodological strength, and the retention-recall evaluation uses full ground-truth high-fidelity labels, which makes the qualitative ranking of methods credible. If the quantitative claims survive correction, the method would be a useful addition to the AV safety-evaluation toolbox. However, the headline variance-reduction numbers are currently not reliable because of an apparent miscalibration of the MC baseline and because several hyperparameters were selected using the same metrics and data reported as results.","major_comments":[{"comment":"The reported MC relative variance is internally inconsistent with the stated protocol. With pγ=0.01, N=44,911, and K=5pγN≈2246 high-fidelity samples, the theoretical MC relative variance is (1−pγ)/(pγK)≈0.044, i.e. 100·RV≈4.4, or about 4.2 with a finite-population correction. Table 2 reports 100·RV=13.5±2.86 for MC, which is roughly three times larger and corresponds to K≈733 samples or pγ≈0.03, neither of which matches Section 4.1. Because the abstract's '15 times narrower' claim is a ratio against this MC number, the corrected ratio would be roughly 5 times, not 15 times. Please recompute the MC baseline, report the theoretical value alongside the empirical value, and revise the abstract and Table 2 ratios accordingly. The qualitative ordering may survive, but the quantitative claim as stated is not supported.","section":"Section 4.2, Table 2"},{"comment":"Three key hyperparameters are tuned on the same metrics and the same dataset used for the headline results: S=6 is selected by recall at 5pγN after Batch 3 (Appendix C.1, Figure 6), α=2.5 is selected by minimizing relative variance and recall at 5pγN (Appendix C.2, Figure 7), and c(y1)=5/32 is selected by recall at 5pγN after Batch 3 (Appendix C.3, Figure 8). These are exactly the quantities reported in Table 2 and Figure 3. This selection-on-evaluation can inflate the reported advantage of BAMS over baselines. The authors should either evaluate on a held-out portion of the data, use a nested validation scheme, or report the sensitivity of Table 2 to each of these choices so that the reader can judge the robustness of the claimed margins.","section":"Section 4 and Appendix C"},{"comment":"The text states that 'the theoretical results above extend directly to the multifidelity setting where we consider P as the empirical distribution over all inputs and all fidelities.' However, the quantity of interest pγ in Eq. (1) is defined with respect to the distribution over run segments X, not over augmented inputs (x,l). Proposition 1 and Corollary 1 bound the variance of an estimator that averages the indicator g over that augmented distribution. The final rate estimator in Section 4.1 uses only high-fidelity samples (l=0). It is therefore unclear why minimizing J over the augmented distribution should control the variance of the high-fidelity rate estimator. Please clarify the measure with respect to which J is defined, or provide a bound that applies to the actual estimator used in the experiments.","section":"Section 3, 'Multifidelity sampling', after Eq. (5)"},{"comment":"The multifidelity model θ(y_l)=θ0(x)+α_l(x) assumes that the discrepancy between the 5-rollout and 32-rollout simulators is an independent, zero-mean GP, uncorrelated with θ0 and with the input x. If the low-fidelity error is correlated with the high-fidelity function or with input features, the cost-normalized acquisition in Eq. (5) will systematically misprice cheap evaluations, and the claimed efficiency gains may shrink. The paper provides no diagnostic for this assumption on the WOSAC data. Please add a quantitative check (e.g., correlation of f0−f1 with f0 and with the embedding coordinates) or a robustness experiment that varies the low-fidelity noise model.","section":"Section 3, 'Multifidelity sampling'"}],"minor_comments":[{"comment":"There is a typo in 'A Vs' in the abstract; it should read 'AVs'. Also, the phrase 'discovers 10 times as many issues' refers to recall at a specific retention budget (5pγN) and should be qualified as such.","section":"Abstract"},{"comment":"The Matérn kernel formula is written as k(x,x')∝tνKν(√(2ν)t), which omits the normalization 2^{1−ν}/Γ(ν). In addition, ν is described as a 'lengthscale parameter' in the text, but ν is the smoothness parameter; the lengthscales appear in the distance t via Σ. Please correct the terminology.","section":"Section 2, Gaussian processes"},{"comment":"The input line of Algorithm 2 has malformed set notation: 'data points {(zi}N i=1' should be '{(zi)}N i=1'. The same line also uses variable z without defining the lengthscale-rescaling step before the algorithm listing.","section":"Appendix A.4, Algorithm 2"},{"comment":"The text says 'yielding an overall speedup as shown in Section A.4' but there is no Section A.4 with speedup results; the intended reference is to the complexity analysis earlier in the same appendix or to Appendix C.1. Please correct the cross-reference.","section":"Appendix A.4"},{"comment":"Section 4.1 says the IS evaluation uses 200 repeated trials, while Table 2 reports standard errors over 10 seeds. Please clarify how the 200 trials and 10 seeds are combined to produce the reported means and standard errors.","section":"Section 4.1 and Table 2"},{"comment":"The retention-recall curves do not show any measure of variability, although the experimental section mentions multiple seeds. Adding error bars or shaded bands would help the reader assess whether the reported differences between methods are significant.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practically important problem and the core idea is reasonable, but the quantitative claims in the abstract need to be corrected after addressing the MC baseline miscalibration. The hyperparameter-tuning point is also important: the authors should present some form of validation that is not directly optimizing the same headline numbers. The multifidelity P-measure issue should be clarified before publication, as it affects the theoretical grounding of Eq. (5). I would not recommend rejection, because these issues appear fixable within the scope of the manuscript, but the revised version should be re-reviewed for the corrected numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BAMS does what it says, combining GP-based stepwise uncertainty reduction with cost-normalized multifidelity sampling and applying it to logged AV run segments. The engineering is sensible, and the retention-recall curves give real evidence that the acquisition works. But the abstract's \"15x narrower variance than MC\" is built on an MC baseline that is miscalibrated by about a factor of three. The correct ratio is closer to 5x. That is still a good result, but the headline number is wrong.\n\nWhat's new: the cost-normalized multifidelity acquisition (Eq. 5) is a genuine extension of the single-fidelity SUR idea, and the rate-informed discovery framing is a useful way to tie falsification-style exploration to rate estimation. The unbiased IS estimator and ground-truth retention-recall evaluation are good choices: the recall ordering does not depend on the problematic variance numbers. The WOSAC experiments are realistic, and the appendix comparison to multilevel splitting is thoughtful.\n\nSoft spots, in order of severity:\n\n1. The MC baseline variance is internally inconsistent. For a Bernoulli event with pγ=0.01 and K≈2246, 100·RV should be about 4.4; Table 2 reports 13.5 ± 2.86. That indicates a different sample size, a different definition of RV, or a bug. The abstract's 15x claim is just the ratio to that inflated baseline. Corrected, the ratio is about 5x. The qualitative conclusion survives, but the quantitative claim is overstated.\n\n2. Three hyperparameters (α, S, c(y1)) are selected on the same dataset by optimizing the same recall/RV metrics reported in the main table. The appendix shows the tuning honestly, but the headline numbers are in-sample. An out-of-sample split or a sensitivity analysis would strengthen the case.\n\n3. No code or data artifacts are released. For an industrial paper that is not fatal, but it limits reproducibility and makes it hard to verify the IS estimator behavior.\n\n4. The additive-noise independence assumption for the low-fidelity GP is standard but untested; the retention-recall results give indirect support, so I'd call that minor.\n\nSummary: this paper is for the AV safety evaluation community. The method is incremental, but the application is real, and the recall improvements look robust. As a referee, I would ask for the MC baseline to be rechecked or the claim softened, for a note on hyperparameter selection, and ideally code/data. I would send it to peer review rather than desk reject.","headline":"Solid, incremental method paper on AV safety evaluation; the headline variance reductions are inflated by a miscalibrated MC baseline, so the quantitative claims need correcting before the abstract can be trusted.","tokens_in":19647,"tokens_out":4455,"would_cite":false,"duration_ms":41933,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian adaptive multifidelity sampling discovers ten times as many adverse driving events as Monte Carlo and importance-sampling baselines at the same simulation budget, while producing rate estimates with variance 15 and 6 times…","keywords":["autonomous vehicle safety","rare-event simulation","Bayesian adaptive sampling","multifidelity simulation","Gaussian process","importance sampling","failure discovery","rate estimation"],"falsifier":"Build a task where the cheap simulator's error is deliberately correlated with the true failure score (for example, the 5-rollout underestimate grows as true time-to-collision shrinks), then compare BAMS against the single-fidelity BAS under the same budget; if recall and relative variance do not degrade toward or below BAS, the independence assumption is not doing the work claimed, and if they do degrade, the central efficiency claim fails.","tokens_in":72,"feed_emoji":"🚗","tokens_out":7076,"duration_ms":118737,"temperature":0.7,"pith_summary":"BAMS is a sampling rule for simulation-based safety evaluation of autonomous vehicles: instead of running expensive full simulations on randomly chosen logged driving segments, it fits a Gaussian process to the performance scores already seen and adaptively picks the next segments—and the simulator fidelity—that will most reduce uncertainty about where failures are. The paper's central claim is that this rule solves both halves of the evaluation problem at once: it discovers roughly ten times as many adverse-event segments as Monte Carlo or importance-sampling baselines, and its rate estimates have variance 15 times and 6 times narrower than those baselines. That matters because failure cases are rare and expensive to find, and a team that finds an order of magnitude more of them per simulation budget, while also getting a tighter confidence interval on the overall adverse-event rate, can iterate on planner safety much faster.","feed_headline":"Adaptive sampler finds 10x more AV failures on same budget","feed_subtitle":"BAMS mixes cheap and full simulators to surface rare failures while cutting rate-estimate variance up to 15x.","key_machinery":"The load-bearing mechanism is a Gaussian process posterior over the performance score $f(x)$ combined with a forward-looking point-variance acquisition. For any candidate next point, $p_n(x)$ is the posterior probability that $f(x)\\le\\gamma$, and $h_n(x)=p_n(x)(1-p_n(x))$ is its point variance; BAMS chooses points to minimize the expectation of this point variance over the empirical distribution of run segments, an upper bound on the variance of the rate estimator. In the multifidelity version, each fidelity $l$ is modeled as the high-fidelity GP plus an independent zero-mean GP $\\alpha_l(x)$, and the acquisition is divided by the fidelity's simulation cost $c(x,l)$, so the algorithm spends budget where expected variance reduction per unit cost is largest. Sequential selection and clustering keep the search tractable for large pools of logged segments.","core_discovery":"The paper introduces rate-informed discovery as the joint goal of returning novel, high-likelihood examples of undesirable behavior while accurately estimating the probability $p_\\gamma = P(f(X) \\le \\gamma)$. It proposes BAMS, which models the performance metric with a Gaussian process, uses an acquisition function that upper-bounds the variance of the GP-based rate estimator by an average pointwise variance, selects points sequentially in clusters to keep the search tractable, and extends the model to multiple simulator fidelities with a cost-normalized acquisition. On 44,911 real logged driving segments, with high fidelity equal to 32 planner rollouts and low fidelity equal to 5, BAMS reaches 80% recall at a retention of 5 times the expected number of failures and, in an importance-sampling evaluation, attains the best recall and the smallest relative variance among all methods. The authors' conclusion is that adaptive Bayesian multifidelity sampling is an efficient way to both estimate AV safety rates and find the diverse failure cases that drive planner improvements.","pith_inferences":["A natural extension the paper leaves implicit is to make the fidelity/cost trade-off adaptive: instead of fixing the low-fidelity simulator at 5 rollouts, the acquisition could choose among several rollout counts each round, potentially improving the measured gains further.","The independence assumption deserves a direct diagnostic: fit the GP to high-fidelity data, then regress low-fidelity residuals on the high-fidelity prediction; a non-zero slope would predict that BAMS's advantage shrinks on out-of-distribution or dense-traffic segments.","The same machinery transfers to any black-box simulator hierarchy with an empirical input distribution, such as robotics or drone safety testing, where discrete logged scenarios are evaluated at multiple fidelities; the AV framing is not essential to the algorithm.","Closing the loop is testable: feed BAMS-discovered failures into a planner training step and measure whether end-to-end development cost drops; the current paper stops at discovery and rate estimation."],"forward_implications":["At a fixed simulation budget, an AV team can expect to surface an order of magnitude more novel failure segments than MC or IS sampling, giving planner developers more actionable inputs per dollar.","The same trained GP can be reused for several threat thresholds $\\gamma$, so multiple safety-rate estimates can be updated without fresh simulation campaigns.","Adding a cheap 5-rollout fidelity improves both discovery and rate precision compared with using the 32-rollout simulator alone, because the cost-normalized acquisition buys more information.","Because the final rate estimate is constructed by importance sampling, it remains unbiased even if the GP model is misspecified, so the variance gains are not bought by trading away correctness.","The method's focus shifts from broad exploration in the first batch to tight sampling around failure-region boundaries by the third batch, avoiding the collapse of cross-entropy sampling to one failure cluster."],"supporting_citations":[{"why":"Supplies the Gaussian process posterior mean, variance, and kernel machinery on which the acquisition function is built.","marker":"[42]"},{"why":"Supplies the fast stepwise uncertainty-reduction formula that BAMS's point-variance acquisition restates and exploits for efficient computation.","marker":"[6]"},{"why":"Provides the contrastive embedding of run segments and the intrinsic-difficulty model used by the DS and DS-GP baselines.","marker":"[5]"},{"why":"Supplies the 44,911 logged run segments and the 32-rollout simulation outputs used in the AV experiments.","marker":"[32]"},{"why":"Introduces the additive-noise modeling of fidelity dependence that BAMS's multifidelity GP adopts.","marker":"[31]"},{"why":"Extends additive-noise multifidelity modeling and informs the cost-normalized acquisition used to trade off fidelity against simulation cost.","marker":"[38]"},{"why":"Defines the cross-entropy baseline whose collapse to a single failure region BAMS is shown to avoid.","marker":"[43]"}],"fun_headline_variants":["Bayesian sampler finds 10x more AV failures, tighter rates","Multifidelity sampling boosts AV failure discovery 10x","Adaptive sampling reveals 10x more driving risks","Rate-aware sampling outperforms MC/IS in AV testing","BAMS: 10x more failures, 15x narrower rate estimates"],"cache_read_input_tokens":21760,"weakest_assumption_plain":"BAMS assumes the cheap simulator is the expensive simulator plus zero-mean independent noise; if the cheap simulator's errors correlate with the dangerous cases or with the inputs, the cost-normalized acquisition misprices cheap evaluations and the reported gains shrink.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian sampler finds 10x more AV failures, tighter rates","Multifidelity sampling boosts AV failure discovery 10x","Adaptive sampling reveals 10x more driving risks","Rate-aware sampling outperforms MC/IS in AV testing","BAMS: 10x more failures, 15x narrower rate estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3233,"prompt_tokens":865,"completion_tokens":2368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2282}},"tokens_in":481,"tokens_out":2368,"duration_ms":14835,"temperature":1.0,"reasoning_tokens":2282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:49:00.973517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a task where the cheap simulator's error is deliberately correlated with the true failure score (for example, the 5-rollout underestimate grows as true time-to-collision shrinks), then compare BAMS against the single-fidelity BAS under the same budget; if recall and relative variance do not degrade toward or below BAS, the independence assumption is not doing the work claimed, and if they do degrade, the central efficiency claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the cross-entropy baseline whose collapse to a single failure region BAMS is shown to avoid."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian process posterior mean, variance, and kernel machinery on which the acquisition function is built."},{"cited_title":"Chevalier, J","cited_arxiv_id":null,"evidence_quote":"Supplies the fast stepwise uncertainty-reduction formula that BAMS's point-variance acquisition restates and exploits for efficient computation."},{"cited_title":"Bronstein, S","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive embedding of run segments and the intrinsic-difficulty model used by the DS and DS-GP baselines."},{"cited_title":"Montali, J","cited_arxiv_id":null,"evidence_quote":"Supplies the 44,911 logged run segments and the 32-rollout simulation outputs used in the AV experiments."},{"cited_title":"Marco, F","cited_arxiv_id":null,"evidence_quote":"Introduces the additive-noise modeling of fidelity dependence that BAMS's multifidelity GP adopts."},{"cited_title":"Poloczek, J","cited_arxiv_id":null,"evidence_quote":"Extends additive-noise multifidelity modeling and informs the cost-normalized acquisition used to trade off fidelity against simulation cost."}],"review_version":1}