Pith. sign in

REVIEW 4 major objections 6 minor 60 references

Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bayesian adaptive multifidelity sampling discovers ten times as many adverse driving events as Monte Carlo and importance-sampling baselines at the same simulation budget, while producing rate estimates with variance 15 and 6 times…

desk verdict Solid, incremental method paper on AV safety evaluation; the headline variance reductions are inflated by a miscalibrated MC baseline, so the quantitative claims need correcting before the abstract can be trusted. read the letter →

arxiv 2411.17826 v1 pith:PBDEJ5H6 submitted 2024-11-26 cs.RO cs.LGstat.ML

Aman Sinha , Payam Nikdel , Supratik Paul , Shimon Whiteson This is my paper · ORCID
classification cs.ROcs.LGstat.ML
keywords autonomousvehiclesafetyrare-eventsimulationBayesianadaptivesamplingmultifidelityGaussianprocessimportancefailurediscoveryrateestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BAMS is a sampling rule for simulation-based safety evaluation of autonomous vehicles: instead of running expensive full simulations on randomly chosen logged driving segments, it fits a Gaussian process to the performance scores already seen and adaptively picks the next segments—and the simulator fidelity—that will most reduce uncertainty about where failures are. The paper's central claim is that this rule solves both halves of the evaluation problem at once: it discovers roughly ten times as many adverse-event segments as Monte Carlo or importance-sampling baselines, and its rate estimates have variance 15 times and 6 times narrower than those baselines. That matters because failure cases are rare and expensive to find, and a team that finds an order of magnitude more of them per simulation budget, while also getting a tighter confidence interval on the overall adverse-event rate, can iterate on planner safety much faster.

What carries the argument

The load-bearing mechanism is a Gaussian process posterior over the performance score $f(x)$ combined with a forward-looking point-variance acquisition. For any candidate next point, $p_n(x)$ is the posterior probability that $f(x)\le\gamma$, and $h_n(x)=p_n(x)(1-p_n(x))$ is its point variance; BAMS chooses points to minimize the expectation of this point variance over the empirical distribution of run segments, an upper bound on the variance of the rate estimator. In the multifidelity version, each fidelity $l$ is modeled as the high-fidelity GP plus an independent zero-mean GP $\alpha_l(x)$, and the acquisition is divided by the fidelity's simulation cost $c(x,l)$, so the algorithm spends budget where expected variance reduction per unit cost is largest. Sequential selection and clustering keep the search tractable for large pools of logged segments.

What would settle it

Build a task where the cheap simulator's error is deliberately correlated with the true failure score (for example, the 5-rollout underestimate grows as true time-to-collision shrinks), then compare BAMS against the single-fidelity BAS under the same budget; if recall and relative variance do not degrade toward or below BAS, the independence assumption is not doing the work claimed, and if they do degrade, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

The paper introduces rate-informed discovery as the joint goal of returning novel, high-likelihood examples of undesirable behavior while accurately estimating the probability $p_\gamma = P(f(X) \le \gamma)$. It proposes BAMS, which models the performance metric with a Gaussian process, uses an acquisition function that upper-bounds the variance of the GP-based rate estimator by an average pointwise variance, selects points sequentially in clusters to keep the search tractable, and extends the model to multiple simulator fidelities with a cost-normalized acquisition. On 44,911 real logged driving segments, with high fidelity equal to 32 planner rollouts and low fidelity equal to 5, BAMS reaches 80% recall at a retention of 5 times the expected number of failures and, in an importance-sampling evaluation, attains the best recall and the smallest relative variance among all methods. The authors' conclusion is that adaptive Bayesian multifidelity sampling is an efficient way to both estimate AV safety rates and find the diverse failure cases that drive planner improvements.

Load-bearing premise

BAMS assumes the cheap simulator is the expensive simulator plus zero-mean independent noise; if the cheap simulator's errors correlate with the dangerous cases or with the inputs, the cost-normalized acquisition misprices cheap evaluations and the reported gains shrink.

Editorial extensions

If this is right

  • At a fixed simulation budget, an AV team can expect to surface an order of magnitude more novel failure segments than MC or IS sampling, giving planner developers more actionable inputs per dollar.
  • The same trained GP can be reused for several threat thresholds $\gamma$, so multiple safety-rate estimates can be updated without fresh simulation campaigns.
  • Adding a cheap 5-rollout fidelity improves both discovery and rate precision compared with using the 32-rollout simulator alone, because the cost-normalized acquisition buys more information.
  • Because the final rate estimate is constructed by importance sampling, it remains unbiased even if the GP model is misspecified, so the variance gains are not bought by trading away correctness.
  • The method's focus shifts from broad exploration in the first batch to tight sampling around failure-region boundaries by the third batch, avoiding the collapse of cross-entropy sampling to one failure cluster.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to make the fidelity/cost trade-off adaptive: instead of fixing the low-fidelity simulator at 5 rollouts, the acquisition could choose among several rollout counts each round, potentially improving the measured gains further.
  • The independence assumption deserves a direct diagnostic: fit the GP to high-fidelity data, then regress low-fidelity residuals on the high-fidelity prediction; a non-zero slope would predict that BAMS's advantage shrinks on out-of-distribution or dense-traffic segments.
  • The same machinery transfers to any black-box simulator hierarchy with an empirical input distribution, such as robotics or drone safety testing, where discrete logged scenarios are evaluated at multiple fidelities; the AV framing is not essential to the algorithm.
  • Closing the loop is testable: feed BAMS-discovered failures into a planner training step and measure whether end-to-end development cost drops; the current paper stops at discovery and rate estimation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BAMS (Bayesian adaptive multifidelity sampling) for two coupled tasks in autonomous-vehicle safety evaluation: estimating the rate pγ of adverse events under an empirical distribution over logged run segments, and discovering novel, likely failure scenarios. The method fits a GP to a performance metric, sequentially selects evaluation points by minimizing a cost-normalized upper bound on the expected point variance, exploits a cheap low-fidelity simulator, and uses the resulting GP scores to build an importance-sampling estimator that is unbiased by construction. Experiments on the WOSAC dataset and a synthetic problem compare BAMS with Monte Carlo, importance-sampling, cross-entropy, and ablated baselines, reporting higher recall at fixed retention and lower relative variance. The abstract claims that BAMS 'discovers 10 times as many issues' and produces rate estimates with variances '15 and 6 times narrower' than MC and IS baselines.

Significance. The problem is practically important: efficient failure discovery and rate estimation are both needed in AV development loops, and the paper proposes a plausible black-box combination of GP-based multifidelity active learning with importance sampling. The unbiasedness of the IS rate estimator is a genuine methodological strength, and the retention-recall evaluation uses full ground-truth high-fidelity labels, which makes the qualitative ranking of methods credible. If the quantitative claims survive correction, the method would be a useful addition to the AV safety-evaluation toolbox. However, the headline variance-reduction numbers are currently not reliable because of an apparent miscalibration of the MC baseline and because several hyperparameters were selected using the same metrics and data reported as results.

major comments (4)
  1. [Section 4.2, Table 2] The reported MC relative variance is internally inconsistent with the stated protocol. With pγ=0.01, N=44,911, and K=5pγN≈2246 high-fidelity samples, the theoretical MC relative variance is (1−pγ)/(pγK)≈0.044, i.e. 100·RV≈4.4, or about 4.2 with a finite-population correction. Table 2 reports 100·RV=13.5±2.86 for MC, which is roughly three times larger and corresponds to K≈733 samples or pγ≈0.03, neither of which matches Section 4.1. Because the abstract's '15 times narrower' claim is a ratio against this MC number, the corrected ratio would be roughly 5 times, not 15 times. Please recompute the MC baseline, report the theoretical value alongside the empirical value, and revise the abstract and Table 2 ratios accordingly. The qualitative ordering may survive, but the quantitative claim as stated is not supported.
  2. [Section 4 and Appendix C] Three key hyperparameters are tuned on the same metrics and the same dataset used for the headline results: S=6 is selected by recall at 5pγN after Batch 3 (Appendix C.1, Figure 6), α=2.5 is selected by minimizing relative variance and recall at 5pγN (Appendix C.2, Figure 7), and c(y1)=5/32 is selected by recall at 5pγN after Batch 3 (Appendix C.3, Figure 8). These are exactly the quantities reported in Table 2 and Figure 3. This selection-on-evaluation can inflate the reported advantage of BAMS over baselines. The authors should either evaluate on a held-out portion of the data, use a nested validation scheme, or report the sensitivity of Table 2 to each of these choices so that the reader can judge the robustness of the claimed margins.
  3. [Section 3, 'Multifidelity sampling', after Eq. (5)] The text states that 'the theoretical results above extend directly to the multifidelity setting where we consider P as the empirical distribution over all inputs and all fidelities.' However, the quantity of interest pγ in Eq. (1) is defined with respect to the distribution over run segments X, not over augmented inputs (x,l). Proposition 1 and Corollary 1 bound the variance of an estimator that averages the indicator g over that augmented distribution. The final rate estimator in Section 4.1 uses only high-fidelity samples (l=0). It is therefore unclear why minimizing J over the augmented distribution should control the variance of the high-fidelity rate estimator. Please clarify the measure with respect to which J is defined, or provide a bound that applies to the actual estimator used in the experiments.
  4. [Section 3, 'Multifidelity sampling'] The multifidelity model θ(y_l)=θ0(x)+α_l(x) assumes that the discrepancy between the 5-rollout and 32-rollout simulators is an independent, zero-mean GP, uncorrelated with θ0 and with the input x. If the low-fidelity error is correlated with the high-fidelity function or with input features, the cost-normalized acquisition in Eq. (5) will systematically misprice cheap evaluations, and the claimed efficiency gains may shrink. The paper provides no diagnostic for this assumption on the WOSAC data. Please add a quantitative check (e.g., correlation of f0−f1 with f0 and with the embedding coordinates) or a robustness experiment that varies the low-fidelity noise model.
minor comments (6)
  1. [Abstract] There is a typo in 'A Vs' in the abstract; it should read 'AVs'. Also, the phrase 'discovers 10 times as many issues' refers to recall at a specific retention budget (5pγN) and should be qualified as such.
  2. [Section 2, Gaussian processes] The Matérn kernel formula is written as k(x,x')∝tνKν(√(2ν)t), which omits the normalization 2^{1−ν}/Γ(ν). In addition, ν is described as a 'lengthscale parameter' in the text, but ν is the smoothness parameter; the lengthscales appear in the distance t via Σ. Please correct the terminology.
  3. [Appendix A.4, Algorithm 2] The input line of Algorithm 2 has malformed set notation: 'data points {(zi}N i=1' should be '{(zi)}N i=1'. The same line also uses variable z without defining the lengthscale-rescaling step before the algorithm listing.
  4. [Appendix A.4] The text says 'yielding an overall speedup as shown in Section A.4' but there is no Section A.4 with speedup results; the intended reference is to the complexity analysis earlier in the same appendix or to Appendix C.1. Please correct the cross-reference.
  5. [Section 4.1 and Table 2] Section 4.1 says the IS evaluation uses 200 repeated trials, while Table 2 reports standard errors over 10 seeds. Please clarify how the 200 trials and 10 seeds are combined to produce the reported means and standard errors.
  6. [Figure 3] The retention-recall curves do not show any measure of variability, although the experimental section mentions multiple seeds. Adding error bars or shaded bands would help the reader assess whether the reported differences between methods are significant.

Circularity Check

3 steps flagged · score 6.0 of 10

The reported variance reduction and discovery ratios are partially circular because the IS exponent, cluster count, and low-fidelity cost were tuned on the same RV/recall metrics at the same 5pγN budget that the headline results report.

  1. fitted input called prediction [Section 4.1, 'Rate estimation and discovery via IS'; Appendix C.2]
    "For GP methods, we define our importance sampler as pn(x)α, with α = 2.5 chosen to minimize relative variance (see Appendix C.2)."

    The headline 'variances 15 and 6 times narrower than MC and IS' is computed from the relative variance of an importance sampler whose exponent α was selected by minimizing that same relative variance on the same experimental data. Table 2's BAMS 100(RV±SE)=0.851±0.121 is therefore the optimized value of the selection objective, not an independent prediction of the method's variance. The '15x/6x' ratios are fitted minima compared against untuned baselines, so the variance-reduction claim is forced by the tuning procedure rather than by an out-of-sample evaluation.

  2. fitted input called prediction [Section 4, 'Multifidelity setup'; Appendix C.3; Section 4.2, Figure 3]
    "This parameter was chosen based on a parameter-tuning study (see Appendix C.3) conducted over Q0.25(mT T C[1] . . . mT T C[i]) with corresponding cost i/32, evaluating the model's recall at a retention budget of 5 pγN."

    The low-fidelity cost c(y1)=5/32 was selected by maximizing recall at exactly 5pγN retention after Batch 3, and the central discovery result is then reported at the same retention level: 'BAMS achieves 80% recall at a retention of 5pγN.' The '10 times as many issues' claim is therefore an in-sample score of a hyperparameter fitted to that metric and budget, not an independent assessment of discovery performance.

1 more flagged steps
  1. fitted input called prediction [Section 4, 'Experimental procedure'; Appendix C.1; Section 4.2, Figure 3]
    "We utilize BAMS with S = 6 clusters, based on the parameter-tuning study presented in Appendix C.1."

    Appendix C.1 selects S by plotting recall at a retention budget of 5pγN after Batch 3 against the number of clusters. The main results then use S=6 and report recall at the same 5pγN retention level. Thus the cluster-count hyperparameter is tuned to the exact evaluation metric and retention point used to demonstrate BAMS's discovery advantage, making the reported recall gain partially a product of the tuning protocol rather than a fresh prediction.

full rationale

BAMS's acquisition function, Proposition 1, and the unbiasedness of the final IS rate estimator are internally derived and not circular: the GP-based variance bound and the IS construction stand on their own. However, the paper's quantitative headline claims are partially circular in evaluation. The IS exponent α=2.5 is explicitly chosen to minimize relative variance on the same data, after which the abstract reports that BAMS produces rate estimates with variances '15 and 6 times narrower than MC and IS' baselines; this is fitting the objective and then reporting the fitted value. Likewise, S=6 and the low-fidelity cost c(y1)=5/32 were selected by maximizing recall at 5pγN retention, and the discovery claim of '80% recall at a retention of 5pγN' is evaluated at that same metric and budget. Hyperparameter tuning on the evaluation metric is a legitimate model-selection step, but presenting the resulting in-sample numbers as the method's demonstrated advantage without an independent validation set is a fitted-input-called-prediction pattern. Separately, the MC baseline's reported 100·RV=13.5 is inconsistent with the stated K=5pγN and pγ=0.01, for which the theoretical value is about 4.4; this is not itself circularity but it further inflates the headline variance ratio. The central derivation of the adaptive sampling scheme remains independent, so the paper is not globally circular, but the main empirical ratios are substantially weakened by tuning-on-the-metric.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central statistical model is a GP prior on the performance metric, with an additive-noise model for the low-fidelity simulator. The method's theoretical bounds (Proposition 1, Corollary 1) are standard variance bounds from Chevalier et al. The empirical claims rely on three tuned hyperparameters (alpha, S, c(y1)) that were optimized on the same evaluation metrics used to report results, which is a data-fitting concern rather than a derivation-level circularity.

free parameters (4)
  • Importance sampling exponent alpha = 2.5
    Chosen in Appendix C.2 to minimize relative variance of the rate estimate on the same WOSAC data used to report RV; this fits the reported variance-reduction headline.
  • Number of clusters S = 6
    Tuned in Appendix C.1 to trade off recall at 5 p_gamma N against computational time on the evaluation data; used for BAMS main results.
  • Low-fidelity cost c(y1) = 5/32
    Selected in Appendix C.3 by varying number of rollouts i to maximize BAMS recall at 5 p_gamma N retention; directly affects multifidelity acquisition trade-off.
  • GP kernel lengthscales and noise = Fitted via MLL
    Maximized marginal log-likelihood on the initial random batch (Section 4, procedure A). Standard GP fitting, but still data-fitted.
assumptions (6)
  • domain assumption The performance metric f(x) is a sample path of a Gaussian process with a Matérn kernel.
    Section 2 and 3: the GP prior over f is the core statistical model; violation of GP assumptions would change uncertainty estimates and acquisition.
  • domain assumption The additive multifidelity model theta(y_l) = theta_0(x) + alpha_l(x) with independent zero-mean GPs for l>=1.
    Section 3, 'Multifidelity sampling': this model is taken from Marco et al. and Poloczek et al.; if the low-fidelity simulator's error is not an independent zero-mean GP, the cost-benefit trade-off in the acquisition is miscalibrated.
  • standard math Fubini's theorem applies for exchanging expectations in the variance bounds.
    Appendix B.2 and B.3 require interchanging E_theta_n and E_X; the paper assumes integrability without proof of finite moments.
  • domain assumption The empirical distribution over N logged run segments is the correct operational design domain distribution P.
    Section 3: P is an empirical distribution over run segments; the rate estimate is only meaningful relative to this logged distribution, not the real world.
  • ad hoc to paper Sequential greedy selection with cost normalization approximates joint batch selection well enough.
    Section 3 'Sequential selection' and 'Multifidelity sampling': joint selection over C(N,m) is intractable; greedy sequential selection is used as a heuristic, and the overbudget parameter eta and cluster budgets are heuristic.
  • domain assumption The 25th percentile of minimum TTC across 32 rollouts captures safety-relevant performance.
    Section 4, Performance metric: f(x) = Q0.25(mTTC[1]..mTTC[32]); the choice of percentile and threshold gamma=0.43 defines adverse events.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling." pith.science (2026). https://pith.science/paper/PBDEJ5H6

@misc{pith2026241117826,
  author       = {Pith},
  title        = {Pith review of: Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBDEJ5H6}},
  note         = {Machine review of arXiv:2411.17826}
}
read the original abstract

Ensuring the safety of autonomous vehicles (AVs) requires both accurate estimation of their performance and efficient discovery of potential failure cases. This paper introduces Bayesian adaptive multifidelity sampling (BAMS), which leverages the power of adaptive Bayesian sampling to achieve efficient discovery while simultaneously estimating the rate of adverse events. BAMS prioritizes exploration of regions with potentially low performance, leading to the identification of novel and critical scenarios that traditional methods might miss. Using real-world AV data we demonstrate that BAMS discovers 10 times as many issues as Monte Carlo (MC) and importance sampling (IS) baselines, while at the same time generating rate estimates with variances 15 and 6 times narrower than MC and IS baselines respectively.

Figures

Figures reproduced from arXiv: 2411.17826 by the authors.

Figure 1
Figure 1. Rate-informed discovery loop – A: A diverse set of simulations are run with the latest version of the AV planner. B: Some simulations lead to adverse events which are attributable to the planner. C: The results from all simulations are used to compute an unbiased estimate of the rate of adverse events. D: The adverse events found in B are used to understand and address the failure cases. E: An improved planner is de… view at source ↗
Figure 2
Figure 2. Illustration of BAMS. a) Iterative loop of our approach, as defined in Algorithm 1 (Appendix A.1). In each iteration, we separate the inputs into clusters, solve problem (5) over each cluster, and then select the final points from the candidates in all the clusters. After performing simulations over these points, we update the GP posterior. For simplicity, we neglect illustrating multiple fidelities in this graphic.… view at source ↗
Figure 3
Figure 3. Retention-recall curves for AV experiments. The x-axis represents the retention budget scaled by pγN while the y-axis shows the recall at each corresponding retention budget. BAMS MCM-GP [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: We select five samples from Batch 3 data using a DPP sampler to maximize diversity. The ego vehicle is blue and the snapshot corresponds to the timestamp at which the minimum TTC occurs; large transparent circles indicate the trajectory history, and small circles repre…
Figure 5
Figure 5. Figure 5: AV setting 90% confidence in￾tervals for IS rate estimate with K = 5pγN samples. The ground truth is pγ = 0.01. Smaller error bars indicate a more precise estimator. Method Recall ±SE 100(RV ± SE) BAMS 0.511 ± 0.0013 0.851 ± 0.121 BAS 0.388 ± 0.0013 0.969 ± 0.037 MC 0.…
Figure 6
Figure 6. Figure 6: Exploring how the number of clusters affects both recall and computational time of BAMS’s sequential selection stage at 5pγN retention. Recall is shown in blue, time in red [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Influence of IS parameter α on relative variance of rate estimate and recall achieved using a budget of 5pγN samples. Recall is shown in blue, RV in red. Based on these findings, our clustering approach can significantly speed up the process while maintaining an accept…
Figure 8
Figure 8. Figure 8: Impact of varying the cost of the lowest fidelity level (level 1) on recall at 5pγN retention after Batch 3. The data points represent the average recall across 6 random seeds. C.3 Multifidelity cost To determine the optimal level l = 1 cost in the AV setting, we condu…
Figure 9
Figure 9. Figure 9: Retention-recall curves for synthetic experiments. The x-axis represents the retention budget scaled by pγN while the y-axis shows the recall at each corresponding retention budget. top-right areas in Batch 2, revealing potential regions of interest. Finally, in Batch …
Figure 10
Figure 10. Figure 10: Synthetic setting 90% confi￾dence intervals for IS rate estimate with K = 2pγN samples. Method Recall ±SE 100(RV ± SE) BAMS 1.00 ± 0.000 2.00 ± 0.003 BAS 0.917 ± 0.003 3.85 ± 0.366 MC 0.010 ± 0.001 710. ± 214. MC-GP 0.394 ± 0.004 39.4 ± 3.88 MCM-GP 0.537 ± 0.005 29.76…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 50 canonical work pages

  1. [1]

    Althoff and J

    M. Althoff and J. M. Dolan. Online verification of automated road vehicles using reachability analysis. IEEE Transactions on Robotics, 30(4):903–918, 2014

    work page 2014

  2. [2]

    Arief, Z

    M. Arief, Z. Huang, G. K. S. Kumar, Y. Bai, S. He, W. Ding, H. Lam, and D. Zhao. Deep probabilistic accelerated evaluation: A certifiable rare-event simulation methodology for black- box autonomy. arXiv preprint arXiv:2006.15722 , 1(4), 2020

    arXiv 2006

  3. [3]

    Balabanov and G

    V. Balabanov and G. Venter. Multi-fidelity optimization with high-fidelity analysis and low- fidelity gradients. In 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Confer- ence, page 4459, 2004

    work page 2004

  4. [4]

    E. V. Bonilla, K. Chai, and C. Williams. Multi-task gaussian process prediction. Advances in neural information processing systems , 20, 2007

    work page 2007

  5. [5]

    Bronstein, S

    E. Bronstein, S. Srinivasan, S. Paul, A. Sinha, M. O’Kelly, P. Nikdel, and S. Whiteson. Embed- ding synthetic off-policy experience for autonomous driving via zero-shot curricula. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research, pages 188–198. PMLR...

    work page

  6. [6]

    Chevalier, J

    C. Chevalier, J. Bect, D. Ginsbourger, E. Vazquez, V. Picheny, and Y. Richet. Fast paral- lel kriging-based stepwise uncertainty reduction with application to the identification of an excursion set. Technometrics, 56(4):455–465, 2014

    work page 2014

  7. [7]

    C. Choi, J. H. Choi, J. Li, and S. Malla. Shared cross-modal trajectory prediction for au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 244–253, 2021

    work page 2021

  8. [8]

    Corso, R

    A. Corso, R. Moss, M. Koren, R. Lee, and M. Kochenderfer. A survey of algorithms for black- box safety validation of cyber-physical systems. Journal of Artificial Intelligence Research, 72: 377–428, 2021

    work page 2021

Show all 60 references
  1. [9]

    Damle, V

    A. Damle, V. Minden, and L. Ying. Simple, direct and efficient multi-way spectral clustering. Information and Inference: A Journal of the IMA , 8(1):181–203, 06 2018. ISSN 2049-8772. doi: 10.1093/imaiai/iay008. URL https://doi.org/10.1093/imaiai/iay008

    2018 doi

  2. [10]

    W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao. A survey on safety-critical driving scenario generation—a methodological perspective. IEEE Transactions on Intelligent Trans- portation Systems, 24(7):6971–6988, 2023. 12

    2023

  3. [11]

    Z. Drezner. Computation of the bivariate normal integral. Mathematics of Computation , 32 (141):277–279, 1978

    1978

  4. [12]

    Drezner and G

    Z. Drezner and G. O. Wesolowsky. On the computation of the bivariate normal integral. Journal of Statistical Computation and Simulation , 35(1-2):101–107, 1990

    1990

  5. [13]

    S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, and H. X. Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953):620–627, 2023

    2023

  6. [14]

    P. I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018

    2018 arXiv

  7. [15]

    A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 , 2023

    2023 arXiv

  8. [16]

    Kalra and S

    N. Kalra and S. M. Paddock. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94:182–193, 2016

    2016

  9. [17]

    G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CA V 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pa...

    2017

  10. [18]

    M. Kennedy. Bayesian quadrature with non-normal approximating functions. Statistics and Computing, 8(4), 1998. URL https://doi.org/10.1023/A:1008832824006

    1998 doi

  11. [19]

    M. C. Kennedy and A. O’Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1–13, 2000

    2000

  12. [20]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    Klein, S

    A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics , pages 528–536. PMLR, 2017

    2017

  14. [22]

    Koopman and M

    P. Koopman and M. Wagner. Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety , 4(1):15–24, 2016

    2016

  15. [23]

    Koren, S

    M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer. Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV) , pages 1–7. IEEE, 2018

    2018

  16. [24]

    Kulesza, B

    A. Kulesza, B. Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning , 5(2–3):123–286, 2012

    2012

  17. [25]

    Kwiatkowska, G

    M. Kwiatkowska, G. Norman, and D. Parker. Prism 4.0: Verification of probabilistic real- time systems. In Computer Aided Verification: 23rd International Conference, CA V 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings 23 , pages 585–591. Springer, 2011

    2011

  18. [26]

    R. Lam, D. L. Allaire, and K. E. Willcox. Multifidelity optimization using statistical sur- rogate modeling for non-hierarchical information sources. In 56th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference , page 0143, 2015

    2015

  19. [27]

    Le Gratiet and C

    L. Le Gratiet and C. Cannamela. Cokriging-based sequential design strategies using fast cross- validation techniques for multi-fidelity computer codes. Technometrics, 57(3):418–427, 2015

    2015

  20. [28]

    R. Lee, O. J. Mengshoel, A. Saksena, R. W. Gardner, D. Genin, J. Silbermann, M. Owen, and M. J. Kochenderfer. Adaptive stress testing: Finding likely failure events with reinforcement 13 learning. Journal of Artificial Intelligence Research , 69:1165–1201, 2020

    2020

  21. [29]

    J. O. L¨ ubsen, C. Hespe, and A. Eichler. Safe multi-task bayesian optimization. arXiv preprint arXiv:2312.07281, 2023

    2023 arXiv

  22. [30]

    C. D. Manning, P. Raghavan, and H. Sch¨ utze. Xml retrieval. Introduction to Information Retrieval, 2008

    2008

  23. [31]

    Marco, F

    A. Marco, F. Berkenkamp, P. Hennig, A. P. Schoellig, A. Krause, S. Schaal, and S. Trimpe. Vir- tual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization. In 2017 IEEE International Conference on Robotics and Automation...

    2017

  24. [32]

    Montali, J

    N. Montali, J. Lambert, P. Mougin, A. Kuefler, N. Rhinehart, M. Li, C. Gulino, T. Emrich, Z. Yang, S. Whiteson, B. White, and D. Anguelov. The waymo open sim agents challenge. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks , 2023

    2023

  25. [33]

    Nayakanti, R

    N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 2980–2987. IEEE, 2023

    2023

  26. [34]

    Norden, M

    J. Norden, M. O’Kelly, and A. Sinha. Efficient black-box assessment of autonomous vehicle safety. arXiv preprint arXiv:1912.03618 , 2019

    1912 arXiv

  27. [35]

    A. O’Hagan. Bayes-hermite quadrature. Journal of Statistical Planning and Inference , 1991

    1991

  28. [36]

    M. E. O’Kelly, H. Abbas, S. Gao, S. Kato, S. Shiraishi, and R. Mangharam. Apex: Autonomous vehicle plan verification and execution. Technical report, SAE Technical Paper, 2016

    2016

  29. [37]

    Pleiss, J

    G. Pleiss, J. Gardner, K. Weinberger, and A. G. Wilson. Constant-time predictive distributions for gaussian processes. In International Conference on Machine Learning , pages 4114–4123. PMLR, 2018

    2018

  30. [38]

    Poloczek, J

    M. Poloczek, J. Wang, and P. Frazier. Multi-information source optimization. Advances in neural information processing systems , 30, 2017

    2017

  31. [39]

    N. V. Queipo, R. T. Haftka, W. Shyy, T. Goel, R. Vaidyanathan, and P. K. Tucker. Surrogate- based analysis and optimization. Progress in aerospace sciences, 41(1):1–28, 2005

    2005

  32. [40]

    Quinonero-Candela and C

    J. Quinonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research , 6:1939–1959, 2005

    1939

  33. [41]

    C. E. Rasmussen and Z. Ghahramani. Bayesian monte carlo. Neural Information Processing Systems, 2003

    2003

  34. [42]

    C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) . The MIT Press, 2005

    2005

  35. [43]

    R. Y. Rubinstein and D. P. Kroese. The cross-entropy method: a unified approach to combi- natorial optimization, Monte-Carlo simulation, and machine learning , volume 133. Springer, 2004

    2004

  36. [44]

    J. M. Scanlon, K. D. Kusano, T. Daniel, C. Alderson, A. Ogle, and T. Victor. Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain. Accident Analysis & Prevention , 163:106454, 2021

    2021

  37. [45]

    Sch¨ utt, J

    B. Sch¨ utt, J. Ransiek, T. Braun, and E. Sax. 1001 ways of scenario generation for testing of self-driving cars: A survey. In 2023 IEEE Intelligent Vehicles Symposium (IV) , pages 1–8. 14 IEEE, 2023

    2023

  38. [46]

    Segal, E

    S. Segal, E. Kee, W. Luo, A. Sadat, E. Yumer, and R. Urtasun. Universal embeddings for spatio-temporal tagging of self-driving logs. In Conference on Robot Learning, pages 973–983. PMLR, 2021

    2021

  39. [47]

    S. A. Seshia, D. Sadigh, and S. S. Sastry. Formal methods for semi-autonomous driving. In Proceedings of the 52nd Annual Design Automation Conference , pages 1–5, 2015

    2015

  40. [48]

    Shahriari, K

    B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015

    2015

  41. [49]

    Shi and J

    J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence , 22(8):888–905, 2000

    2000

  42. [50]

    Sinha, M

    A. Sinha, M. O’Kelly, R. Tedrake, and J. C. Duchi. Neural bridge sampling for evaluating safety-critical autonomous systems. Advances in Neural Information Processing Systems , 33: 6402–6416, 2020

    2020

  43. [51]

    Swersky, J

    K. Swersky, J. Snoek, and R. P. Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013

    2013

  44. [52]

    M. Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics , pages 567–574. PMLR, 2009

    2009

  45. [53]

    V. Tresp. A bayesian committee machine. Neural computation, 12(11):2719–2741, 2000

    2000

  46. [54]

    C. E. Tuncali, T. P. Pavlic, and G. Fainekos. Utilizing s-taliro as an automatic test generation framework for autonomous vehicles. In 2016 ieee 19th international conference on intelligent transportation systems (itsc) , pages 1470–1475. IEEE, 2016

    2016

  47. [55]

    Wickramarachchi, C

    R. Wickramarachchi, C. Henson, and A. Sheth. An evaluation of knowledge graph embeddings for autonomous driving data: Experience and practice. arXiv preprint arXiv:2003.00344, 2020

    2003 arXiv

  48. [56]

    Wilson and H

    A. Wilson and H. Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International conference on machine learning , pages 1775–1784. PMLR, 2015

    2015

  49. [57]

    Wilson, F

    J. Wilson, F. Hutter, and M. Deisenroth. Maximizing acquisition functions for bayesian opti- mization. Advances in neural information processing systems , 31, 2018

    2018

  50. [58]

    Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, A. J. Yang, and R. Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 1389–1399, 2023

    2023

  51. [59]

    overbudget

    Z. Zhong, Y. Tang, Y. Zhou, V. d. O. Neves, Y. Liu, and B. Ray. A survey on scenario- based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXiv:2112.00964, 2021. 15 A Algorithmic details A.1 BAMS algorithm For notational brevity in Algorithm...

    2021 arXiv

  52. [2023]

    URL https://proceedings.mlr.press/v205/bronstein23a.html

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.