{"id":"161f5a53-c0d7-429e-a0e3-0e0b4d0dea49","arxiv_id":"2506.01199","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A goal-prompt search with Bayesian optimization over a data-driven traffic simulator automatically finds safety-critical scenarios for testing autonomous vehicle planners.","lead":"Researchers automated the search for dangerous traffic scenarios by letting a learning-based simulator steer simulated drivers toward goals, and using Bayesian optimization to pick goals that cause collisions. The method lets engineers stress-test self-driving planners against realistic human behaviors without hand-crafting every test case.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The realism claim rests on an unvalidated distribution shift: overwriting ProSim's ego policy with an external MPC planner may make other agents' reactions unnatural.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing concern: the realism of the generated multi-agent interactions is asserted but not validated after the ego policy is replaced by an external MPC planner. The central claim bundles realism with safety-criticality and efficiency, and while the latter two have some quantitative support (Table 1), realism is supported only by qualitative examples and ProSim's pretraining. The distribution shift from overwriting the ego plan is a concrete mechanism by which realism could break, and no experiment checks it. The paper's conclusion acknowledges broader limitations (e.g., inability to model irrational behaviors) but does not address this specific unvalidated integration step. A targeted experiment comparing trajectory distributions with and without the overwrite, or against WOMD data, would settle the concern. Given that the paper is a proof-of-concept with plausible methodology, the conditional verdict remains appropriate; the missing realism validation strengthens the need for that condition rather than overturning the verdict.","tokens_in":6542,"tokens_out":3031,"duration_ms":38673,"concrete_test":"Using the released ProSim checkpoint, run the three reported scenarios (front, front-right, behind) under two conditions: (a) ProSim's original ego policy, and (b) the external MPC overwrite, with identical goal prompts and initial states. Compare the distributions of the simulated non-ego agent trajectories using a tractable metric—e.g., mean pairwise displacement error between paired episodes, off-road rate, kinematic feasibility, or negative log-likelihood under a learned behavior model. If the overwrite causes statistically significant divergence (for instance, mean pairwise displacement above 1 m or a meaningful increase in off-road rate), the realism claim fails. Additionally, compare the non-ego trajectories to real WOMD highway trajectories to check whether the generated interactions remain within the training distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that BO prompt sampling generates 'realistic and safety-critical' test cases. The safety-critical part is supported by collision rates in Table 1 (e.g., BO collision rates 16%/35%/21% vs random 0%/12%/3%), and the efficiency part by the BO comparison. The 'realistic' part, however, depends on an unvalidated distribution shift. In the episode generation module, the ego agent's ProSim-generated plan is overwritten with a plan from an external MPC planner (Section 3, 'Data-driven Traffic Simulator'). ProSim's policies for the other agents were trained closed-loop under ProSim's own ego policy distribution. Replacing the ego policy with an MPC planner that assumes other agents track lane centers at constant velocity creates a different closed-loop distribution. When the MPC brakes, accelerates, or changes lanes in ways a learned human-like driver would not, the simulated agents' reactions—trained under the original ego distribution—may become unnatural, overly passive, or artificially collision-prone. No experiment validates that the overwritten ego plan preserves realism for the other agents. The paper offers only qualitative examples (Fig. 4) and an appeal to ProSim's pretraining. There is no comparison to WOMD trajectories, no human-likeness metric, and no measurement of the distribution shift introduced by the external planner. Since 'realistic' is half of the central claim, this missing validation is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an automated method for generating safety-critical and realistic interactive scenarios to evaluate an autonomous vehicle (AV) planner. Human behaviors are parameterized by low-dimensional goal positions, which are fed as prompts to ProSim, a pretrained promptable closed-loop traffic simulator. A Bayesian optimization (BO) module explores the goal domain and maximizes a criticality score defined as the negative minimum distance between the ego vehicle and other agents. The planner under test (an MPC controller) runs in closed loop with the simulated agents, and the paper reports experiments in a 2-agent highway setting with three initial positions of the other agent. The central claims are that BO-sampled prompts yield more safety-critical behaviors (higher collision rates, lower minimum distances, lower time-to-collision) than random sampling, while generating diverse and realistic behaviors.","tokens_in":6921,"tokens_out":3891,"duration_ms":46088,"significance":"If the claims hold, the paper makes a practical contribution by showing how a low-dimensional prompt space can control a data-driven traffic simulator for automated AV testing, avoiding the high dimensionality of prior search-based methods. The use of an existing pretrained simulator (ProSim) without fine-tuning, and the integration of BO with a user-defined criticality metric, are appealing and could be broadly applicable. The paper honestly states limitations (only goal-driven behaviors, no out-of-distribution or irrational behaviors). However, the 'realistic' part of the central claim is asserted but not measured: the paper provides no validation that the closed-loop interactions, after overwriting the ego agent's learned plan with an external MPC planner, remain realistic. The efficiency claim also rests on a single run of BO and random sampling without error bars or multiple seeds. These gaps are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The central claim is that the method generates 'realistic and safety-critical' test cases. The safety-critical part is supported by Table 1, but the realism part is not validated. In the episode generation module, the ego agent's ProSim-generated plan is overwritten by the external MPC planner (Figure 3), so the other agents' closed-loop reactions are produced by policies trained under ProSim's original ego-policy distribution. No experiment compares the resulting multi-agent interactions against ground-truth human driving data (e.g., WOMD trajectories) or any human-likeness/reality-gap metric. The qualitative examples in Figure 4 and the appeal to ProSim's pretraining do not establish that the distribution shift introduced by the MPC planner preserves realism. Since 'realistic' is half of the central claim, this missing validation is load-bearing and should be addressed with a concrete quantitative comparison.","section":"Section 2, 'Episode Generation Module'; Section 3, 'Data-driven Traffic Simulator'"},{"comment":"The efficiency claim—that BO outperforms random sampling—is based on point estimates from a single BO run and a single Sobol random sample per initial condition. Collision rates are reported as single numbers (e.g., 16% vs 0%) with no confidence intervals, and the mean metrics have overlapping standard deviations (e.g., Front right Min Dist: 1.84 ± 1.87 vs 3.01 ± 1.66; TTC: 0.66 ± 0.86 vs 1.34 ± 0.93). Without multiple independent runs or a statistical analysis, the observed differences could be within noise. The paper should report error bars or a small number of repeated BO and random runs with their variability.","section":"Table 1 and Section 3, 'Effectiveness of Bayesian optimization'"}],"minor_comments":[{"comment":"The phrase 'in a automated manner' should be 'in an automated manner.'","section":"Section 1, Figure 1 caption"},{"comment":"The text after Eq. (3) refers to 'ns' as the total number of episodes, but the formulas use 'ne'. Please make the notation consistent.","section":"Equations (2) and (3)"},{"comment":"The paper states that BO 'achieves comparable diversity as random sampling,' but for the 'Behind' condition, EgoASD is 0.39 for random versus 4.48 for BO, a large difference. Please clarify whether the diversity comparison is meant to be across the combined metrics or only for agent diversity.","section":"Table 1, 'Behind' row for EgoASD"},{"comment":"The BO setup is described only briefly ('Matern Kernel,' 'Upper Confidence Bound'); the acquisition-function parameter and kernel hyperparameters are not specified, which makes the experiments difficult to reproduce. Please provide these details.","section":"Section 2, 'Prompt Generation Module'"},{"comment":"The generator is run for 75 BO iterations 'without initialization,' but it is unclear whether the 75 random samples are Sobol points in the same goal domain and whether the comparisons use the same evaluation protocol. Please clarify the exact number of samples used for the random baseline and whether the metrics are computed over the full 75 episodes or only the non-colliding ones.","section":"Section 3, 'Evaluation of the automated behavior generation'"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.AI and the idea is promising. However, the realism validation gap and the lack of statistical rigor in the efficiency comparison are both central to the paper's claims. The authors should be required to provide quantitative evidence for realism and to strengthen the BO-vs-random comparison before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a modest integration, but the new bit is real. The novel step is using Bayesian optimization to search ProSim's goal-prompt space for safety-critical episodes, and Table 1 shows it finds more collisions than Sobol random sampling at the same sample count: 16/35/21 percent versus 0/12/3 across three initial conditions. The diversity numbers are roughly comparable, so the method does not just collapse to one adversarial goal. I also give credit for the closed-loop setup and for testing an external MPC planner without tuning it to the simulator.\n\nThe soft spots are real. The load-bearing one is the realism claim. The paper overwrites ProSim's ego plan with the external MPC planner, and ProSim's other-agent policies were trained under ProSim's own ego-policy distribution. The planner predicts that other vehicles track lane centers at constant velocity, so the closed-loop distribution is shifted. No experiment checks that the other agents still behave like humans under that shift: there is no comparison to WOMD trajectories, no human-likeness metric, only Figure 4. Since 'realistic' is half of the central claim, this missing validation matters.\n\nThe safety-critical half is better supported but still thin. Collision rates are point estimates with no error bars and no repeated seeds, so we cannot tell whether 16% versus 0% is stable. The demonstration covers one planner and three initial conditions, and no code or data are provided. The conclusion does honestly state that the method only covers uncertainty from unknown goal positions, not drunk, distracted, or panicked behavior, which is a useful caveat but does not address the distribution shift.\n\nThis paper is for people working on scenario-based testing and simulation-based AV evaluation. It is a useful data point, not a breakthrough. The integration is plausible and worth building on, so I would send it to review with a request for statistical repetition and a direct realism check. It deserves a serious referee, even though the evidence currently undersells the strongest claim.","headline":"A modest but genuine integration of BO over goal prompts in a promptable traffic simulator, with efficiency evidence that holds up but a realism claim that is asserted rather than validated.","tokens_in":7312,"tokens_out":2179,"would_cite":true,"duration_ms":25652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By sampling goal-position prompts with Bayesian optimization, an automated pipeline turns a promptable traffic simulator into a source of realistic, safety-critical test episodes for autonomous vehicle planners.","keywords":["autonomous vehicle planning","traffic simulation","scenario-based testing","Bayesian optimization","goal prompting","safety-critical scenarios","closed-loop simulation","promptable traffic simulator"],"falsifier":"Run the same BO goal-prompt search and compare the generated multi-agent trajectories against real Waymo Open Motion Dataset driving clips using standard realism metrics (e.g., kinematics distributions, interaction likelihood, or a learned discriminator). If episodes generated with the external planner overwriting ProSim's ego plan are systematically distinguishable from native ProSim rollouts—or from real data—while episodes without the overwrite are not, the claim that the safety-critical test cases remain 'realistic' would be refuted.","tokens_in":6343,"feed_emoji":"🚗","tokens_out":6491,"duration_ms":59320,"temperature":0.7,"pith_summary":"This paper claims that an autonomous vehicle (AV) planner can be stress-tested automatically by treating human drivers' unknown destinations as tunable 'goal prompts' in a data-driven traffic simulator, and then searching over those prompts with Bayesian optimization. The authors argue that goal positions are a low-dimensional, human-meaningful parameterization of driving behavior: they preserve the realism of a learned simulator while making the search for dangerous interactions tractable. In closed-loop experiments with an optimization-based planner, Bayesian optimization over goal positions produced collisions in 16%, 35%, and 21% of episodes across three initial traffic configurations, versus 0%, 12%, and 3% for random sampling, with comparable behavioral diversity. The upshot is a planner-agnostic way to surface safety-critical interaction scenarios more efficiently than random generation, useful for pre-deployment evaluation.","feed_headline":"Bayesian goal sampling finds crash-inducing simulated driver behaviors","feed_subtitle":"A promptable traffic simulator plus Bayesian goal search automatically finds safety-critical AV test scenarios.","key_machinery":"The load-bearing object is the goal-position prompt: a low-dimensional vector specifying where each simulated agent intends to drive, which the transformer-based ProSim generator converts into a per-agent policy token. Bayesian optimization, implemented with a Gaussian process surrogate (Matern kernel) and an upper-confidence-bound acquisition function, searches the goal domain to maximize a hindsight criticality score defined as the negative minimum Euclidean distance between the ego vehicle and any other agent over the episode. The episodes are produced in closed loop: the external planner under test overwrites the ego plan inside ProSim's policy rollout, while the simulated agents react to updated state observations. This combination—expressive learned behavior, a two-dimensional prompt space, and a sample-efficient optimizer—is what lets the method claim both realism and efficiency.","core_discovery":"The paper's central claim is that complex, realistic human behaviours in interactive traffic can be parameterized by a small number of numerical goal positions, and that Bayesian optimization over these goal prompts efficiently identifies the ones that are safety-critical for a planner under test. Using the ProSim promptable closed-loop traffic simulator, the authors assign each non-ego agent a goal prompt sampled from a user-defined drivable domain, run the planner in the loop, and score each episode by the minimum distance between the ego vehicle and other agents. The optimizer then proposes new goal positions that minimize that distance (equivalently, that maximize criticality). In the reported 2-agent highway experiments, the approach generated diverse behaviours for each initial condition and discovered planner failures—such as rear-end collisions after an unexpected brake, or a merge cut-in from behind—that reveal the planner's incorrect assumptions about other agents' goals. The authors conclude that the method produces realistic and safety-critical test cases automatically, with higher collision rates than random goal sampling at equal sample counts.","pith_inferences":["A natural next experiment, not run in the paper, would compare the distribution of multi-agent trajectories (with the external planner overwriting the ego plan) against native ProSim rollouts or WOMD ground-truth data on standard human-likeness metrics; if the overwrite breaks realism, the 'realistic' half of the central claim would need qualification.","The same goal-prompt search could be applied to learning-based planners, but BO's UCB acquisition may need re-tuning because learned planners react differently to distribution shift than the MPC baseline used here.","The criticality metric is pluggable; replacing minimum distance with, say, TTC or a learned risk model could yield different goal positions that are dangerous in other senses, so the set of 'safety-critical' behaviors found is metric-dependent.","Since the paper notes that irrational human behaviors (drunk, distracted, panic) fall outside goal prompting, a hybrid that mixes goal prompts with stochastic behavioral perturbations could cover a wider failure space; this is a plausible path the authors themselves gesture at."],"forward_implications":["AV planners can be evaluated against a broad, user-tunable range of human behaviors by simply constraining the goal domain, from traffic-rule-compliant goals to all physically reachable positions.","Because goal prompts are low-dimensional, Bayesian optimization stays sample-efficient and scales as the number of simulated agents grows.","The method is planner-agnostic: it makes no assumptions about the planner under test, so the same pipeline can be pointed at any planner by swapping the closed-loop integration.","Safety-critical episodes found by BO (16%, 35%, 21% collisions across the three initial conditions) demonstrate that rare dangerous interactions can be surfaced with far fewer samples than random search.","Diversity of generated behaviors is comparable to Sobol random sampling, so the planner is not just being attacked along one repeated failure mode."],"supporting_citations":[{"why":"ProSim provides the promptable closed-loop traffic simulator that turns goal prompts into realistic agent policies.","marker":"[15]"},{"why":"Waymo Open Motion Dataset is the underlying training data and map configuration for ProSim in the experiments.","marker":"[8]"},{"why":"A Bayesian-optimization worst-case scenario search that the paper compares against conceptually and extends via low-dimensional goal prompts.","marker":"[10]"},{"why":"An earlier Bayesian-optimization-based test-case identification method for automated driving that motivates the BO choice.","marker":"[9]"},{"why":"Lookout supplies the average self-distance metric used to quantify episode diversity for ego and other agents.","marker":"[4]"}],"fun_headline_variants":["Goal-prompt search exposes planner blind spots in interactive driving","Bayesian optimization over goal prompts finds crashing traffic scenarios","Automatic goal sampling yields safety-critical interactive traffic tests","Promptable traffic sim plus Bayesian search automates AV safety tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results are only as convincing as the untested assumption that overwriting the ego vehicle's plan inside ProSim with an external MPC planner does not distort the realism of the other agents' behavior in the resulting multi-agent interactions.","fun_headline_variants_meta":{"raw":{"variants":["Goal-prompt search exposes planner blind spots in interactive driving","Bayesian optimization over goal prompts finds crashing traffic scenarios","Automatic goal sampling yields safety-critical interactive traffic tests","Promptable traffic sim plus Bayesian search automates AV safety tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2970,"prompt_tokens":893,"completion_tokens":2077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2011}},"tokens_in":509,"tokens_out":2077,"duration_ms":16301,"temperature":1.0,"reasoning_tokens":2011,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:49:34.124703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same BO goal-prompt search and compare the generated multi-agent trajectories against real Waymo Open Motion Dataset driving clips using standard realism metrics (e.g., kinematics distributions, interaction likelihood, or a learned discriminator). If episodes generated with the external planner overwriting ProSim's ego plan are systematically distinguishable from native ProSim rollouts—or from real data—while episodes without the overwrite are not, the claim that the safety-critical test cases remain 'realistic' would be refuted.","supporting_citations":[{"cited_title":"Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset","cited_arxiv_id":null,"evidence_quote":"Waymo Open Motion Dataset is the underlying training data and map configuration for ProSim in the experiments."},{"cited_title":"A bayesian optimization method for finding the worst-case scenarios of autonomous vehicles","cited_arxiv_id":null,"evidence_quote":"A Bayesian-optimization worst-case scenario search that the paper compares against conceptually and extends via low-dimensional goal prompts."},{"cited_title":"Iden- tification of test cases for automated driving systems using bayesian optimization","cited_arxiv_id":null,"evidence_quote":"An earlier Bayesian-optimization-based test-case identification method for automated driving that motivates the BO choice."},{"cited_title":"Lookout: Diverse multi-future predic- tion and planning for self-driving","cited_arxiv_id":null,"evidence_quote":"Lookout supplies the average self-distance metric used to quantify episode diversity for ego and other agents."}],"review_version":1}