{"id":"68f9e990-b9bc-4346-a1a0-b4353df644ee","arxiv_id":"2507.13729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An agentic LLM framework augments real-world traffic scenarios from text instructions, with output quality close to human-generated scenarios in blind expert evaluation.","lead":"This paper introduces an LLM-agent framework that edits real-world traffic scenes into more challenging versions using natural language instructions. The system performs nearly as well as human experts in blind visual comparisons and produces scenarios that are as difficult for a state-of-the-art planner as hand-made ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed-loop 'challenging' claim rests on one hand-modified planner with no variance; multi-planner test needed.","rationale":"The reader's weakest_assumption already identifies the closed-loop test's reliance on a single modified planner and the lack of error bars; my review agrees and sharpens this into the most load-bearing concern. The central claim of the paper, as stated in the abstract and contributions, is that the framework generates high-quality augmented scenarios comparable to human-created ones and that these scenarios challenge state-of-the-art planners. The Elo and displacement-error evaluations support quality in terms of following instructions and visual appearance, but they do not by themselves establish difficulty in closed-loop driving. The only direct evidence for the challenge claim is Table V, which uses one planner (PDM-Closed), modified by the authors to sample lateral offsets up to ±4m, and reports no variance. Because the paper explicitly claims 'SotA planning algorithms' in the plural, a single, hand-tuned planner is insufficient support. The proposed concrete test—evaluating multiple planners, including the unmodified PDM-Closed, with confidence intervals—would settle whether the parity in mean driving score is a general property or an artifact of one planner's configuration. The reader's verdict of CONDITIONAL remains appropriate, as this is an addressable gap rather than a fundamental flaw; my analysis does not change the verdict, so UNCHANGED is recommended.","tokens_in":10943,"tokens_out":6209,"duration_ms":76499,"concrete_test":"Run the 50 GPT-4oOTM-generated and 50 interPlan scenarios in nuPlan with (a) the original PDM-Closed (lateral offsets ±1m) and (b) at least one additional planner from the nuPlan benchmark, e.g., IDM with default parameters or UrbanDriver. For each planner, report mean driving score with bootstrap 95% CIs over the 50 scenarios. The concern lands if the interPlan-vs-GPT-4oOTM ordering reverses (e.g., generated scenarios become significantly easier than interPlan) or if the 2.3-point gap is within noise for some planners; it is refuted if generated scenarios are consistently at least as challenging as interPlan across all tested planners.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's contribution 3 ('scenarios which challenge SotA planning algorithms') is supported by a single closed-loop experiment in Section V-D using PDM-Closed, a rule-based planner that the authors modify by widening lateral-offset sampling from ±1m to ±4m specifically to handle blocked centerlines. The headline numbers (interPlan 51.9%, GPT-4oOTM 49.6%) are means over 50 scenarios with no confidence intervals, so parity is asserted without evidence of statistical equivalence. More fundamentally, 'challenge' is measured relative to one planner's hand-expanded proposal set; a scenario that requires a large lateral offset for this planner may be trivial for a more capable planner or unsolvable for the unmodified one. The Elo and BEV-preference results measure static visual appearance, not closed-loop difficulty, so they do not rescue the challenge claim. If the relative difficulty of generated vs human scenarios reverses across planners, the central claim that the framework produces equally challenging scenarios for SotA planners is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AGENTS-LLM, an agentic LLM framework for augmenting real-world traffic scenarios from natural-language instructions. The framework encodes scenarios as entity vectors and uses a Scenario Modifier Agent, optionally extended with function calling or with text-only/visual QA agents, to produce modified scenarios compatible with the interPlan interface. The evaluation recreates the 50 human-augmented interPlan scenarios and assesses quality via placement displacement error, pairwise human expert rankings converted to Elo ratings, and closed-loop nuPlan simulation with the PDM-Closed planner. The central claims are that the framework adheres to user intent, that GPT-4o with the one-time-modifier variant produces scenarios visually nearly indistinguishable from human-created ones, that advanced prompting lets smaller models approach this quality, and that the generated scenarios challenge a state-of-the-art planner comparably to interPlan.","tokens_in":11085,"tokens_out":6383,"duration_ms":78434,"significance":"If the claims hold, this is a practical step toward scalable creation of long-tail traffic scenarios for autonomous-driving evaluation. The paper has notable strengths: no training or fine-tuning of the LLMs is required, the scenario representation is generic and not tied to a specific simulator, human expert evaluation with Elo ratings and bootstrap confidence intervals is a thoughtful attempt at quantifying scenario quality, and a GitHub repository is provided. The main risks are that the 'challenging scenarios' claim rests on a single hand-modified planner with no variance reporting, and that the statistical language around 'indistinguishable' and 'equally challenging' is stronger than the evidence supports.","major_comments":[{"comment":"Contribution 3, that the framework 'generates scenarios which challenge SotA planning algorithms,' is supported by only one closed-loop experiment. Table V reports mean driving scores for PDM-Closed: Val14 90.8%, interPlan 51.9%, GPT-4oOTM 49.6%, Gemini-1.5-FlashOTM 53.5%, Llama3.1-70BOTM 54.0%. No confidence intervals, standard deviations, or per-scenario distributions are given, so the 2.3-point difference between interPlan and GPT-4oOTM cannot be distinguished from sampling noise. In addition, the planner is a hand-modified version of PDM-Closed with lateral offsets widened to ±4m, and no comparison with the original planner or with a learning-based planner (e.g., one of the nuPlan leaders cited in related work) is provided. The 'challenge' property is therefore demonstrated for one specific planner configuration only, and the plural 'planning algorithms' in the contribution is not supported. I would ask for at least one additional planner, for error bars or a paired test on the 50 scenarios, or for a softened claim.","section":"§V-D (Table V)"},{"comment":"The statement that GPT-4oOTM scenarios are 'almost indistinguishable' from human-generated interPlan scenarios rests on Elo ratings of 1039 versus 1042 with 95% CIs of -9/+11 for both systems. These intervals overlap substantially, which is not evidence of equivalence; a non-significant difference can be due to limited sample size (here, the two systems have 1720 and 1960 votes, respectively). The model-rank definition using CI bounds is also not a standard equivalence test. I recommend either adding an equivalence test with a pre-specified margin (e.g., a bound on the Elo difference considered practically negligible) or explicitly weakening the language to 'no statistically significant difference was detected' and discussing the power of the comparison.","section":"§V-C (Table IV)"},{"comment":"The text QA variant uses 'a list of common problems compiled from typical mistakes observed during initial experimentation' (Section III-B1). Because the evaluation set is the same 50 interPlan scenarios, the QA components may be implicitly tuned to failure modes of exactly those scenarios. This does not affect the GPT-4o OTM result, but it weakens the general claim that advanced prompting (tQA/vQA) closes the gap for smaller models: the improvement may not transfer to novel user instructions or scenario types. The paper should disclose this as a limitation and, ideally, evaluate on a held-out set of scenario descriptions not used to build the QA problem list.","section":"§III-B1 and §IV"}],"minor_comments":[{"comment":"The phrase 'who's goal' should be 'whose goal'.","section":"§III-B"},{"comment":"Figure 4 would be clearer if the caption stated that accident-site and construction-zone scenarios are excluded and if the number of scenarios per variant were annotated; currently this exclusion is mentioned only in the text.","section":"§V-B (Figure 4)"},{"comment":"Adding standard deviations or box plots of the per-scenario driving scores would help the reader judge the spread; the current mean-only presentation hides whether a few collisions dominate the average.","section":"§V-D (Table V)"},{"comment":"The QA rating threshold (average rating below 4 triggers revision) is a free parameter; a sentence on how this threshold was chosen or on its sensitivity would strengthen reproducibility.","section":"§IV"},{"comment":"The LCTGen comparison is partial because LCTGen could not produce two of the five scenario types; the text already says this, but the paragraph would be clearer if it explicitly noted that the per-category comparison is conditional on generated outputs only.","section":"§V-A"},{"comment":"Since vQA was run only for Gemini-1.5-Flash, the comparison between tQA and vQA is not fully controlled across model classes; please state this in the main text, as it is currently only implied.","section":"§V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an autonomous-driving/robotics venue and the engineering contribution is solid. My main recommendation for revision is to align the claims with the statistical support, especially for the closed-loop challenge evaluation and the Elo equivalence statement. The QA circularity is a secondary concern but should be disclosed clearly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on Agents-LLM. It's a genuine first-use paper: applying an agentic LLM design (function calling, QA loops) to augment real-world traffic scenarios from natural language, rather than generating from scratch. The evaluation is above the field's usual bar: displacement errors against the manual interPlan baselines, a blind Elo ranking with 5760 comparisons from nine experts, and closed-loop simulation in nuPlan. The practical insight that function calling lets cheaper models (Gemini-1.5-Flash, Llama3.1-70B) close much of the gap to GPT-4o is credible and useful.\n\nThe soft spots are real but not fatal. The 'equally challenging' headline rests on a single planner, PDM-Closed, and the authors modify it by widening lateral-offset sampling to ±4m. The resulting means (interPlan 51.9%, GPT-4oOTM 49.6%) have no confidence intervals, so parity is asserted rather than demonstrated. The paper also drops two of five scenario types in the advanced-prompting comparison, so the function-calling benefit is shown on a subset, though the authors acknowledge the ambiguity of those types. The Elo CIs overlap, making 'indistinguishable' partly a claim of non-significance, though the point estimate is at parity. Minor circularity: the QA question list was developed on the same interPlan scenarios used for evaluation; not a big deal given no parameter fitting, but worth noting.\n\nThe stress-test note about the closed-loop claim is largely right: relative difficulty could reverse across planners. But that does not undermine the core contribution, which is the generation framework itself. The paper would benefit from tempering the 'challenge SotA planners' contribution and adding at least one more planner or bootstrapped intervals on the driving scores.\n\nWho's this for: the AD testing community and anyone using LLMs for structured scene generation. It deserves a serious referee; I'd send it out with a request to address the statistical gaps and narrow the claims. Code is promised but not yet verifiable.","headline":"A solid first-use paper with a credible agentic LLM framework for scenario augmentation; the 'challenging to planners' claim is narrower than stated.","tokens_in":11649,"tokens_out":2512,"would_cite":true,"duration_ms":27708,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an agentic LLM framework can augment real-world traffic scenarios from natural-language instructions, producing scenarios that human experts rank on par with manually crafted ones and that challenge a…","keywords":["LLM agents","traffic scenario generation","scenario augmentation","autonomous driving testing","nuPlan","interPlan","human preference evaluation","function calling"],"falsifier":"Compare PDM-Closed's standard three-offset version (not the modified ±4m version) on a larger set of generated scenarios with matched user instructions: if its score on generated scenarios is far above interPlan's, or if a second blind expert study ranks the top LLM variant below interPlan by a wide margin in the previously excluded accident-site and construction-zone types, the parity claim would be weakened.","tokens_in":10744,"feed_emoji":"🚗","tokens_out":4387,"duration_ms":48741,"temperature":0.7,"pith_summary":"Rare safety-critical traffic situations are too scarce in real-world data to test autonomous-driving planners at scale, and manually augmenting recorded scenes is slow. This paper attempts to automate that augmentation: a team of LLM agents, coordinated by a Scenario Modifier Agent and optional quality-assurance agents, rewrites a text-based scenario description according to a user's natural-language request, then feeds the modified vectors into the nuPlan simulator. The central claim is that the generated scenarios match human-made ones: in a blind pairwise expert ranking, the best variant (GPT-4o with one-time modification) scores an Elo of 1039 against interPlan's 1042 with overlapping confidence intervals, and the PDM-Closed planner scores 49.6 percent on generated scenarios versus 51.9 percent on interPlan's, indicating comparable challenge. The authors also claim that cheaper LLMs, equipped with function calling or QA loops, close much of the gap to frontier models without any training or fine-tuning.","feed_headline":"LLM agents build traffic tests as challenging as human-made ones","feed_subtitle":"Blind expert ranking and a closed-loop planner put the top variant on par with manual scenario augmentation.","key_machinery":"The load-bearing mechanism is the agentic loop built around a text-based scenario description. A Scenario Modifier Agent receives the original scenario and user instructions, produces a summary, a modification dictionary, calculations, and modified agent vectors, optionally calling a lane-pose retrieval function when lanes are encoded as Bezier curves. An optional Quality Assurance stage—either a text QA agent using a checklist of common mistakes or a visual QA engineer plus a vision-language model answering questions about rendered bird's-eye-view images—rates compliance, realism, and logic, and sends corrective feedback back to the modifier. This loop, together with tool use, is what the paper credits for allowing smaller LLMs to approach the performance of frontier models.","core_discovery":"The paper's discovery is that an agentic, training-free LLM pipeline can edit recorded traffic scenes according to high-level user instructions and produce augmented scenarios that survive human and planner scrutiny. The generator represents scenes as fixed-form vectors for agents, lanes, lane connectors, and areas, so an LLM can reason about geometry, call a tool to retrieve lane coordinates, and emit modified vectors that interPlan's interface accepts. With GPT-4o as the modifier, the output is visually nearly indistinguishable from human-crafted scenarios in the blind Elo study, and it stresses the PDM-Closed planner almost as much as human augmentations do. The paper further claims that the agentic design is the reason compact models can compete: function calling corrects the dominant retrieval-error problem, and visual QA lets a cheap utility model (Gemini-1.5-Flash) approach frontier-model visual quality.","pith_inferences":["Editorial inference: the strongest comparison (Elo parity) is based on visual preference on bird's-eye-view images; the paper does not establish that this visual preference reliably predicts which scenarios will make planners fail, so closed-loop testing on a broader set of planners is the natural next check.","Editorial inference: the framework's natural-language interface turns scenario design into a specification task, which suggests a testable extension where a planner's observed failures are described back to the system to generate targeted regression scenarios.","Editorial inference: because the advanced-prompting comparison drops accident-site and construction-zone scenarios, the conclusion that small models catch up applies mainly to point-placement tasks; extending function calling to ambiguous, multi-object zones is an open test."],"forward_implications":["If the parity claim holds, autonomous-driving test suites can be expanded semiautomatically from natural-language specifications instead of requiring domain experts to hand-edit each scenario.","Because the scenario representation is a generic text format, the generated scenarios can be imported into nuPlan and could be adapted to other simulators without retraining a generator.","With function calling, utility and open-weight models reduce the displacement error gap, suggesting that expensive closed APIs are not strictly necessary for accurate placement.","The closed-loop result implies that such generated scenarios can serve as benchmark material for improving planners, since PDM-Closed's score drops to roughly the same level as on human-augmented scenarios.","The framework offers fine-grained control over placement, which matters for creating specific safety-critical situations such as a vehicle blocking an intersection rather than merely appearing near it."],"supporting_citations":[{"why":"Supplies the 50 human-augmented interPlan scenarios that the framework recreates and that serve as the ground truth and head-to-head comparator.","marker":"[9]"},{"why":"Provides the nuPlan closed-loop simulator and driving-score evaluation, including the PDM-Closed planner used for benchmarking.","marker":"[2]"},{"why":"Supplies the Elo rating and bootstrapped confidence-interval methodology used for the human pairwise comparison of scenario quality.","marker":"[11]"},{"why":"The language-conditioned traffic generation baseline adapted to the interPlan catalogue for displacement-error comparison.","marker":"[23]"},{"why":"The Hungarian algorithm used to match LLM-modified agents to human-modified agents when computing placement accuracy.","marker":"[40]"},{"why":"The agentic design reference that motivates the multi-step reasoning-and-acting structure of the scenario modifier and QA agents.","marker":"[10]"}],"fun_headline_variants":["LLM agent edits traffic scenes for harder AV validation","Agentic LLM generates traffic scenarios rivaling human-made ones","Training-free LLM framework augments road scenes via language","Tool-augmented LLM crafts high-quality traffic test scenarios","Language-guided LLM edits road scenes for autonomous driving tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that matching the visual appearance of and closed-loop score on the human-made interPlan scenarios is sufficient evidence that the generated scenarios are equally challenging and realistic for autonomous-driving planners.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent edits traffic scenes for harder AV validation","Agentic LLM generates traffic scenarios rivaling human-made ones","Training-free LLM framework augments road scenes via language","Tool-augmented LLM crafts high-quality traffic test scenarios","Language-guided LLM edits road scenes for autonomous driving tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2951,"prompt_tokens":939,"completion_tokens":2012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1930}},"tokens_in":555,"tokens_out":2012,"duration_ms":19104,"temperature":1.0,"reasoning_tokens":1930,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:17:51.093540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare PDM-Closed's standard three-offset version (not the modified ±4m version) on a larger set of generated scenarios with matched user instructions: if its score on generated scenarios is far above interPlan's, or if a second blind expert study ranks the top LLM variant below interPlan by a wide margin in the previously excluded accident-site and construction-zone types, the parity claim would be weakened.","supporting_citations":[{"cited_title":"Can Vehicle Motion Planning Generalize to Realistic Long- tail Scenarios?","cited_arxiv_id":null,"evidence_quote":"Supplies the 50 human-augmented interPlan scenarios that the framework recreates and that serve as the ground truth and head-to-head comparator."},{"cited_title":"NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles,","cited_arxiv_id":null,"evidence_quote":"Provides the nuPlan closed-loop simulator and driving-score evaluation, including the PDM-Closed planner used for benchmarking."},{"cited_title":"Chatbot arena: An open platform for evaluating llms by human preference,","cited_arxiv_id":null,"evidence_quote":"Supplies the Elo rating and bootstrapped confidence-interval methodology used for the human pairwise comparison of scenario quality."},{"cited_title":"Language Conditioned Traffic Generation,","cited_arxiv_id":null,"evidence_quote":"The language-conditioned traffic generation baseline adapted to the interPlan catalogue for displacement-error comparison."},{"cited_title":"React: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"The agentic design reference that motivates the multi-step reasoning-and-acting structure of the scenario modifier and QA agents."}],"review_version":1}