{"id":"2f073d17-39b1-4603-9e0f-f1fcbc4aa539","arxiv_id":"2608.04896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Actor-blind probes outrank human replay and PDM-Closed on NAVSIM's full navtest split when a dependency-sensitive velocity refit diverges and reference-conditioned forgiveness turns shared failures into passes.","lead":"This paper shows that a numerical instability in NAVSIM's shared trajectory re-simulation, combined with a forgiveness rule, lets actor-blind driving policies beat human replay and strong baselines on the standard navtest split. The result matters because it means defensive-driving leaderboard scores can be inverted by computational noise, and it offers an audit protocol for catching such score-basis failures before drawing behavioral conclusions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Minimal solver controls are not run on the full navtest split, so the abstract's 'direct trigger' causal claim is not fully secured on the headline measurement.","rationale":"I read the paper as an unusually disciplined audit. The full-navtest anomaly is reproduced bit-exactly, the overwrite mechanism is quantified, ingredient ablation shows forgiveness is necessary, and the exact-input diagnostic localizes the pseudoinverse behavior. Machine-checked proofs are absent, but the artifacts and frozen hashes are concrete evidence. The one place the central causal claim overreaches is the move from scoped solver and stack controls to the abstract's 'direct trigger' wording for the full-split margin. The full-split stable-basis reversals replace the whole rollout transform or guard divergence, not just the solver; the minimal intervention is only run on 450 and 32 tokens. This does not undermine the descriptive result or the proposed audit protocol, but it means the causal generalization should remain conditional pending the same control on navtest. This matches the reader's weakest_assumption, so I agree with the CONDITIONAL verdict and would not change it.","tokens_in":12576,"tokens_out":7562,"duration_ms":87036,"concrete_test":"Run the Table 5 solver/stack controls on the complete 12,146-token navtest split: with forgiveness enabled, evaluate Ignore-All, human replay, PDM-Closed, and the route-aware actor-blind probe under (a) affected stack plus stock pinv, (b) affected stack plus direct solve, and (c) same-source control stack plus stock pinv, using the released token list and frozen artifacts. Confirm that divergence rate and core overwrite rate drop from roughly 100% and that the pairwise margins reverse (IA minus Human becomes negative) in (b) and (c). If blind-last ordering is restored, the direct-trigger claim is confirmed; if not, an additional navtest-specific cause exists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The descriptive anomaly is solidly evidenced. The load-bearing gap is the causal step from controlled localization to the full 12,146-token navtest reversal. Table 5 shows that on the 450-token navhard stage-1 pool, replacing only the stock pseudoinverse with a direct solve or Hermitian pseudoinverse, or using the same-source control stack, eliminates divergence and makes Ignore-All last. Table 6 repeats this on a 32-token diagnostic. But on the full navtest split, the only stable-basis reversals are the two-sided finite-difference diagnostic (Table 7) and the fallback guard that replaces BatchLQR once divergence exceeds thresholds. Both are broader interventions than the minimal solver substitution. It therefore remains possible that a second dependency-sensitive failure on the navtest distribution contributes to the 79.6 vs 74.0 margin, or that the navhard control pool is not representative of navtest for this instability. The Limitations section honestly labels these as scoped controls, but the abstract's 'direct trigger' statement is stronger than the evidence on the headline split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper audits NAVSIM v2.2 original-scene single-stage scoring. It demonstrates that under a specific documented dependency stack (commit 0a380a9, OpenBLAS, NumPy 1.23.4, SciPy 1.13.1), two actor-blind probes—the route-blind Ignore-All and a route-aware actor-blind probe—outrank human replay and PDM-Closed on the complete 12,146-token navtest split. The paper attributes the reversal to shared-failure forgiveness collapse: the logged human reference and the agent share the same unstable velocity-refit rollout; when the reference fails compliance channels, reference-conditioned forgiveness rewrites agent failures to passes. It supports this claim with ingredient ablation, solver replacement on a 450-token control pool, a fresh-install 32-token reproduction, an exact-input 40x40 diagnostic, and a two-sided finite-difference diagnostic that restores blind-last ordering. The paper also contributes a reusable audit protocol requiring score-basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests.","tokens_in":12686,"tokens_out":6960,"duration_ms":77102,"significance":"If the findings hold, they identify a concrete and reproducible failure mode in a widely used re-simulation benchmark: a dependency-sensitive numerical instability in a shared planner/reference transform can invert the intended ranking of actor-aware versus actor-blind policies. The paper's strengths are its transparency (frozen artifacts, SHA-256 hashed exact inputs, token lists, per-token outputs), its multi-layered controls (filter-off, solver replacement, same-source stack, two-sided diagnostic), and its honest scoping of the 32-token diagnostic as mechanism evidence rather than prevalence. The audit protocol itself is a useful contribution. The main weakness is the gap between the scoped solver-replacement evidence and the full-split causal claim, which the abstract states more strongly than the evidence supports.","major_comments":[{"comment":"The abstract states that 'the numerical instability is the direct trigger' of the full 12,146-token navtest reversal (Ignore-All 79.6 vs. human replay 74.0, Table 2). The minimal solver control that isolates this trigger is run on the 450-token navhard stage-1 pool (Table 5) and the 32-token diagnostic set (Table 6), not on the navtest split where the headline reversal is measured. On the full split, the two-sided finite-difference diagnostic (Table 7) and the fallback guard are broader interventions that change the score basis beyond the minimal velocity-refit solver replacement. The Limitations section (Sec. 6) correctly labels these controls as 'scoped,' but the abstract and §5 do not carry the same caveat. It therefore remains open whether the navtest reversal is fully explained by the refit instability alone, or whether a second dependency-sensitive failure on the navtest distribution, or a representativeness gap between the navhard control pool and navtest, contributes to the 79.6 vs. 74.0 margin. Since the full token list and stack are provided, running the solver replacement (direct solve or Hermitian pseudoinverse) on the full navtest split appears feasible and would directly secure the causal claim; otherwise the 'direct trigger' wording should be relaxed to 'the direct trigger on the audited control pools.'","section":"Abstract and §5 (with Tables 5–7 and §6 Limitations)"}],"minor_comments":[{"comment":"The title contains a spacing typo: 'NA VSIM' should be 'NAVSIM'; similar spacing issues appear in table cells such as 'Ignore-All79.6' and 'Turn+ moving' in Tables 2–8.","section":"Title and throughout"},{"comment":"The scoring notation is inconsistent: the text writes m_j(T(x), s) but the definition of T in the preceding line suggests T takes (x, s). Please clarify the argument structure, e.g., write T(x, s) or define the composition explicitly.","section":"§3.3"},{"comment":"The claim that 'no convex reweighting of the component scores gives PDM-Closed more than a 0.17-point margin over Ignore-All' is not accompanied by a definition of the component score set or the reweighting constraints; please add a footnote or supplementary detail describing the optimization and the components included.","section":"§4.4"},{"comment":"The term 'BatchLQR' is introduced without definition; please state that it is the velocity/curvature refit routine in the NAVSIM scorer when it is first mentioned.","section":"§4.2"},{"comment":"The paper says that a second stock run 'exactly reproduces all per-token score outputs,' but it does not state whether this second run used a different machine or the same audited backend; please specify the conditions of the reproducibility run.","section":"§3.4 and §4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a careful and transparent audit, but the abstract's 'direct trigger' claim is stronger than the split-specific evidence. A full-split solver replacement seems feasible given the provided artifacts and would resolve the main concern. The editor may also want to confirm that the reported NAVSIM commit and dependency versions reproduce with the provided code before acceptance, since the entire contribution rests on stack-specific behavior."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real and unusually careful audit of NAVSIM v2.2 scoring: under the documented affected stack, both a route-blind and a route-aware actor-blind probe outrank human replay and PDM-Closed on the full 12,146-token navtest split. Second, the causal claim that the numerical instability is the 'direct trigger' of that headline reversal is a bit stronger than the evidence on the exact headline split: the minimal solver replacement that isolates the trigger is run on a 450-token navhard pool and a 32-token diagnostic set, not on the 12,146-token navtest split. The full-navtest mechanism evidence comes from broader interventions (two-sided finite differences and the fallback guard). That is a scoped gap, not a fatal one, and the paper openly labels these as scoped controls in the limitations section.\n\nWhat the paper does well: it separates a descriptive ranking anomaly from a scorer-side mechanism using four ordered checks. The ingredient ablation (filter-off zeros) shows the blind advantage requires both the shared failure and forgiveness. The exact-input 40x40 diagnostic is a good piece of evidence linking identical inputs to different outcomes across stacks. The fresh-install reproduction on 32 tokens, with identical source hashes, is solid. The audit protocol itself is a useful contribution: blind probes, overwrite reporting, dependency disclosure, and rollout stability checks. The authors are also appropriately careful not to claim prevalence; they state repeatedly that their findings apply only to the audited configuration and that the 32-token set is not a prevalence sample.\n\nThe main soft spot is exactly the one I mentioned: the abstract's 'direct trigger' sentence is a causal generalization from the 450-token and 32-token controls to the full navtest split. The two-sided and fallback controls on the full split do narrow the explanation, but they are broader than a solver-only swap. If a second dependency-sensitive failure were present on the navtest distribution, the minimal solver replacement might not fully explain the headline margin. I would not call this a load-bearing flaw, because the paper's own limitations section discloses the scope, and the descriptive anomaly is solid. But the wording of the abstract should be softened to match the evidence.\n\nWho is this for: anyone using NAVSIM or similar re-simulation benchmarks to support defensive-driving claims, and people designing evaluation protocols. It deserves a serious referee. I would send it out, with a note asking the authors to align the abstract's causal phrasing with the actual control coverage.","headline":"A careful, honestly scoped audit showing that a shared velocity-refit instability plus reference-conditioned forgiveness can invert NAVSIM rankings; the causal 'direct trigger' claim is slightly ahead of the evidence on the full navtest split, but the paper deserves peer review.","tokens_in":13280,"tokens_out":2461,"would_cite":true,"duration_ms":28005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A documented NAVSIM scorer configuration ranks a route-blind policy above human replay and PDM-Closed on the full navtest split; the paper traces this to a numerical instability that forgiveness turns into compliance credit.","keywords":["defensive driving evaluation","benchmark auditing","reference-conditioned forgiveness","actor-blind probes","NAVSIM","numerical stability","shared-failure forgiveness collapse","rollout stability"],"falsifier":"Run the solver-replacement control and the same-source dependency stack control on the full 12,146-token navtest split with forgiveness enabled, and compare per-token rollouts and final EPDMS scores: if Ignore-All falls below human replay and PDM-Closed and the divergence rate drops to near zero, the instability is confirmed as the direct trigger, while if the blind win persists despite the stabilized shared refit, the causal conclusion is incomplete.","tokens_in":12307,"feed_emoji":"🚗","tokens_out":11147,"duration_ms":104046,"temperature":0.7,"pith_summary":"The paper argues that a defensive-driving score is only meaningful if it keeps actor-aware policies above actor-blind ones, and then shows a documented NAVSIM scorer configuration violates that condition. Under the affected numerical stack, a route-blind policy that ignores all surrounding actors scores 79.6 EPDMS, outranking human replay (74.0), DiffusionDrive (73.7), and PDM-Closed (67.1) over the complete 12,146-token navtest split. The cause is a dependency-sensitive instability in the shared velocity refit: both the agent proposal and the logged human reference traverse the same transformation, and when the reference rollout degenerates, the scorer's reference-conditioned forgiveness—which grants credit whenever the logged human reference fails a compliance channel—overwrites the agent's compliance failures into passes. Replacing the solver or using a same-source dependency stack removes the divergence and restores blind-last ordering on the 450-token control pool, so the paper concludes that the numerical instability is the direct trigger and forgiveness is the amplifier. The upshot is an audit protocol—dependency and backend disclosure, blind probes, overwrite reporting, and rollout stability tests—for any re-simulation score used to support defensive-driving claims.","feed_headline":"Blind 'Ignore-All' probe tops human replay in NAVSIM audit","feed_subtitle":"A numerical glitch in the shared rollout plus forgiveness reverses the leaderboard; swapping the solver restores blind-last order.","key_machinery":"The load-bearing mechanism is shared-failure forgiveness collapse, which needs three ingredients: a transformation $T(\\cdot, s)$ applied to both the agent proposal $x_a$ and the logged human reference $x_r$; a reference failure mask $F_j(x_r,s) = \\mathbb{1}[m_j(T(x_r, s), s) = 0]$ for each compliance channel $j$; and a reference-conditioned scoring rule that sets $\\tilde{m}_j(x_a, x_r, s) = 1$ whenever $F_j(x_r, s) = 1$ and otherwise uses $m_j(T(x_a, s), s)$. When the shared refit is numerically unstable, both sides fail the structural channels, so the masks convert almost every compliance failure into a pass and leave only residual terms to determine the ranking. The diagnostics that carry the argument are the stock filter-on versus filter-off comparison, a two-sided finite-difference diagnostic that bypasses the unstable refit on both sides, solver replacement with a direct solve or Hermitian pseudoinverse, and a same-source dependency stack control.","core_discovery":"The paper's central claim is that, in the audited NAVSIM v2.2 original-scene single-stage scorer under the affected documented-stack condition, reference-conditioned forgiveness plus an unstable shared rollout transformation produces what the authors call shared-failure forgiveness collapse. Concretely, the default pseudoinverse in the velocity refit can produce unphysical profiles (median simulated endpoint of 5.13 km on a fixed 32-token diagnostic set), the logged human reference fails the same compliance channels, and the scorer then overwrites both sides' failures to passes in Drivable Area Compliance, Driving Direction Compliance, and lane keeping on roughly 99.9% of audited tokens. The result is that the route-blind Ignore-All probe (79.6 EPDMS) and a route-aware actor-blind probe (79.2) outrank human replay (74.0), DiffusionDrive (73.7), and PDM-Closed (67.1) on the full 12,146-token navtest split. Because the ordering reverses under a two-sided finite-difference diagnostic and on 450-token control pools when only the solver is replaced, the paper concludes that the numerical instability is the direct trigger, not a flaw in forgiveness alone.","pith_inferences":["If other re-simulation benchmarks share the same structure—a shared rollout transform plus reference-conditioned forgiveness—the collapse could reproduce there; adding a fixed numerical-stability diagnostic to their test suites would catch it cheaply.","Running the solver-replacement control on the full 12,146-token navtest split would settle whether the numerical instability is the whole story or only part of it; the paper leaves that run undone.","A practical implication the authors leave implicit is that benchmark leaders could require a score-basis attestation—exact dependency versions, backend, and thresholds—as a condition of submission, making ranking anomalies auditable.","The route-aware actor-blind probe could be reused as a standard sanity-check baseline in any planner benchmark, since it separates route-following competence from actor-awareness without retuning."],"forward_implications":["Published NAVSIM-style rankings produced under the affected stack can invert: a policy that receives no surrounding-actor input can top actor-aware comparators, so such scores do not by themselves support actor-sensitive defensive-driving claims.","The fail-to-pass overwrite is concentrated in structural channels—DAC, DDC, and lane keeping—while TTC and no-at-fault collision are barely touched, so the collapse removes the very constraints that would penalize non-compliant trajectories.","Stabilizing the shared rollout, either by replacing the solver or by using a same-source dependency stack, removes divergence and restores blind-last ordering while keeping forgiveness enabled, showing the instability, not the forgiveness rule, is the direct trigger.","Reference-conditioned forgiveness remains useful against genuinely bad references, but it needs rollout stability checks, since it has no built-in way to tell a shared numerical failure from a legitimate reference exception.","Any re-simulation score used for defensive-driving claims should report the full score basis: dependency versions, numerical backend, reference failure masks, overwrite rates, and fitted-state and rollout stability, plus route-blind and route-aware actor-blind probes."],"supporting_citations":[{"why":"Defines NAVSIM, the benchmark and scorer whose original-scene single-stage configuration is audited.","marker":"[12]"},{"why":"Introduces pseudo-simulation for autonomous driving, the re-simulation paradigm whose reference-conditioned scoring is under test.","marker":"[4]"},{"why":"Supplies the PDM-style metric family used for the PDM-Closed actor-aware comparator.","marker":"[11]"},{"why":"Defines nuPlan, the closed-loop benchmark used as the contrasting evaluation context in the related-work framing.","marker":"[2]"},{"why":"Provides the DiffusionDrive planner checkpoint used as an actor-aware comparator in the audit.","marker":"[23]"},{"why":"Provides the released LEAD planner configuration and checkpoint (LTFV6) rescored on the matched navtest intersection.","marker":"[25]"},{"why":"Shows that ego-status-only policies can appear competitive under open-loop metrics, motivating the actor-blind probe as a sanity check.","marker":"[22]"},{"why":"Asks whether open-loop pseudo-simulation scores predict closed-loop performance, the background for asking whether scores preserve actor-sensitive distinctions.","marker":"[33]"}],"fun_headline_variants":["Numerical glitch flips NAVSIM leaderboard: blind probes beat humans","Solver swap fixes NAVSIM scoring: blind probes no longer top","NAVSIM audit: shared rollout bug credits blind agents","Why Ignore-All outranks humans in NAVSIM: solver instability","Blind probes top NAVSIM due to shared-failure forgiveness collapse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the causal diagnosis from the 450-token navhard stage-1 control pool and the 32-token diagnostic set transfers to the full 12,146-token navtest split; the paper does not run the solver-replacement or same-source stack control on the full split, so an additional cause of the navtest anomaly cannot be excluded.","fun_headline_variants_meta":{"raw":{"variants":["Numerical glitch flips NAVSIM leaderboard: blind probes beat humans","Solver swap fixes NAVSIM scoring: blind probes no longer top","NAVSIM audit: shared rollout bug credits blind agents","Why Ignore-All outranks humans in NAVSIM: solver instability","Blind probes top NAVSIM due to shared-failure forgiveness collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000687,"raw_usage":{"total_tokens":3166,"prompt_tokens":1047,"completion_tokens":2119,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2026}},"tokens_in":663,"tokens_out":2119,"duration_ms":14557,"temperature":1.0,"reasoning_tokens":2026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:05:27.262975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the solver-replacement control and the same-source dependency stack control on the full 12,146-token navtest split with forgiveness enabled, and compare per-token rollouts and final EPDMS scores: if Ignore-All falls below human replay and PDM-Closed and the divergence rate drops to near zero, the instability is confirmed as the direct trigger, while if the blind win persists despite the stabilized shared refit, the causal conclusion is incomplete.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems (NeurIPS)37, 28706–28719 (2024)","cited_arxiv_id":null,"evidence_quote":"Defines NAVSIM, the benchmark and scorer whose original-scene single-stage configuration is audited."},{"cited_title":"In: Proceedings of the 9th Conference on Robot Learning","cited_arxiv_id":null,"evidence_quote":"Introduces pseudo-simulation for autonomous driving, the re-simulation paradigm whose reference-conditioned scoring is under test."},{"cited_title":"In: Conference on Robot Learning (CoRL)","cited_arxiv_id":null,"evidence_quote":"Supplies the PDM-style metric family used for the PDM-Closed actor-aware comparator."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Provides the DiffusionDrive planner checkpoint used as an actor-aware comparator in the audit."},{"cited_title":"In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)","cited_arxiv_id":null,"evidence_quote":"Provides the released LEAD planner configuration and checkpoint (LTFV6) rescored on the matched navtest intersection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that ego-status-only policies can appear competitive under open-loop metrics, motivating the actor-blind probe as a sanity check."}],"review_version":1}