{"id":"8b91b657-723e-4cb7-87f8-78b0d782593f","arxiv_id":"2608.07621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CMU-Drive adds up to 16 connected autonomous vehicles to closed-loop driving scenarios, and V2V-VLA shows that sharing merged occupancy views and communication suggestions improves driving score over a single-agent VLA baseline.","lead":"This paper introduces CMU-Drive, a driving simulator benchmark where 2 to 16 connected self-driving cars share the same safety-critical traffic scenarios, and V2V-VLA, a model that lets those cars exchange merged camera views and communication hints within a single neural network. It reports better route completion and fewer infractions than a single-agent baseline, while claiming the evaluation needs only one GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's advantage for V2V-VLA is confounded: the model is additionally fine-tuned on 180K PDM-lite expert samples with a merged-occupancy input, while the SimLingo baseline is not, so the DS/SR gap may not be attributable to cooperation.","rationale":"The reader's weakest assumption is that PDM-lite oracle supervision provides safe, generalizable labels; I partially agree. The more immediate load-bearing issue is that Table 2 cannot support the causal claim 'cooperation helps' because the two compared methods differ in extra fine-tuning data, merged occupancy input, and communication policy simultaneously. V2V-VLA is initialized from Simlingo and then trained on 180K CMU-Drive samples, while the baseline appears to be the unmodified Simlingo model. Any DS/SR gap could come from continued training on a large expert-labeled dataset rather than from V2V communication. This concern is load-bearing because the abstract and conclusion explicitly claim the model's superiority, and the benchmark's value as a cooperative VLA baseline depends on an interpretable comparison. I do not see an internal inconsistency in the method; the weakness is the uncontrolled comparison. The proposed ego-only ablation would isolate the cooperation component. Since the reader's CONDITIONAL verdict already reflects the need for stronger evidence, my read does not change that verdict.","tokens_in":8393,"tokens_out":5035,"duration_ms":52959,"concrete_test":"Re-run the 220 CMU-Drive evaluation with an ego-only ablation of V2V-VLA: the same architecture and the same 180K training samples, but with the merged occupancy branch disabled (only the ego vehicle's camera image and ego BEV occupancy) and the communication mechanism disabled. If this ablated model matches the reported DS 63.67 within plausible run-to-run variation, the Table 2 advantage is not due to cooperation; if it drops substantially, the merged-occupancy or communication branch is the source.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on Table 2, where V2V-VLA (DS 63.67, SR 34.55%) is compared against 'Simlingo' (DS 56.32, SR 30.91%). Section 3.4 describes collecting 180K training samples using PDM-lite as expert; PDM-lite uses ground-truth states of all objects, and the data include rule-based language annotations, future waypoints, and communication supervision. Section 3.5 states V2V-VLA is initialized from Simlingo and LoRA-fine-tuned on these samples with the merged-occupancy branch. No ablation removes the merged occupancy/communication branch, removes the extra 180K samples, or fine-tunes Simlingo on the same data without the V2V branch. Therefore the observed gain conflates at least three factors: (i) additional supervised fine-tuning on CMU-Drive data distilled from an oracle expert, (ii) the extra merged BEV occupancy input, and (iii) the language-guided communication policy. The stated claim that V2V-VLA achieves a superior driving score is not yet established because none of these factors is isolated. This is an experimental-design issue, not an internal contradiction, and it is directly testable by a controlled ablation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CMU-Drive, a closed-loop cooperative multi-agent driving benchmark that extends Bench2Drive by simulating 2 to 16 connected autonomous vehicles (CAVs) sharing the same CARLA environment, with 220 evaluation routes across 44 scenario types. The authors also propose V2V-VLA, a vision-language-action model that, in a single forward pass, generates driving waypoints, language reasoning, and a communication policy, using an ego camera image plus a merged BEV occupancy map from a selected neighboring CAV. The model is initialized from SimLingo and LoRA-fine-tuned on 180K training samples collected with the PDM-lite expert. The only quantitative comparison, reported in Table 2, shows V2V-VLA achieving a higher driving score (63.67 vs. 56.32) and success rate (34.55% vs. 30.91%) than the unmodified SimLingo baseline. The paper also includes one qualitative example illustrating a rain-blurred bicycle detected by a leading CAV via the proposed communication mechanism.","tokens_in":8684,"tokens_out":5072,"duration_ms":50158,"significance":"If fully validated, CMU-Drive would be a useful addition to the cooperative-driving benchmark landscape: it supports a substantially larger number of cooperating agents (up to 16) than InterDrive or MDrive, retains safety-critical scenario diversity from Bench2Drive, and claims single-GPU evaluation feasibility. The V2V-VLA model is a reasonable first baseline for this benchmark, and the authors commit to releasing code, benchmark, and checkpoints, which is a strength. However, the experimental support for the model's central claim is currently thin: the sole comparison in Table 2 is confounded, no ablations isolate the contribution of cooperation, and no statistical repeatability is reported. The benchmark itself would be more convincing with a breakdown by scenario type or by number of CAVs, and with a clearer demonstration that the proposed cooperative mechanism (not merely additional fine-tuning) is responsible for the reported gains.","major_comments":[{"comment":"The comparison between V2V-VLA and SimLingo is confounded by at least three simultaneous differences: (i) V2V-VLA is LoRA-fine-tuned on 180K CMU-Drive samples distilled from the PDM-lite oracle, while SimLingo is used without any fine-tuning; (ii) V2V-VLA receives an extra merged BEV occupancy map from a neighboring CAV; and (iii) V2V-VLA is trained to emit a communication policy, while SimLingo is not. Because none of these factors is isolated, the observed DS/SR gains cannot be attributed to cooperation. Please add ablations, at minimum (a) SimLingo fine-tuned on the same 180K samples without the V2V branch, and (b) V2V-VLA with the merged-occupancy input removed or masked at inference. Without such a controlled comparison, the concluding claim in §5 that V2V-VLA achieves a superior driving score and success rate is not established.","section":"§4.1, Table 2"},{"comment":"The learned communication policy is trained using annotations derived from ground-truth visibility and oracle knowledge of which CAV observes the critical object, yet the paper does not analyze how the policy behaves in closed-loop evaluation. Specifically, the paper does not report how often the model selects a non-default communicating CAV, whether those selections are correct, or what fraction of the driving-score improvement comes from the default closest-front-CAV communication versus the learned target switching. Since §3.1 states the model can override the default CAV, an evaluation of the communication decision accuracy and its direct effect on driving outcomes is needed to validate the reasoning-assisted communication component.","section":"§3.1 and §3.4"},{"comment":"All quantitative results are reported as single point estimates with no standard deviations, no confidence intervals, and no repeated evaluation seeds. In closed-loop CARLA evaluation, route initialization and nondeterministic traffic behavior can produce run-to-run variance, and the reported gaps (63.67 vs. 56.32 DS; 34.55% vs. 30.91% SR) may be within the noise range. Please either fix the random seeds and state this explicitly for all routes, or report aggregated results over multiple seeds so that the reader can judge the reliability of the difference.","section":"§2.2 and Table 2"},{"comment":"The training data are generated by PDM-lite, an expert that uses ground-truth locations and velocities of all objects 'regardless of whether objects are visible or invisible.' The communication supervision is therefore based on a perfect visibility model, whereas the deployed V2V-VLA perceives the world through its own camera and learned BEV occupancy. This creates a distribution shift: the model is trained to output communication requests when an object is invisible in ground truth, but at inference it must decide this from its own imperfect perception. The paper does not analyze how often the model's perceived visibility matches the ground-truth annotation, nor the consequences of misalignment for closed-loop safety. Please provide at least a quantitative comparison of ground-truth communication labels versus model-predicted communication targets on a held-out set, or explicitly discuss this limitation and its potential impact on the reported results.","section":"§3.4"}],"minor_comments":[{"comment":"The section heading '3.3 Training' is followed immediately by '3.4 Training Data Collection' with no text under 3.3; this heading should either be removed or filled with a brief description of the training objective and procedure.","section":"§3.3"},{"comment":"The formatting in the CMU-Drive row of Table 1 is mangled: the entries for minimum, maximum, and average number of CAVs appear to be run together as '220 216 6.56 1'. Please present these values as separate columns with clear headers (Min. # CAVs, Max. # CAVs, Avg. # CAVs), as done for the other rows.","section":"Table 1"},{"comment":"In the text following Eq. (8), the word 'warpped' should be 'warped', and the notation for poses P_ego and P_c is used without a formal definition; please state that each pose includes position and orientation in a common coordinate frame.","section":"§3.2"},{"comment":"Figure 3 is mentioned in §4.1 as showing the average driving score per scenario type, but the figure is not displayed in the manuscript text provided. Please ensure all figures are included in the submission and that any color/legend information is legible.","section":"Fig. 3"},{"comment":"Reference [1] for PDM-lite points to a personal GitHub report; if a formal publication or arXiv record exists, citing it would be more appropriate and verifiable.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark contribution is potentially valuable, but the empirical validation in its current form is too weak for the central claim. The authors should be asked to provide a proper ablation hierarchy (fine-tuned single-agent baseline, V2V-VLA without communication input, and with communication input but oracle-only target selection) and to report metrics over multiple seeds. I also note that the manuscript cites several works dated 2026, which is consistent with the arXiv posting date but may be worth verifying during copy editing. The scope of the paper seems more appropriate for a workshop or a short conference paper than a full journal contribution in its current depth; however, the release of code and checkpoints would strengthen reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: CMU-Drive is a useful benchmark and worth a serious look; the V2V-VLA's reported advantage over SimLingo is not yet established by the experiments as written.\n\nThe genuinely new pieces are the multi-agent extension of Bench2Drive's 220 routes to 2-16 CAVs, the route-level product metrics (RC, IS averaged/product), and a VLA model that emits a communication target in the same forward pass as actions and reasoning. The single-GPU evaluation protocol is a practical contribution; prior cooperative benchmarks need three GPUs per route. The benchmark construction is clear, the scenario inheritance from Bench2Drive is sensible, and the metric definitions are precise. Credit is due for building on an existing widely-used benchmark rather than inventing a new simulator.\n\nWhere the paper is soft is exactly where the stress-test note lands. Table 2 compares V2V-VLA (DS 63.67, SR 34.55%) against SimLingo (DS 56.32, SR 30.91%). But V2V-VLA is initialized from SimLingo and LoRA-fine-tuned on 180K CMU-Drive samples distilled from PDM-lite, an oracle with access to ground-truth states of all objects. The baseline is not fine-tuned on that data, has no merged-occupancy input, and no communication branch. So the gap conflates at least three explanations: extra supervised fine-tuning on CMU-Drive data, the extra occupancy input, and the learned communication policy. No ablation or seed repeats, no error bars. A reader cannot tell which component drives the improvement. The qualitative example is suggestive but not proof.\n\nA secondary concern is the oracle supervision: PDM-lite uses ground-truth locations and velocities of all objects, so the trained model may be learning to imitate an omniscient expert rather than to cooperate under partial observability. The communication annotations are rule-based and derived from the same ground truth, so they are not evidence of emergent communication. None of this is fatal to the benchmark, but it does weaken the central claim made in Section 4.1.\n\nWho benefits: researchers working on cooperative end-to-end driving will likely find CMU-Drive a practical evaluation tool, and the route-extension methodology is reusable. The paper deserves a serious referee: the benchmark is solid enough that the community should see it with fixes rather than have it desk-rejected. My recommendation: send it to review, and make the referee insist on a proper ablation that fine-tunes SimLingo on the same 180K samples without the V2V branch, plus seed repeats and release of code/data.","headline":"A genuinely useful cooperative driving benchmark held back by a confounded model comparison; referee should ask for an ablation.","tokens_in":9160,"tokens_out":2264,"would_cite":true,"duration_ms":18743,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sharing a nearby car's view improves driving scores in simulation","keywords":["cooperative autonomous driving","vision-language-action","vehicle-to-vehicle communication","BEV occupancy map","closed-loop benchmark","multi-agent driving","language reasoning","end-to-end driving"],"falsifier":"Run V2V-VLA on the 220 CMU-Drive routes with the communication channel disabled, for example by setting $O_{\\mathrm{merged}}=O_{\\mathrm{ego}}$ while keeping every other input identical; if the driving score and success rate do not drop below those of the baseline, the central claim that cooperative perception improves closed-loop driving would be refuted. A complementary check would retrain the model on the same data but without the merged occupancy map and compare the two.","tokens_in":8217,"feed_emoji":"🚗","tokens_out":7560,"duration_ms":60542,"temperature":0.7,"pith_summary":"CMU-Drive is a closed-loop benchmark that extends single-agent urban driving scenarios to cooperative multi-agent settings, with 220 evaluation routes of 2 to 16 connected autonomous vehicles interacting with background vehicles, pedestrians, and cyclists across 44 safety-critical scenario types. The paper also proposes V2V-VLA, a vision-language-action model that, in one forward pass, generates driving actions, future waypoints, language reasoning, and a communication policy. V2V-VLA merges the ego vehicle's bird's-eye-view occupancy map with a selected neighboring vehicle's occupancy map, transformed into the ego frame, and uses this fused perception for closed-loop control. On CMU-Drive, V2V-VLA outperforms a non-cooperative VLA baseline, reaching a driving score of 63.67 and a success rate of 34.55 percent, versus 56.32 and 30.91 percent for the baseline. If correct, these results establish the first benchmark and baseline for cooperative vision-language-action driving and suggest that learned vehicle-to-vehicle communication can improve closed-loop multi-agent driving in safety-critical scenarios.","feed_headline":"Sharing a nearby car's view improves driving scores in simulation","feed_subtitle":"V2V-VLA beats a single-vehicle baseline on all 220 closed-loop routes, with up to 16 cooperative cars.","key_machinery":"The load-bearing mechanism is the merged bird's-eye-view occupancy map, $O_{\\mathrm{merged}} = \\mathrm{ElementwiseOr}(O_{\\mathrm{ego}}, \\mathrm{CoordinateTransform}(O_c, P_{\\mathrm{ego}}, P_c))$, where $O_{\\mathrm{ego}}$ and $O_c$ are camera-based BEV occupancy maps generated by UniAD, and the communicating vehicle's map is warped into the ego frame using both poses. The model also outputs a communication policy in language, recommending a different cooperative vehicle as the next communication target when the current one cannot see a critical object. These two pieces let the VLA model jointly reason about perception, action, and communication in a single forward pass, rather than relying on multi-round negotiation.","core_discovery":"The central claim is that V2V-VLA achieves a superior driving score and a higher success rate than the non-cooperative baseline on the CMU-Drive benchmark. The paper argues that cooperative perception—merging the ego vehicle's BEV occupancy map $O_{\\mathrm{ego}}$ with the coordinate-transformed occupancy map $O_{c\\to\\mathrm{ego}}$ of a chosen communicating vehicle via element-wise OR—plus an explicit communication policy that selects which neighbor to query next, is what delivers the improvement. This makes V2V-VLA the first cooperative vision-language-action driving model to be evaluated in closed loop on a multi-agent benchmark with up to 16 connected autonomous vehicles. The authors present the benchmark and this baseline as a foundation for future research on reasoning-guided, cooperative, end-to-end autonomous driving.","pith_inferences":["The expert used for data collection, PDM-lite, has access to ground-truth positions of all objects; the reported gap might therefore partly reflect the model imitating an oracle that sees through occlusions, rather than learning a generalizable cooperative skill.","A natural stress test would disable the communication channel entirely (e.g., set the merged map to the ego map) and measure the drop in driving score; the paper does not report an ablation of this kind.","The single-communicating-vehicle design could be extended to selecting among multiple views or to fusing temporal information, which might close the gap on the hardest scenario types reported (e.g., NonSignalizedJunctionLeftTurnEnterFlow).","Testing on CARLA routes where the communicating vehicle's camera is deliberately corrupted would verify that the model relies on the shared BEV map rather than spurious cues."],"forward_implications":["On CMU-Drive, switching from single-agent VLA driving to cooperative V2V-VLA raises driving score from 56.32 to 63.67 and success rate from 30.91 percent to 34.55 percent across all 220 routes.","The single-GPU evaluation protocol makes it practical to benchmark cooperative autonomous driving with 2 to 16 agents in closed loop, including scenarios with emergency vehicles and crossing pedestrians.","The model's language output provides interpretable explanations of when and why a neighbor is queried, which could support human oversight of cooperative decisions.","Because the benchmark and checkpoint will be released, future methods can be compared directly against the first cooperative VLA baseline without reimplementing the simulation harness.","The results suggest that fusing a neighbor's BEV occupancy, not just raw images, is sufficient to recover information about occluded critical objects and improve closed-loop safety."],"supporting_citations":[{"why":"provides the single-agent VLA baseline, initialization weights, and the action/output format that V2V-VLA extends to cooperation.","marker":"[18]"},{"why":"supplies the 44 urban scenario types and the single-agent closed-loop evaluation protocol that CMU-Drive generalizes to multiple agents.","marker":"[13]"},{"why":"the PDM-lite expert that generates ground-truth waypoints for the 180K training samples using full ground-truth object states.","marker":"[1]"},{"why":"UniAD generates the camera-based BEV occupancy maps for both ego and communicating vehicles.","marker":"[11]"},{"why":"InternViT serves as the vision encoder for the front-camera image and the merged occupancy map.","marker":"[2]"},{"why":"Qwen2 is the LLM backbone that autoregressively produces language output and waypoint tokens.","marker":"[25]"},{"why":"CARLA supplies the closed-loop simulation environment for both benchmark creation and evaluation.","marker":"[7]"},{"why":"DriveLM's rule-based approach is adapted to annotate the language reasoning and communication supervision in the training data.","marker":"[19]"}],"fun_headline_variants":["V2V-VLA beats solo driving on all 220 closed-loop routes","Cooperative VLA driving: sharing views wins every simulation","First closed-loop multi-agent VLA driving benchmark","Connected cars outperform single vehicle in all tests","V2V-VLA: cooperative perception boosts driving scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training data are collected by imitating PDM-lite, an expert that uses ground-truth location and velocity information for all objects in the simulation, whether visible or not, so the model's apparent advantage could come from imitating an oracle rather than from true cooperative reasoning.","fun_headline_variants_meta":{"raw":{"variants":["V2V-VLA beats solo driving on all 220 closed-loop routes","Cooperative VLA driving: sharing views wins every simulation","First closed-loop multi-agent VLA driving benchmark","Connected cars outperform single vehicle in all tests","V2V-VLA: cooperative perception boosts driving scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1201,"prompt_tokens":899,"completion_tokens":302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":515,"tokens_out":302,"duration_ms":3356,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:28:52.548925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run V2V-VLA on the 220 CMU-Drive routes with the communication channel disabled, for example by setting $O_{\\mathrm{merged}}=O_{\\mathrm{ego}}$ while keeping every other input identical; if the driving score and success rate do not drop below those of the baseline, the central claim that cooperative perception improves closed-loop driving would be refuted. A complementary check would retrain the model on the same data but without the merged occupancy map and compare the two.","supporting_citations":[{"cited_title":"In: Conference on Computer Vision and Pattern Recognition (CVPR) (2025)","cited_arxiv_id":null,"evidence_quote":"provides the single-agent VLA baseline, initialization weights, and the action/output format that V2V-VLA extends to cooperation."},{"cited_title":"In: NeurIPS 2024 Datasets and Benchmarks Track (2024)","cited_arxiv_id":null,"evidence_quote":"supplies the 44 urban scenario types and the single-agent closed-loop evaluation protocol that CMU-Drive generalizes to multiple agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the PDM-lite expert that generates ground-truth waypoints for the 180K training samples using full ground-truth object states."},{"cited_title":"In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)","cited_arxiv_id":null,"evidence_quote":"UniAD generates the camera-based BEV occupancy maps for both ego and communicating vehicles."},{"cited_title":"In: Conference on Robot Learning (CoRL) (2017)","cited_arxiv_id":null,"evidence_quote":"CARLA supplies the closed-loop simulation environment for both benchmark creation and evaluation."},{"cited_title":"In: Europian Conference on Computer Vision (ECCV) (2024)","cited_arxiv_id":null,"evidence_quote":"DriveLM's rule-based approach is adapted to annotate the language reasoning and communication supervision in the training data."}],"review_version":1}