{"id":"3c6b2856-f177-431b-b3ba-7b41b79818c4","arxiv_id":"2504.18010","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sky-Drive extends CARLA with distributed, synchronized multi-agent simulation, multi-sensor human-in-the-loop data collection, and bidirectional human-AI mentoring loops.","lead":"Sky-Drive is an open-source driving simulator built on CARLA that adds distributed multi-terminal control, multi-modal human-in-the-loop sensors, and human-AI collaboration modules. It is aimed at researchers studying how autonomous vehicles and human road users interact in mixed traffic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.3 ms distributed-simulation latency figure is asserted without measurement, and no in-platform synchronization or scalability data supports the multi-terminal architecture that the central claim depends on.","rationale":"The reader's weakest assumption correctly identifies the distributed architecture's unverified latency figure as the central risk. In good faith, this paper is a plausible systems description with advertised code and a sensible choice of CARLA as the base engine; it should not be rejected. However, the paper's own evidence does not support the key mechanism: the 0.3 ms figure in Section IV-A.2 is asserted without measurement conditions, the tables in Section V are taken from the authors' prior work rather than from Sky-Drive experiments, and the VR case study is described only qualitatively. The central claim that Sky-Drive enables synchronized multi-terminal human-AI collaboration therefore rests on an unvalidated premise. A single targeted measurement campaign—latency, synchronization error, and comparison with CARLA's native multi-client mode—would settle whether the concern lands. My recommendation is unchanged from the reader's CONDITIONAL verdict: the paper should be accepted only with revisions that supply in-platform validation of the distributed architecture and clearly separate implemented features from planned enhancements.","tokens_in":20424,"tokens_out":3429,"duration_ms":38957,"concrete_test":"Run the released Sky-Drive code with two terminals on a LAN switch, using the provided VR and steering-wheel clients, and measure: (i) round-trip RPC latency for at least 1,000 control messages; (ii) per-frame divergence in ego-vehicle position and velocity between host and client logs; and (iii) CARLA server tick time. Repeat with four terminals and with a VLAN connection approximating the UW-Madison to Purdue link. If median round-trip latency exceeds 0.3 ms by a substantial margin, or if client-observed state diverges by more than one CARLA tick, the central synchronization claim fails. Report the exact network hardware, CARLA version, and measurement timestamps so the result is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is Section IV-A.2's claim that Sky-Drive can maintain synchronized multi-terminal simulation at 'low latency of 0.3 milliseconds.' This number is presented with no measurement protocol, hardware configuration, number of terminals, payload size, or synchronization-error metric. Because the downstream claims—VR-based AV-HRU interaction, HAIM training, cross-campus VLAN simulation—all assume multiple clients observe and act on one consistent world state, an unverified latency figure leaves the central contribution unvalidated. The concern is not only that 0.3 ms is very low; it is that the paper never reports end-to-end control latency, CARLA tick alignment, or divergence between client-side states. Moreover, the only quantitative performance tables (Tables II–IV) are explicitly recycled from prior papers [9], [57], [63], not measured on Sky-Drive itself. CARLA already allows multiple RPC clients on a single server, so the paper must demonstrate what Sky-Drive's rpclib layer adds beyond native multi-client mode. Without an in-platform latency and synchronization measurement, the 'distributed multi-agent architecture' claim—and everything built on it—remains an assertion rather than a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Sky-Drive, an open-source simulation platform built on CARLA that claims four core innovations: (a) a distributed multi-agent architecture using RPC for synchronized simulation across terminals, (b) a multi-modal human-in-the-loop framework with VR, eye tracking, voice, facial expression, physiological, and steering inputs, (c) a human-AI collaboration mechanism with HAIM (human as AI mentor) and AIHM (AI as human mentor) modules, and (d) a digital twin framework for reconstructing real-world transportation scenes. The paper also describes planned foundation-model and hardware-in-the-loop integrations. Section V presents five use cases: VR-based AV-HRU interaction, HAIM-based deep RL, VLM-enabled RL, personalized safety-critical curriculum learning, and accident data replay. The platform is positioned as a bridge between scenario generation, data collection, algorithm training, and hardware integration for socially-aware autonomous driving research.","tokens_in":20698,"tokens_out":2559,"duration_ms":27679,"significance":"If the distributed architecture and human-in-the-loop capabilities work as described, Sky-Drive would be a useful open-source infrastructure for studying mixed-traffic interactions with real human participants across terminals, an area where existing simulators are weak. The paper's strengths are its broad and clearly organized system description, a plausible modular design, and the authors' prior work on HAIM-DRL, VLM-RL, and CurricuVLM, which provide partial evidence that the individual algorithmic components have value. The paper also transparently marks several components (foundation models, HIL, multi-location VLAN testing) as planned rather than demonstrated. The central weakness is that the platform's own performance and synchronization claims are not measured, and the reported quantitative validations are recycled from earlier papers rather than produced inside Sky-Drive; this leaves the paper's core contribution unvalidated as a systems contribution.","major_comments":[{"comment":"The claim that Sky-Drive achieves \"low latency of 0.3 milliseconds\" for real-time interactions is asserted without any measurement protocol. The paper does not report the number of terminals, network hardware, payload size, end-to-end control latency, CARLA tick alignment, or synchronization error between client-side states. Because the entire distributed multi-agent premise depends on consistent global state across terminals, this figure must be replaced with actual measurements: latency distribution, state divergence (e.g., position/velocity error between host and clients), and scaling behavior as the number of agents and terminals grows. As written, the central performance claim is unsupported.","section":"Section IV-A.2"},{"comment":"The quantitative validations for HAIM-DRL, VLM-RL, and CurricuVLM are explicitly recycled: the table notes state \"The results are based on data reported in [9]\" and similar for [57] and [63]. These tables therefore do not validate Sky-Drive itself; they only show that the authors' earlier algorithms perform well in their respective previous experimental settings. The manuscript needs at least one in-platform experiment that exercises Sky-Drive's distributed multi-terminal synchronization and human-in-the-loop data collection, reporting metrics such as synchronization error, interaction latency, and participant throughput. Without such an experiment, the claim that Sky-Drive \"enables\" these use cases remains an assertion.","section":"Section V, Tables II-IV"},{"comment":"CARLA already supports multiple RPC clients connecting to a single server. The paper does not clearly explain what Sky-Drive's rpclib-based extension adds beyond CARLA's native multi-client mode. To substantiate the novelty of the \"distributed multi-agent architecture,\" the authors should specify the synchronization mechanism (e.g., who owns the authoritative state, how client-side predictions are reconciled with server ticks, how conflicts are resolved) and describe a concrete scenario where CARLA's native mode fails and Sky-Drive succeeds. Without this, the distributed architecture claim is not differentiated from existing CARLA functionality.","section":"Section IV-A.1"},{"comment":"The VR-based AV-HRU interaction case study describes an experimental setup but reports no results: there are no participant counts, no gaze or reaction-time statistics, and no yielding or near-miss outcomes. As presented, this is a demonstration of a planned procedure, not a validation. The authors should either report results from a pilot study or explicitly label this subsection as an illustrative setup rather than an evaluation.","section":"Section V-A"}],"minor_comments":[{"comment":"The subsection title contains a typo: \"Real-time Monitoring Webstie\" should be \"Real-time Monitoring Website.\"","section":"Section IV-A.3"},{"comment":"The phrase \"A Vs\" is inconsistently spaced in the introduction (e.g., \"A Vs, HVs, and pedestrians\" vs. \"AVs\" elsewhere); please standardize the notation.","section":"Section I"},{"comment":"\"Sky-Drive utilize\" should be \"Sky-Drive utilizes\" for subject-verb agreement.","section":"Section IV-A.2"},{"comment":"The table note defines \"Distributed Multi-agent Simulation\" but the caption could also clarify that all other columns are binary capabilities; consider adding a legend for the checkmark/dash symbols.","section":"Table I"},{"comment":"Equation (3) is stated to be \"specifically designed as [9]\" but the parameters α, β, φ are not defined in the text; please provide their roles or at least a pointer to the original paper for these hyperparameters.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is better described as a vision-plus-system-design paper than a validated systems paper. The authors' prior algorithmic work is relevant but does not substitute for in-platform measurement. A revision that adds a proper latency/synchronization study and at least one in-platform human-in-the-loop demonstration would make the contribution significantly stronger; without such evidence, the distributed multi-agent architecture is not demonstrated beyond what CARLA already provides."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a systems paper, and it is an honest one up to a point. The distributed RPC-on-CARLA architecture, the multi-modal human-in-the-loop rig (VR, eye tracking, voice, Garmin, racing wheel), the HAIM/AIHM mentoring loops, and the Wisconsin Flex Lane digital twin pilot are real pieces of engineering, and the authors point to the code and a demo video. The writing is also unusually transparent about what is future work (foundation models, hardware-in-the-loop) and about the fact that the four 'use case' tables in Section V are taken from the authors' earlier papers—the notes under Tables II–IV say so explicitly.\n\nThe soft spot is the central quantitative claim. Section IV-A.2 says the LAN communication achieves 'low latency of 0.3 milliseconds' with no measurement protocol, no terminal count, no payload size, and no synchronization-error metric. That number reads like an RPC round-trip on an idle connection, not an end-to-end distributed simulation latency. Because the distributed architecture is the main thing that distinguishes Sky-Drive from CARLA's native multi-client mode, this is not a minor omission—every downstream claim (VR pedestrian experiments, cross-campus VLAN, HAIM training) depends on consistent multi-terminal state. And the validation sections do not actually validate Sky-Drive. Tables II–IV are results from HAIM-DRL, VLM-RL, and CurricuVLM as reported in prior papers; the text says these were 'implemented and evaluated within the Sky-Drive platform,' but this manuscript contains no in-platform measurements of what Sky-Drive contributes. The one study that appears genuinely in-platform, the VR pedestrian-vehicle interaction, is described only qualitatively.\n\nI would still send this to peer review rather than desk reject. The architecture is coherent, the code is released, and the community needs an open, distributed, human-in-the-loop testbed for rare AV-HRU interactions. The paper would be much stronger with a measured latency and state-synchronization study compared against CARLA's native multi-client mode, at least one quantitative end-to-end experiment that runs inside Sky-Drive, and language that separates plans from demonstrated results. If you work on AV-human interaction or socially-aware driving simulation, this is a useful integration map and a pointer to a codebase; if you are looking for evidence that the platform meets its performance claims, you will not find it yet. For peer review: major revision, with the expectation that the empirical gap gets closed.","headline":"A plausible, honestly-written distributed simulation platform paper whose central latency claim is unmeasured and whose validation tables are recycled from prior work—worth serious revision, not a desk reject.","tokens_in":21251,"tokens_out":3116,"would_cite":false,"duration_ms":30316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sky-Drive claims that a distributed, multimodal, human-in-the-loop simulation platform can turn mixed-traffic human-AI interaction research into a closed loop spanning scenario generation, data collection, algorithm training, and hardware…","keywords":["Driving Simulator","Autonomous Vehicles","Human-AI Collaboration","Multi-Agent Simulation","Digital Twin","Socially-Aware Driving","Human-in-the-Loop","Distributed Simulation"],"falsifier":"Run a controlled scaling experiment in Sky-Drive: start with a host plus one client on a single switch, record end-to-end position divergence between the host's ground-truth agent states and each client's rendered states, then add clients (2, 4, 8) and repeat over Wi-Fi, VLAN, and across-site links. If mean position error grows beyond a small fraction of a vehicle length (e.g., >0.5 m) or per-step latency exceeds the simulation tick budget at any realistic configuration, the synchronized-distributed-simulation claim fails as stated; publishing the 0.3-ms figure with its exact conditions would also settle whether the reported latency is meaningful.","tokens_in":20239,"feed_emoji":"🚗","tokens_out":8310,"duration_ms":71756,"temperature":0.7,"pith_summary":"Sky-Drive is an open-source simulation platform whose central claim is that autonomous-driving research needs to study mixed traffic—autonomous vehicles interacting with human drivers, pedestrians, and cyclists—inside a shared virtual world, and that this requires more than a single-machine simulator. The paper builds a distributed multi-agent architecture on top of a widely used open urban driving simulator so that different agents can be controlled from separate terminals while the world state stays synchronized. It adds a multi-modal human-in-the-loop layer that records gaze, voice, facial expression, heart-rate signals, and steering inputs, plus a two-way human-AI mentoring mechanism in which humans shape AI policies through takeovers and feedback while AI coaches human drivers. If these pieces work as described, researchers gain a reusable environment for collecting safety-critical interaction data, training socially aware driving policies, replaying real accidents, and eventually validating algorithms on physical vehicles. The paper demonstrates the platform on several training methods and reports performance tables for each.","feed_headline":"Distributed simulator syncs humans and AVs in one traffic scene","feed_subtitle":"Sky-Drive's distributed simulator puts human drivers and AVs in one shared scene, enabling mixed-traffic studies.","key_machinery":"The load-bearing object is the RPC-over-simulator distributed layer: a host-client setup in which one terminal runs the global environment and each client terminal injects control for its own agent, with dual-port TCP over LAN/VLAN keeping states in sync and a web-service log for all streams. A second mechanism is the bidirectional mentoring loop formed by HAIM and AIHM: HAIM converts human takeovers and multimodal feedback into preference comparisons that shape a reward-free reinforcement-learning objective, while AIHM generates reference trajectories and adjusts scenario difficulty from driver performance and physiological state. A third is the digital twin pipeline, which fuses multi-source real-world data into virtual scenes and replays accident trajectories extracted by an object tracker. These mechanisms together are what allow scenario generation, data collection, algorithm training, and hardware integration to close into one loop.","core_discovery":"Sky-Drive's claim is that a driving simulator can be turned into a distributed multi-terminal research instrument rather than a single-process benchmark. The platform extends an established open urban driving simulator with a remote-procedure-call (RPC) distributed layer in which one host terminal maintains the global scene and client terminals independently control autonomous vehicles, human-driven vehicles, and pedestrians; a dual-port TCP system over LAN or VLAN keeps the terminals synchronized, and a real-time web monitor streams positions, velocities, video, and sensor data to a central log. On top of this shared world, Sky-Drive layers a multi-modal human-in-the-loop framework (a VR headset with eye tracking, voice transcription, an in-cabin camera, a smartwatch for physiological signals, and a racing wheel) and a human-AI collaboration mechanism with two modules: the human-as-AI-mentor module, which treats takeovers and preference feedback as implicit policy corrections in a reward-free reinforcement-learning objective, and the AI-as-human-mentor module, which uses physics-enhanced residual learning to generate reference trajectories and personalized training scenarios. A digital twin framework ingests high-definition maps, traffic-camera feeds, and historical records to reconstruct real environments, and an accident replay pipeline tracks objects from real crash videos and replays them for analysis. The reported experiments place Sky-Drive as the substrate for human-feedback RL, vision-language-model-reward RL, personalized curriculum learning, and accident replay.","pith_inferences":["If the distributed synchronization holds at scale, Sky-Drive's architecture would enable multi-site human-subject studies where participants in different cities interact in the same traffic scene, which could expose regional differences in yielding norms and right-of-way conventions—something single-site simulators cannot measure.","The 0.3-millisecond latency claim is almost certainly a best-case single-switch figure; a natural extension is a public benchmark reporting state divergence and latency as functions of terminal count, agent count, and network path, without which the distributed claim lacks a quantitative envelope.","The preference-from-takeover mechanism generalizes beyond driving: the same HAIM/AIHM loop could turn Sky-Drive into a general-purpose human-AI alignment testbed for teleoperated robots, delivery drones, or assistive mobility devices, where corrective interventions are cheap to collect.","The accident replay pipeline, combined with a counterfactual AV control policy, suggests a concrete safety-validation workflow: replay real crashes, swap in a candidate AV policy, and measure whether the collision is avoided—an evaluation not possible from static footage."],"forward_implications":["Mixed-traffic interaction studies can move from single-machine rule-based traffic to live human participants controlling AVs, HVs, and pedestrians from separate terminals in one synchronized world.","Human takeovers become training signal: the HAIM formulation lets an AV policy be refined toward human preferences with roughly 60 times fewer samples than PPO (8,000 versus 500,000 in the reported comparison).","AIHM turns the same platform into a driver coach: real-time reference trajectories, annotated replays, and difficulty-adjusted scenario generation are driven by measured performance and physiological state.","Real-world accident footage can be replayed in simulation via tracked object trajectories, providing repeatable safety-critical scenarios for training and post-incident analysis.","The digital twin pipeline connects live traffic data feeds to the simulator, allowing scenario generation and evaluation to track a real road segment's current traffic state."],"supporting_citations":[{"why":"Prior human-as-AI-mentor reinforcement-learning method, used as a proof-of-concept for Sky-Drive's human feedback pipeline.","marker":"[9]"},{"why":"The open urban driving simulator that serves as the core engine Sky-Drive extends with distributed control.","marker":"[10]"},{"why":"Intelligent-driver car-following model, cited as domain knowledge that shapes the human-AI mentoring priors.","marker":"[31]"},{"why":"Lane-changing model cited alongside it as an encoded rule of human driving behavior.","marker":"[32]"},{"why":"Speech recognition model used to transcribe voice commands before language-model interpretation.","marker":"[56]"},{"why":"Vision-language-model reward RL method used to demonstrate semantic reward training inside the platform.","marker":"[57]"},{"why":"Personalized safety-critical curriculum learning method used to validate adaptive scenario generation.","marker":"[63]"},{"why":"Multi-object tracking method used to extract trajectories from accident video for replay.","marker":"[64]"}],"fun_headline_variants":["Sky-Drive: distributed sim for human-AI driving research","Multi-agent simulator merges human and autonomous driving","Human-AI collaboration on a distributed traffic simulator","Digital twin driving sim with human feedback for AVs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that the RPC-based host-client setup can keep the global simulation state synchronized across terminals at interactive latency; the paper reports a 0.3-ms low-latency figure without stating the measurement conditions, number of terminals, or network hardware, so if state consistency degrades as agents or sites are added, the distributed, human-in-the-loop, and collaborative-training claims all rest on unverified ground.","fun_headline_variants_meta":{"raw":{"variants":["Sky-Drive: distributed sim for human-AI driving research","Multi-agent simulator merges human and autonomous driving","Human-AI collaboration on a distributed traffic simulator","Digital twin driving sim with human feedback for AVs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3113,"prompt_tokens":1086,"completion_tokens":2027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":1964}},"tokens_in":702,"tokens_out":2027,"duration_ms":16104,"temperature":1.0,"reasoning_tokens":1964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:27:19.516433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled scaling experiment in Sky-Drive: start with a host plus one client on a single switch, record end-to-end position divergence between the host's ground-truth agent states and each client's rendered states, then add clients (2, 4, 8) and repeat over Wi-Fi, VLAN, and across-site links. If mean position error grows beyond a small fraction of a vehicle length (e.g., >0.5 m) or per-step latency exceeds the simulation tick budget at any realistic configuration, the synchronized-distributed-simulation claim fails as stated; publishing the 0.3-ms figure with its exact conditions would also settle whether the reported latency is meaningful.","supporting_citations":[{"cited_title":"Human as ai mentor: En- hanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Prior human-as-AI-mentor reinforcement-learning method, used as a proof-of-concept for Sky-Drive's human feedback pipeline."},{"cited_title":"Carla: An open urban driving simulator,","cited_arxiv_id":null,"evidence_quote":"The open urban driving simulator that serves as the core engine Sky-Drive extends with distributed control."},{"cited_title":"Congested traffic states in empirical observations and microscopic simulations,","cited_arxiv_id":null,"evidence_quote":"Intelligent-driver car-following model, cited as domain knowledge that shapes the human-AI mentoring priors."},{"cited_title":"General lane-changing model mobil for car-following models,","cited_arxiv_id":null,"evidence_quote":"Lane-changing model cited alongside it as an encoded rule of human driving behavior."},{"cited_title":"Robust speech recognition via large-scale weak super- vision,","cited_arxiv_id":null,"evidence_quote":"Speech recognition model used to transcribe voice commands before language-model interpretation."},{"cited_title":"Tracking objects as points,","cited_arxiv_id":null,"evidence_quote":"Multi-object tracking method used to extract trajectories from accident video for replay."}],"review_version":1}