{"id":"081ef7da-ca0e-4c29-bfef-33f75e541753","arxiv_id":"2411.10603","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A closed-loop study in CARLA/LimSim++ evaluating a GPT-4o prompt-based driving agent across five weather conditions and several camera/LiDAR configurations.","lead":"This paper puts GPT-4o in the driver's seat of a simulated car and measures how safely it drives in heavy rain, storms, fog, and wet roads. It is a benchmark study, not a new invention, and the results are mostly about which camera and LiDAR combinations work best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on undefined or self-contradictory metrics: comfort sub-scores in Eq. (2) are never defined, speed-score Eq. (4) contradicts its prose, and Eq. (3) leaves the efficiency target ambiguous.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: the 'high safety and efficiency' headline is a statement about scores, not raw behavior. Collision counts, infraction rates, and smoothness measurements are never reported; the only evidence is CDFs of custom scores. If those scores are not reproducible from the text, the paper's empirical contribution is unverifiable. I checked the manuscript for definitions of the four comfort components; they appear only as names in Eq. (2). The speed-score equation and its prose cannot both be true; one of them must be wrong, or the equation is an aggregate while the prose is per-frame, but the paper never says which. This is not a disagreement with a baseline or a modeling choice; it is an internal inconsistency in the evaluation instrument. The GitHub link [22] is real independent support, and the test is to see whether the code supplies what the paper omits. If it does and reproduces the figures, the concern is resolved and the paper becomes a more useful benchmark; as written, REJECT is justified. I do not accuse the authors of anything; the issue is that the claims cannot be checked from the presented text.","tokens_in":12286,"tokens_out":6069,"duration_ms":60705,"concrete_test":"Download the linked repository [22], locate the scoring implementation, and verify whether it defines acc_score, jerk_score, lat_acc_score, lat_jerk_score, and the speed-score computation; then recompute the CDFs for one configuration (e.g., Heavy Rain, 3 cameras) and compare against Figs. 4a-4d. If the repository's formulas differ from Eqs. (2)-(4), or if the four comfort sub-scores are absent, the headline safety/comfort/efficiency claims are not supported by the paper text. If the code supplies the missing definitions and reproduces the figures, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion—that MLLM-AD-4o maintains high safety and efficiency and that camera/LiDAR configurations change performance—is expressed entirely through four scores defined in Section V-A. For that conclusion to be computable, the scores must be defined. Section V-A does not do that. Eq. (2) averages 'acc score', 'jerk score', 'lat acc score', and 'lat jerk score', but none of these are defined anywhere in the manuscript; the CDFs in Figs. 5 and 6 therefore cannot be recomputed from the text. Eq. (4) contradicts its own prose: the text says the speed score is 0.9 when exceeding the speed limit and 1 otherwise, while the equation gives 0.9^(fraction_of_speeding_frames × 10), an aggregate exponential penalty; the two agree only at exactly 10% speeding frames. Eq. (3) also leaves v* ambiguous: it is 'vavg or vlimit' depending on traffic density, but the paper never states how sparse vs. regular traffic is determined or how vavg is computed, so efficiency CDFs are not comparable across runs unless this is fixed. The safety score requires a threshold τth that is never specified. Since every headline result is a statement about these scores, the central claim is unsupported until the metrics are precisely defined and implemented. The linked GitHub repository could resolve this, but the manuscript alone does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLLM-AD-4o, a prompt-engineered autonomous driving agent based on GPT-4o integrated into the LimSim++/CARLA closed-loop simulator. The authors extend LimSim++ with six-camera support, a weather-management function, and semantic LiDAR integration, and they evaluate the agent under five weather conditions and several sensor configurations. The headline claims are that the agent maintains high safety and efficiency across adverse weather and that activating front and rear cameras (and, selectively, LiDAR) changes performance. The evaluation uses CDFs of safety, comfort, efficiency, and speed scores defined in Section V-A.","tokens_in":12589,"tokens_out":4670,"duration_ms":42474,"significance":"If the evaluation were fully specified, the paper would be a useful empirical data point on the feasibility of using a general-purpose MLLM for closed-loop driving control under adverse weather. The authors contribute an open-source modification of the LimSim++ platform, which supports reproducibility. The central quantitative claim, however, is currently unsupported because the metrics used to compute every reported score are incomplete or contradictory, and because no baseline or absolute threshold is provided to justify the phrase 'high levels of safety and efficiency.' The manuscript's conclusions about sensor configurations are qualitative and not backed by statistical tests.","major_comments":[{"comment":"The four performance scores that underlie every figure and conclusion are not adequately defined. The comfort score in Eq. (2) is the average of 'acc score', 'jerk score', 'lat acc score', and 'lat jerk score', but none of these sub-scores is defined anywhere in the manuscript. The safety score in Eq. (1) requires a threshold τ_th that is never specified. The efficiency score in Eq. (3) depends on v* ∈ {v_avg, v_limit}, but the paper does not state how v_avg is computed, how sparse versus regular traffic is determined, or how the choice between the two targets is made. The speed score's prose states it is '0.9 when the AV exceeds the speed limit and 1 otherwise', which contradicts Eq. (4), an aggregate exponential penalty over the fraction of speeding frames; the two only coincide at exactly 10% speeding frames. Because every headline result is a statement about these scores, the central claim is currently not reproducible or interpretable.","section":"Section V-A, Eqs. (1)-(4)"},{"comment":"The parameters α1, α2, α3 are assigned values but never appear in any equation or in the analysis. If they are intended for a weighted combination of the four scores, that combination is missing; otherwise they should be removed. Their presence suggests an intended overall performance index that is never defined, and the reader cannot tell which results, if any, depend on these weights.","section":"Section V-A, parameters"},{"comment":"The conclusions are drawn from qualitative comparisons of CDFs without any baseline or statistical significance test. In particular, the abstract's claim that the agent maintains 'high levels of safety and efficiency' is not supported by the data as presented: the safety CDFs in Fig. 4a lie in the range 0.95-1, but without an absolute threshold or a comparison to a reference system there is no basis for calling these values 'high'. Similarly, the statement in Section V-E and the Conclusion that adding rear cameras 'significantly enhances the system's performance' is not backed by any statistical test; 'significantly' appears to be used in a non-statistical sense. The authors should provide a baseline (e.g., a rule-based agent or another MLLM agent) and, ideally, confidence intervals or hypothesis tests for the CDF comparisons.","section":"Section V-E, Figs. 4-6"},{"comment":"The manuscript claims that no prior work has fully assessed MLLM-based driving agents in a closed-loop framework under harsh environmental conditions, yet Table I already lists Co-driver [27] with foggy/gloomy and rainy/gloomy environments. The novelty claim is therefore overstated, and the related-work comparison should clarify exactly what is new relative to that work: is it the specific agent (GPT-4o), the sensor configuration, the set of weather parameters, or the evaluation metrics?","section":"Section II, Table I and Section IV-C"}],"minor_comments":[{"comment":"The caption of Fig. 4d says 'Front cameras' but the figure includes curves labeled 'Good Weather 6 CAMS' and 'Wetness 6 CAMS'; the footnote only partially explains this. The caption should be updated to reflect the inclusion of 6-camera speed scores.","section":"Fig. 4d"},{"comment":"There is inconsistent naming in Table III: 'precipita tion deposits' appears with a space and later as 'precipitation deposits'; 'fog density' is split as 'fog density' in some rows. Unify the parameter names for clarity.","section":"Table III"},{"comment":"The prompt text in Fig. 3 contains a stray triple-quote (\"\"\") after 'SHOULD BE exactly same and no other words!', and the closing delimiter is misaligned. Fix the formatting so the prompt is shown exactly as used.","section":"Fig. 3"},{"comment":"The description of Town-06 says it is a 'low-density town' but also has 'long highways/roads with multiple lanes (4 to 6) per direction'; please clarify whether this is the only map used in all experiments and how representative it is of the weather scenarios studied.","section":"Section IV-A"},{"comment":"The phrase 'best CDF' is used repeatedly (e.g., 'the best safety CDF is achieved in heavy rain'). Since CDFs are curves, 'best' should be defined in terms of a stochastic order (e.g., first-order stochastic dominance) or a summary statistic (e.g., mean score) to make the comparisons precise.","section":"Section V-E"},{"comment":"The abstract and introduction state that the paper evaluates perception, but no perception-specific metrics (e.g., detection accuracy, segmentation quality) are reported; only driving-level scores are given. Please clarify what 'perception' refers to in this context.","section":"Abstract and Section V-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is an engineering-style empirical study; its contribution is more incremental than the authors claim, and the novelty relative to Co-driver [27] should be made precise. The GitHub repository may resolve some of the metric ambiguities, but the manuscript must stand alone. I would advise the editor to require the authors to fully specify the metrics and provide a baseline before considering the paper for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest empirical extension of known prompt-engineered MLLM driving work, with a useful sensor-ablation dataset, but the evaluation layer is too broken to support the headline claims. The authors test GPT-4o in LimSim++/CARLA under five weather conditions and three sensor configurations, and they report a few findings I have not seen elsewhere: the speed-score CDF is identical for fog, heavy rain, and storm, and adding LiDAR to the six-camera setup degrades safety and efficiency. The GitHub link is a mark in their favor; for a simulation paper, that kind of artifact helps reproducibility.\n\nThe problems are in the scoring, and they are load-bearing. The comfort score in Eq. (2) averages four sub-scores that are never defined, so the comfort CDFs in Figs. 5 and 6 cannot be recomputed. Eq. (4) contradicts its own prose: the text says speed score is 0.9 when exceeding the limit and 1 otherwise, while the equation is an exponential aggregate over frames; they agree only at exactly 10% speeding. Eq. (3) leaves v* ambiguous, the safety threshold tau_th is unspecified, and alpha1-alpha3 appear in the setup but are never used in any equation. There are also no baselines, no repeated-run statistics, and no error bars. Given that every conclusion is a statement about these scores, the central claim of \"high safety and efficiency\" is not supported by the manuscript alone.\n\nThe novelty claim also needs toning down. The paper says it integrates harsh environments into CARLA \"for the first time,\" but Table I lists Co-driver already using foggy/gloomy and rainy/gloomy conditions. That is an overclaim the authors can fix quickly.\n\nWho is this for? People tracking MLLM driving-agent benchmarks might skim it, but I would not rely on the quantitative results until the metrics are defined, baselines added, and the contradictions fixed. I would not send this to peer review in its current form; it is a revise-into-a-benchmark paper, not a publishable result. Ask the authors to define the score components, reconcile Eq. (4), run a few seeds, and compare against a non-LLM baseline.","headline":"Useful sensor-ablation data for a prompt-engineered GPT-4o driving agent, but undefined and inconsistent evaluation metrics sink the headline claims.","tokens_in":13129,"tokens_out":2767,"would_cite":false,"duration_ms":28784,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A prompt-engineered GPT-4o agent can drive a simulated car well in heavy rain, storm, fog, and wetness.","keywords":["autonomous driving","multimodal large language models","GPT-4o","weather conditions","CARLA","LimSim++","prompt engineering","sensor fusion"],"falsifier":"Re-run the same Town-06 route and weather conditions with the comfort score recomputed from a standard definition (e.g., the 95th percentile of jerk magnitude) and the speed score computed exactly as its prose says (0.9 when speeding, 1 otherwise); if the weather-condition rankings of the CDFs change, the reported safety, comfort, and efficiency conclusions are artifacts of the undefined metric formulas.","tokens_in":12078,"feed_emoji":"🚗","tokens_out":5210,"duration_ms":47512,"temperature":0.7,"pith_summary":"This paper tries to establish that a multimodal large language model, GPT-4o, can act as the sole decision-maker for an autonomous vehicle inside a simulator, even when the weather is bad. The authors build an agent called MLLM-AD-4o by wrapping GPT-4o in the LimSim++ closed-loop framework, which connects to the CARLA driving simulator, and they evaluate it in good weather, heavy rain, storm, fog, and wetness. Their central claim is that the prompt-engineered agent maintains high safety and efficiency across all these conditions, and that its performance changes in specific ways when rear cameras or LiDAR are added. If true, this would show that careful prompt engineering alone can give a general-purpose model enough driving competence for simulation-based studies, without fine-tuning or reinforcement learning.","feed_headline":"GPT-4o drives a simulated car through rain, storm, and fog","feed_subtitle":"Prompt-engineered MLLM keeps safety and efficiency high across five weather conditions; sensor mix shifts behavior.","key_machinery":"The central object is MLLM-AD-4o, a prompt-engineered agent that couples GPT-4o (a multimodal large language model) to the LimSim++ closed-loop simulation framework, which in turn co-simulates CARLA (the driving simulator) and SUMO (the traffic simulator). At each decision frame, the agent sends a prompt—containing the current camera images, optionally LiDAR data, navigation information, and an action set—to the GPT-4o API, and receives back a single action (idle, accelerate, decelerate, turn left, turn right). The authors also add a 'set weather' function to CARLA that configures eight parameters (cloudiness, precipitation, precipitation deposits, wind intensity, sun altitude angle, fog density, fog distance, wetness) to create the five weather conditions tested.","core_discovery":"The central claim is that MLLM-AD-4o, a GPT-4o-based agent that receives only a hand-crafted prompt and sensor images, can keep high safety, comfort, efficiency, and speed scores across five weather conditions in the CARLA simulator, and that these scores trade off against each other in a weather-dependent way. In good weather the agent drives more aggressively, being riskier but more efficient; in heavy rain and storm it drives slower, being safer and smoother but less efficient. The agent's sensor configuration matters: adding three rear cameras improves performance, especially in good weather, and combining LiDAR with three front cameras gives the best safety and comfort in heavy rain, while combining LiDAR with all six cameras degrades the results.","pith_inferences":["An implicit extension is that the same prompt-engineering recipe could be ported to other MLLMs or other simulators, but the paper does not demonstrate this.","The authors do not compare against a fine-tuned or reinforcement-learned agent; a natural next test would be whether MLLM-AD-4o's weather robustness beats those baselines.","The observation that rear cameras improve performance mainly in good weather could reflect the model's training distribution rather than a general perceptual benefit; that would be testable by varying the prompt wording.","The metrics are the load-bearing part; an external reader could re-run the same CDF analysis with standard jerk and TTC definitions to check whether the reported rankings persist."],"forward_implications":["If the central claim holds, prompt-engineered MLLMs can serve as zero-shot driving decision-makers without fine-tuning, suggesting a lower-cost route to simulation-based AD research.","The sensor-ablation results imply that adding perception channels does not monotonically improve an MLLM agent; the right sensor mix depends on weather conditions.","The weather-specific trade-offs (e.g., safer and smoother but less efficient in heavy rain) suggest that an MLLM agent can implicitly adapt its risk-taking to environmental conditions.","The finding that LiDAR with six cameras degrades performance points to a need for careful sensor fusion rather than simply adding more sensors."],"supporting_citations":[{"why":"Supplies the CARLA driving simulator, the testbed for all weather and sensor experiments.","marker":"[29]"},{"why":"Supplies the LimSim++ closed-loop framework that the paper modifies to integrate harsh weather and new sensors.","marker":"[31]"},{"why":"Identifies GPT-4o as the MLLM used as the driving agent's decision-maker.","marker":"[23]"},{"why":"Provides the open-access GitHub repository with the implementation and weather-setup function for reproducibility.","marker":"[22]"},{"why":"Frames the application of MLLMs to autonomous driving and motivates the gap of harsh-environment evaluation.","marker":"[4]"}],"fun_headline_variants":["GPT-4o agent varies driving style across rain, fog, and storm","MLLM-AD-4o retains safety and efficiency in harsh weather","Sensor setup shifts GPT-4o driving performance in bad weather","In CARLA, GPT-4o agent slows in storm for safety, speeds in sun","Weather impact on GPT-4o driving: safety vs efficiency trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the evaluation metrics in Section V-A being valid measures of safety, comfort, and efficiency; the paper never defines the sub-scores used in the comfort equation, and the speed score's prose description contradicts its equation, so if the metric definitions are wrong or arbitrary the 'high safety and efficiency' result is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o agent varies driving style across rain, fog, and storm","MLLM-AD-4o retains safety and efficiency in harsh weather","Sensor setup shifts GPT-4o driving performance in bad weather","In CARLA, GPT-4o agent slows in storm for safety, speeds in sun","Weather impact on GPT-4o driving: safety vs efficiency trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1468,"prompt_tokens":938,"completion_tokens":530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":554,"tokens_out":530,"duration_ms":6346,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:31:04.604175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same Town-06 route and weather conditions with the comfort score recomputed from a standard definition (e.g., the 95th percentile of jerk magnitude) and the speed score computed exactly as its prose says (0.9 when speeding, 1 otherwise); if the weather-condition rankings of the CDFs change, the reported safety, comfort, and efficiency conclusions are artifacts of the undefined metric formulas.","supporting_citations":[{"cited_title":"CARLA: An open urban driving simulator,","cited_arxiv_id":null,"evidence_quote":"Supplies the CARLA driving simulator, the testbed for all weather and sensor experiments."},{"cited_title":"LimSim++: A closed-loop platform for deploying multimodal LLMs in autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Supplies the LimSim++ closed-loop framework that the paper modifies to integrate harsh weather and new sensors."},{"cited_title":"GPT-4o: The cutting-edge advancement in multimodal LLM,","cited_arxiv_id":null,"evidence_quote":"Identifies GPT-4o as the MLLM used as the driving agent's decision-maker."},{"cited_title":"MLLM applied to autonomous driving across vari- ous weather conditions,","cited_arxiv_id":null,"evidence_quote":"Provides the open-access GitHub repository with the implementation and weather-setup function for reproducibility."},{"cited_title":"A survey on multimodal large language models for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Frames the application of MLLMs to autonomous driving and motivates the gap of harsh-environment evaluation."}],"review_version":1}