Pith. sign in

REVIEW 3 major objections 4 minor 18 references

Large Language Models in Code Co-generation for Safe Autonomous Vehicles

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that when six LLMs were asked to write controllers for four automotive functions, only GPT-4 produced code that passed every test scenario for the hardest function—an unsupervised evasive lane change—and it did so in just…

desk verdict Useful LLM-evaluation pipeline and failure-mode catalogue wrapped around a statistically fragile 'GPT-4 is the only one' claim. read the letter →

arxiv 2505.19658 v1 pith:AEE4LBJU submitted 2025-05-26 cs.SE cs.AI

Ali Nouri , Beatriz Cabrero-Daniel , Zhennan Fei , Krishna Ronanki , H{aa}kan Sivencrona , Christian Berger This is my paper
classification cs.SEcs.AI
keywords DevOpsAutonomousDrivingSystemAutomatedCodeGenerationLargeLanguageModelVerificationSimulationSafety-criticalsoftwareBenchmarkleakage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two things: that a fast automated pipeline can pre-filter LLM-generated code before human review, and that today's LLMs are not yet reliable enough to produce complex safety-critical driving logic unaided. It compares six language models—five open-source ones plus GPT-4—on four automotive controller tasks, running each of the 24 (model, function) combinations twenty times. Across the simple braking and lane-change tasks the models often succeeded, but for the most complex task, an unsupervised evasive lane change to avoid collision, only GPT-4 generated code that passed all seven simulated test scenarios, and it did so once out of twenty attempts. The paper's process contribution is a 'generate fast, eliminate fast' workflow: compile the generated code, run it in a Software-in-the-Loop simulator, rank the candidates automatically, and send only the promising versions to a human reviewer. If the results hold, they imply that teams should not ship LLM-generated code for complex driving functions without an automated verification gate and human oversight.

What carries the argument

The load-bearing mechanism is the evaluation pipeline itself: a prompt containing only the function description is sent to each LLM, the generated Python code is extracted, compiled, run in the open-source Software-in-the-Loop simulator esmini, and checked against seven test scenarios plus four requirements (R1–R4); results are logged as JSON and ranked automatically. Test-case descriptions are deliberately withheld from prompts to avoid benchmark leakage, so the simulator tests generalization rather than memorization. The named central object is esmini, the simulation environment that acts as the oracle: it executes the controller against TC1–TC7 and reports whether lane changes, decelerations, and collision-avoidance behaviours stay within the drivable area and satisfy the requirements.

What would settle it

Run the same CAEM task with the same prompts and no interface hints on an open-source model for twenty attempts: if any version passes all seven simulator scenarios, the paper's claim that only GPT-4 can do it is contradicted. Alternatively, take a GPT-4 controller that passed all seven scenarios and run it in a new traffic situation not among TC1–TC7; if it collides or leaves the drivable area, the oracle used for ranking is not sufficient.

Watch

Extended reading notes

Core claim

The central empirical discovery is a capability gap in LLM-generated automotive software. The study defines four functions of increasing complexity, ending with CAEM (Collision Avoidance by Evasive Manoeuvre), an unsupervised lane change to avoid an imminent collision. For CAEM, the paper finds that GPT-4 was the only model able to deliver code that compiled, ran in the simulator, and passed all seven test cases, with a success rate of one in twenty; every open-source model failed completely. The authors attribute the failures to identifiable causes—syntax errors, wrong threshold values, hardcoded test scenarios, division-by-zero edge cases, and unnecessary code that breaks integration—and they argue that task novelty and complexity, not parameter count, drive performance. They further claim that the proposed pipeline successfully ranks candidates and can supply a failure-mode catalogue for reviewers and prompt refinement.

Load-bearing premise

The central claim relies on the assumption that passing the seven simulated test scenarios without violating requirements R1–R4 is a fair and sufficiently sensitive measure of whether a generated controller is good enough to send to human review.

Editorial extensions

If this is right

  • A reviewer can spend time only on ranked, pre-verified candidates instead of reading every generated version, because the pipeline filters out code that does not compile, does not run, or fails the simulator scenarios.
  • Open-source LLMs in the tested set are not yet suitable for generating complex lateral-control functions such as CAEM without manual tuning of thresholds and further verification.
  • Model rankings on simple functions do not predict performance on complex functions; selecting an LLM for a task requires per-task evaluation.
  • The failure-mode catalogue—threshold miscalibration, syntax errors, hardcoded test cases, division-by-zero, redundant code—can be used to refine prompts and to design targeted checks.
  • Human oversight remains necessary: the pipeline is a pre-review filter, not a certification, and the authors position it as satisfying the human-oversight demand of the EU AI Act.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that a 1-in-20 success rate for GPT-4 on CAEM is too thin to attribute stable capability; replicating with more runs or varied traffic scenarios could shift the ranking, and the threshold values the authors deliberately withheld might be the real gate.
  • The same pipeline could be turned into an adversarial benchmark generator: failure modes from the catalogue (e.g., hardcoded test cases, zero-relative-speed division) can seed new scenarios, making it progressively harder for models to fake competence.
  • A natural extension for practitioners is to test whether providing domain-standard parameter ranges—without hinting at the strategy—raises open-source success rates enough to make them viable candidates.
  • The authors' 'safe enough to review' oracle could be validated against human judgment by giving reviewers both passing and near-miss failing codes and asking whether the pipeline's ranking agrees with their risk assessment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an LLM-based code co-generation pipeline for automotive functions, combining prompt-based generation, compilation checks, esmini SIL simulation, automatic acceptance testing, and failure-mode analysis. It compares six LLMs on four functions (F1 braking, F2 lane change, F3 ACC, F4 CAEM) with 20 runs per setup, and reports that only GPT-4 produced any code passing all scenarios for CAEM (1/20), while open-source models succeeded on simpler tasks. The paper also proposes integrating the pipeline into an LLM-augmented review process and discusses threats to validity, including benchmark leakage and the need for human oversight.

Significance. If the result holds, the paper provides a useful, reproducible pre-review filtering workflow and a rare leakage-conscious benchmark task (CAEM) for safety-related automotive code. Strengths include the explicit avoidance of test-case leakage, automated acceptance checking in an open simulator, the failure-mode catalogue, and the public artifact link. The engineering contribution is more robust than the model-ranking conclusion: the headline quantitative claim currently rests on statistically fragile counts and an underspecified prompt-selection step, so the paper is better read as a workflow demonstration than as a definitive capability ranking.

major comments (3)
  1. [Sec. 4, Fig. 3] The central claim that GPT-4 is the only model capable of generating CAEM code is not supported by the reported counts: 1 success in 20 runs for GPT-4 versus 0 in 20 for each of five open-source models. Conditional on exactly one success among 120 runs, the probability that it falls in the GPT-4 group under equal success probabilities is 20/120 ≈ 0.17, so the data are statistically indistinguishable from chance. Please provide confidence intervals, a significance test (e.g., Fisher's exact test on the 6×2 table), or more runs, and soften or rephrase the 'only LLM' claim accordingly.
  2. [Sec. 3.1] The prompt-selection procedure is underspecified. The text says four alternative prompts were used for each model and 'the most effective prompt was then selected' before esmini testing, but it does not state the criterion for 'most effective' or whether the selection used the same functions/test cases as the final 20-run evaluation. If selection was based on F4 CAEM test outcomes, the reported counts are best-of-four results and the comparison is biased. Please report the selection criterion, the results for all prompts, or use a held-out prompt set.
  3. [Sec. 3.2 and Sec. 5] The success oracle consists of seven hand-designed scenarios with fixed speeds and a 0.4 s time-to-brake. The paper correctly avoids claiming full coverage, but the conclusion that models 'could deliver the code' or are 'not yet fully ready' is an inference from this oracle to a general capability. Since the oracle's sensitivity and specificity are not characterized (e.g., how many unsafe or interface-mismatched controllers pass all TC1–TC7), I ask the authors to either limit the conclusion to 'passes the seven scenarios' or provide evidence that the oracle separates review-ready from non-review-ready code, beyond the environmental correlation in [5].
minor comments (4)
  1. [Sec. 4] In the paragraph after Fig. 3, the sentence 'Comparing the reported results of the models for F1 and F2, in Fig. 3, with the results for ACC and CAEM, in Fig. 4' references the wrong figures; F1/F2 appear in Fig. 2 and ACC/CAEM in Fig. 3.
  2. [Sec. 3.1] The tool name 'esmini' is typeset inconsistently (italic, code font, and plain text); please unify the typography throughout the manuscript.
  3. [Sec. 3.2] The phrase 'The time to brake in the scenarios is 0.4 seconds' should specify whether this is a time-to-collision at scenario start or a driver reaction time parameter, and how it is realized in the OpenSCENARIO files.
  4. [Sec. 6] The conclusion states that success rates 'vary from 15% to 100%' across setups; this range conflates different functions and models and would be clearer as per-model, per-function rates.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: an empirical measurement study whose counts are observed LLM outputs; the only self-citation ([5], esmini-track correlation) is backed by an external study, and remaining concerns are statistical validity, not circularity.

full rationale

This paper makes no derivation that reduces to its own inputs: every central claim is an observed count from a disclosed experimental protocol (6 models x 4 functions x 20 runs), so none of the seven circularity patterns applies. The success criterion, 'An experiment run is considered successful only if the generated code passes all test cases without violating any of the predefined requirements (R1–R4)' (Sec. 3.2), is the authors' own operationalization, but the resulting counts (e.g., GPT-4 1/20 for F4 in Sec. 4) are measured outputs of the LLMs, not consequences of the definition by construction; self-defined benchmarks are standard and not circular. The nearest dependency is the esmini SIL oracle, whose construct validity is supported by self-citation: 'the selected simulation environment (esmini) is part of the correlated SIL toolchain used in industrial AD development, as reported in [5]' (Sec. 5, Threats to Validity), where [5] (Fei et al., arXiv:2406.03040) shares an author with the present paper. This is real evidence rather than circular support: [5] is an independent correlation study of esmini output against physical test-track data, is parameter-free with respect to the present results, and is externally checkable, so per the review rules it does not raise the circularity score. Two flagged concerns are validity issues, explicitly out of scope for circularity: (i) 'Four alternative prompts were used for each model... The most effective prompt was then selected' (Sec. 3.1) makes the 20-run counts best-of-4 outcomes; and (ii) the headline F4 claim rests on 1 versus 0 successful runs, which is statistically fragile. Both affect whether the counts measure the claimed capability gap, not whether the counts are derived from their inputs; the paper is transparent about both, and it explicitly declines threshold tuning ('this was not done to avoid compromising the validity of the experiment', Sec. 4), the opposite of fitting a parameter to force a result. The pipeline, test descriptions, and sample codes are publicly released (Zenodo link in Sec. 3.2), so the measurements are externally reproducible. Verdict: no significant circularity; the score of 2 solely reflects the minor, evidence-backed self-citation [5] in the construct-validity argument.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper has no mathematical derivation, so the ledger lists hand-chosen experimental constants and domain assumptions that the empirical claims depend on. No invented entities (new particles, forces, dimensions) are introduced; the only 'new' artifact is a software pipeline, which is a method, not a postulated entity.

free parameters (2)
  • Test scenario speeds and brake time = 120/80/40 kph; 0.4 s
    Hand-chosen scenario parameters grounded in UNECE R157; they define the difficulty of the test oracle and thus the reported success rates.
  • Repetitions per setup and prompt variants = 20; 4
    Chosen by the authors; 20 runs bound the resolution of the binary success estimates (e.g., 1/20 vs 0/20).
assumptions (4)
  • domain assumption esmini SIL simulation is a credible proxy for physical ADAS/AD testing for these functions and scenarios.
    Invoked in Sec. 3.1 and Threats to Validity (Sec. 5), supported only by co-authored citation [5].
  • domain assumption The seven test scenarios (TC1-TC7) provide sufficient coverage to distinguish safe from unsafe controllers for the four functions.
    The paper defines success as passing all test cases without violating R1-R4 (Sec. 4), but acknowledges focus on early sanity checks, not full coverage.
  • domain assumption The four functions and nine capabilities (Table 1) are representative of automotive safety-related programming tasks relevant to LLM assessment.
    The capabilities were identified from preliminary experiments (Sec. 3.2); representativeness is asserted, not empirically established.
  • domain assumption The designed functions, especially F4, are not part of the LLMs' training data (i.e., no benchmark leakage).
    Sec. 3.2 and Threats to Validity claim leakage is minimized by task novelty, but training data are undisclosed; the paper provides only an indirect check (removing interface descriptions).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models in Code Co-generation for Safe Autonomous Vehicles." pith.science (2026). https://pith.science/paper/AEE4LBJU

@misc{pith2026250519658,
  author       = {Pith},
  title        = {Pith review of: Large Language Models in Code Co-generation for Safe Autonomous Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEE4LBJU}},
  note         = {Machine review of arXiv:2505.19658}
}
read the original abstract

Software engineers in various industrial domains are already using Large Language Models (LLMs) to accelerate the process of implementing parts of software systems. When considering its potential use for ADAS or AD systems in the automotive context, there is a need to systematically assess this new setup: LLMs entail a well-documented set of risks for safety-related systems' development due to their stochastic nature. To reduce the effort for code reviewers to evaluate LLM-generated code, we propose an evaluation pipeline to conduct sanity-checks on the generated code. We compare the performance of six state-of-the-art LLMs (CodeLlama, CodeGemma, DeepSeek-r1, DeepSeek-Coders, Mistral, and GPT-4) on four safety-related programming tasks. Additionally, we qualitatively analyse the most frequent faults generated by these LLMs, creating a failure-mode catalogue to support human reviewers. Finally, the limitations and capabilities of LLMs in code generation, and the use of the proposed pipeline in the existing process, are discussed.

Figures

Figures reproduced from arXiv: 2505.19658 by the authors.

Figure 1
Figure 1. Designed and implemented pipeline, including the LLM model, prompts, and simulation environment. The function description is automatically inserted into the prompt, sent to the LLM, and the generated Python code is extracted from the LLMs’ responses. Then, the compilable codes are sent to esmini and tested against relevant test cases for the specific function. Finally, a report is generated and attached to the code … view at source ↗
Figure 2
Figure 2. Reports the performance of all LLM models on two simple functions (F1 and F2). The left bar of each model presents the results for F1 (i.e., brake if the speed is higher than 10 m/s²), and the right bar presents the results for F2 (i.e., lane change until reaching the rightmost lane). The performance of the models is ranked first based on the total number of successful codes for F2 and then based on F1, as F2 is con… view at source ↗
Figure 3
Figure 3. Performance of six models on two advanced automotive functions (F3 and F4). The ACC bar in each group indicates the models’ performance in F3, and the CAEM bars present the results for F4. The models are ranked first based on total successful generations for F3 and F4, and then by the number of executable code generations. Contrary to the open-source models, only 3 of the code versions generated by GPT-4 for F3 and … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Pipeline in [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

  1. [5]

    arXiv preprint arXiv:2406.03040 (2024), https://arxiv.org/abs/2406.03040

    Fei, Z., Andersson, M., Tingberg, A.: Correlation of Software-in-the-Loop Simulation with Physical Testing for Autonomous Driving. arXiv preprint arXiv:2406.03040 (2024), https://arxiv.org/abs/2406.03040

    arXiv 2024

  2. [1]

    arXiv–2409 (2024)

    Ashani Mahawatta Dona, M., Cabrero-Daniel, B., Yu, Y., Berger, C.: LLMs Can Check Their Own Results to Mitigate Hallucinations in Traffic Understanding Tasks pp. arXiv–2409 (2024)

    work page 2024

  3. [2]

    FAccT ’21, Association for Computing Machinery, New York, NY, USA (2021)

    Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021ACMConferenceonFairness,Accountability,andTransparency.p.610‚Äì623. FAccT ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3442188.3445922

    arXiv 2021

  4. [3]

    Cabrero-Daniel, B., Herda, T., Pichler, V., Eder, M.: Exploring Human-AI Collab- orationinAgile:CustomisedLLMMeetingAssistants.In:InternationalConference on Agile Software Development. pp. 163–178. Springer Nature Switzerland Cham (2024) 16 A. Nouri et al

    work page 2024

  5. [4]

    CNECT (2021), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206

    European Commission, Directorate-General for Communications Networks, Con- tent and Technology: EUR-Lex - 52021PC0206 - EN - EUR-Lex. CNECT (2021), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206

    work page 2021

  6. [6]

    Publications Office of the European Union, Luxembourg„ EUR30942 (2021)

    Fernández Llorca, D., Gómez, E.: Trustworthy Autonomous Vehicles. Publications Office of the European Union, Luxembourg„ EUR30942 (2021)

    work page 2021

  7. [7]

    High-Level Expert Group on Artificial Intelligence: Ethics Guidelines for Trustwor- thy Artificial Intelligence (AI (2019),https://www.europarl.europa.eu/doceo/ document/TA-9-2024-0138_EN.pdf

    work page 2019

  8. [8]

    Knabe, E.: Environment Simulator Minimalistic (esmini).https://github.com/ esmini/esmini, accessed: 2025-02-23

    work page 2025

Show all 18 references
  1. [9]

    In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

    Kumar, J., Chimalakonda, S.: Code Summarization without Direct Access to Code - Towards Exploring Federated LLMs for Software Engineering. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. pp. 100–109 (2024)

    2024

  2. [10]

    Applied Sciences14(3) (2024)

    Liu, M., Wang, J., Lin, T., Ma, Q., Fang, Z., Wu, Y.: An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs. Applied Sciences14(3) (2024). https://doi.org/10.3390/app14031046

    2024 doi

  3. [11]

    In: Proceedings of the Genetic and Evolution- ary Computation Conference

    Liventsev, V., Grishina, A., Härmä, A., Moonen, L.: Fully Autonomous Program- ming with Large Language Models. In: Proceedings of the Genetic and Evolution- ary Computation Conference. GECCO ‚Äô23, ACM (Jul 2023).https://doi.org/ 10.1145/3583131.3590481, http://dx.doi.org/10.1...

    2023

  4. [12]

    Frontiers of Computer Science19(8), 198341 (2025)

    Lyu, Z., Li, X., Xie, Z., Li, M.: Top pass: improve code generation by pass@ k- maximized code ranking. Frontiers of Computer Science19(8), 198341 (2025)

    2025

  5. [13]

    In: 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR)

    Nguyen, N., Nadi, S.: An Empirical Evaluation of GitHub Copilot’s Code Sug- gestions. In: 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR). pp. 1–5 (2022).https://doi.org/10.1145/3524842.3528470

    2022

  6. [14]

    In: 2024 IEEE 32nd International Requirements Engineering Conference (RE)

    Nouri, A., Cabrero-Daniel, B., Törner, F., Sivencrona, H., Berger, C.: Engineering Safety Requirements for Autonomous Driving with Large Language Models. In: 2024 IEEE 32nd International Requirements Engineering Conference (RE). pp. 218–228. IEEE (2024)

    2024

  7. [15]

    In: Pro- ceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

    Ságodi, Z., Antal, G., Bogenfürst, B., Isztin, M., Hegedűs, P., Ferenc, R.: Reality Check: Assessing GPT-4 in Fixing Real-World Software Vulnerabilities. In: Pro- ceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. pp. 252–261 (2024)

    2024

  8. [16]

    In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

    Siddiq, M.L., Da Silva Santos, J.C., Tanvir, R.H., Ulfat, N., Al Rifat, F., Car- valho Lopes, V.: Using Large Language Models to Generate JUnit Tests: An Em- pirical Study. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineerin...

    2024

  9. [17]

    157 - Automated Lane Keeping Systems (ALKS) (2021),https://unece.org/transport/documents/2021/03/standards/ un-regulation-no-157-automated-lane-keeping-systems-alks

    UNECE: UN Regulation No. 157 - Automated Lane Keeping Systems (ALKS) (2021),https://unece.org/transport/documents/2021/03/standards/ un-regulation-no-157-automated-lane-keeping-systems-alks

    2021

  10. [18]

    arXiv preprint arXiv:2311.01964 (2023)

    Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W.X., Chen, X., Lin, Y., Wen, J.R., Han, J.: Don’t Make Your LLM an Evaluation Benchmark Cheater. arXiv preprint arXiv:2311.01964 (2023)

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.