Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Generating tests after an LLM's faulty code cuts fault detection roughly in half versus writing tests from the specification alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 08:11 UTC pith:Q4KBVO4K

load-bearing objection Real, quantified risk that code-then-test LLM workflows mask faults (~14% detection vs ~25% for independent tests), with solid multi-model controls; agentic arm is a bit forced but the core comparison holds. the 3 major comments →

arxiv 2607.05139 v1 pith:Q4KBVO4K submitted 2026-07-06 cs.SE

On the risk of coding before testing: An empirical study on LLM-based test generation workflow

Michael Konstantinou , Florian Tambon , Mike Papadakis This is my paper
classification cs.SE
keywords LLM-based test generationerror propagationagentic workflowsfault detectiontest oraclescode-before-testprompt independence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when the same large language model writes code and then writes the tests for that code, faults in the implementation systematically leak into the tests. The tests end up validating the same mistaken behavior instead of catching it, so incorrect code and incorrect tests look mutually consistent. Across standard coding benchmarks and several models, tests generated only from the natural-language task description detect faults about 25 percent of the time, while tests generated after the model has already produced faulty code detect them only about 14 percent of the time. The gap survives popular prompting tricks such as summarization, chain-of-thought, and chain-of-verification, and it appears in multi-step agentic sessions that keep conversation history. The practical claim is that independence between code and tests is not automatic in LLM workflows; without deliberate separation, automated testing loses its main purpose.

Core claim

Error propagation is real and costly: when tests are generated after (or conditioned on) faulty LLM code, fault-detection effectiveness falls from roughly 25 percent to roughly 14 percent compared with generating tests from the task description alone, and the same reduction appears under summarization, chain-of-thought, chain-of-verification, and agentic multi-step workflows that reuse earlier code as context.

What carries the argument

Error propagation: the systematic replication of faults from an LLM-generated implementation into subsequent test artifacts (especially assertions), measured by comparing fault detection of tests written with versus without exposure to that faulty code.

Load-bearing premise

That forcing a model to continue from a pre-collected faulty implementation, and scoring detection against each benchmark's reference suite, faithfully captures real agentic coding sessions and true expected behavior.

What would settle it

Repeat the same prompt-only versus code-exposed comparison on a larger, industrial corpus where ground-truth oracles are written independently by humans, and check whether the 14 percent versus 25 percent gap disappears or reverses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Code-before-test agentic pipelines can create a false sense of correctness: high pass rates may simply mean tests inherited the same mistakes.
  • Specification-first or independent test-generation workflows detect more faults than co-evolutionary same-model loops.
  • Empirical studies that couple code and test generation in one pipeline risk systematically underestimating true fault-finding power.
  • Prompt-engineering techniques that still see the faulty implementation do not restore independence; separation of context is required.
  • Even under-specified task descriptions remain more useful for fault-revealing tests than exposure to the buggy code itself.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Tool builders may need multi-model or external-oracle designs so that test generation never conditions on the candidate implementation.
  • The same contamination risk likely applies to other co-generated artifacts (docs, types, formal specs) produced by one model session.
  • Benchmarks that score 'tests pass on generated code' without an independent oracle may be measuring consistency rather than correctness.
  • Human-in-the-loop review focused only on failing tests will miss the silent aligned-failure cases this study highlights.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper empirically studies error propagation in LLM-based software workflows: when a model first produces faulty code and then generates tests (or is given that code as context), the resulting tests tend to encode the same incorrect behavior, reducing fault detection relative to generating tests from the task description alone. Using faulty implementations filtered from HumanEval+, MBPP, and BigCodeBench across five models, the authors compare prompt-only, code-exposed, summarization, CoT, CoVe, and an agentic conversation-history workflow. Fault detection falls from roughly 25% (test-driven / prompt-only) to about 14% (code-after or agentic), with Mann–Whitney tests and controls for test count, line coverage, fault triggering, and under-specified prompts. The authors conclude that lack of independence between generated code and tests is a structural risk for agentic and code-before-test pipelines and a threat to validity for coupled evaluation protocols.

Significance. If the result holds, it is a practically important finding for LLM-assisted software engineering: the convenient code-then-test (or vibe-coding) ordering can systematically mask defects rather than reveal them, and common reasoning prompts do not remove the bias. Strengths include a multi-model, multi-benchmark design; an external ground truth via reference implementations and reference suites; explicit separation of fault triggering vs detection; statistical testing; and diagnostic controls showing that the gap is not explained by suite size or line coverage. The work also usefully reframes prior oracle-bias observations as a workflow-level threat. The contribution is empirical and methodological rather than theoretical, but it is timely and actionable for tool builders and for empirical studies that couple generation steps.

major comments (3)
  1. [Section V, RQ3 / Agentic workflow] Section V (Agentic workflow) and RQ3: the agentic condition forces the model’s first response to equal a pre-collected faulty implementation (“we control the model’s initial code generation response to exactly match the faulty implementation(s) collected at an earlier stage”) while preserving conversation history. This is not a live code-then-test trajectory under the same sampling path, temperature, or intermediate reasoning that produced the fault. Because the headline 14% vs 25% comparison and Finding 3 rest on this setup, the paper needs either (a) a live-generation arm in which the same session produces both code and tests, or (b) a clear threat analysis of how forced injection could over- or under-state coupling relative to natural generation. Without that, the agentic claim is only partially supported even though RQ1 already shows a related prompt-level bias.
  2. [Section IV-A, Fault ranking and selection] Section IV-A (Fault ranking and selection): for each task the authors keep a single implementation that fails ≤50% of reference tests and, among candidates, the one that fails the fewest tests (highest D(i)). This focuses evaluation on hard faults, which is defensible, but the reported absolute detection rates (and the abstract’s 14% vs 25%) are then conditioned on this filter and on one fault per task. The paper should report how many tasks survive per benchmark/model, how sensitive results are to the 50% threshold and to selecting the hardest vs a random or median-difficulty fault, and whether the workflow gap remains when easier or multiple faults per task are included. Otherwise the magnitude of the effect may not generalize to the full distribution of LLM faults.
  3. [Section VI, Figures 2–4 and Tables II–IV] Results (Figures 2–4, Tables II–IV): main claims are stated as percentages without a consolidated table of denominators (number of retained faulty implementations per model × benchmark × condition) or absolute detected-fault counts. Mann–Whitney tests are reported, but readers cannot assess effect size stability or power from the text alone. Please add N, absolute detections, and preferably per-benchmark breakdowns for the primary Prompt-only vs Prompt+Code vs Agentic comparisons so that the 13–15% average drops can be audited.
minor comments (5)
  1. [Section IV-A / IV-C] Clarify temperature and decoding settings for every generation step (code sampling is given as 0.8; test-generation temperature and whether reasoning models use extended thinking are less explicit).
  2. [Figure 5] Figure 5 stacks fault triggering and detection but the caption and legend are easy to misread; state explicitly which segment is detection vs additional triggering-only.
  3. [Introduction / Abstract] The informal term “vibecoding” appears in the introduction and abstract without definition; a short neutral definition or a more standard “agentic code-then-test” phrasing would help international readers.
  4. [Section VIII] Related work correctly cites prior oracle-bias results (e.g., Konstantinou et al., Huang et al.); make the novelty boundary one sentence sharper: prompt-level bias vs multi-step workflow / conversation-history bias.
  5. [Section IX] Threats to Validity should explicitly list the forced-injection agentic design and the single-hardest-fault selection, not only non-determinism and benchmark generalizability.

Circularity Check

0 steps flagged

No significant circularity: fault-detection rates are measured against external benchmark reference implementations and suites; self-citations are background only.

full rationale

The paper’s central claim (error propagation reduces fault detection from ~25% independent/test-driven to ~14% code-then-test/agentic) is an empirical comparison of LLM workflows on HumanEval+, MBPP and BigCodeBench. Faulty implementations are filtered by the benchmarks’ own reference test suites; detection is defined strictly as “fails on the faulty implementation while passing on the corresponding reference implementation.” No parameter is fitted to a subset of the target metric and then re-presented as a prediction; no uniqueness theorem or ansatz is imported from the authors’ prior work to force the result; the 14% vs 25% numbers are direct experimental outcomes, not algebraic identities. Self-citations (e.g., Konstantinou et al. [7], [13]) appear only as related-work context on oracle bias and LLM-Plain; they do not close a definitional loop that produces the reported percentages. The controlled injection of pre-collected faults into the agentic conversation is a methodological choice that may affect external validity, but it does not make the measured detection rates circular by construction. Hence the derivation chain is self-contained against external oracles and scores at most a minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard software-testing independence assumptions, public coding benchmarks as ground truth, a chosen fault-filtering policy, and a simplified agent simulation (history reuse + compilation-only repair). No new physical entities are postulated; “error propagation” is a named empirical phenomenon. Free parameters are experimental knobs (temperature, filter threshold, samples per task) that shape the dataset of faults under study rather than fitted constants in a predictive equation.

free parameters (3)
  • code_sampling_temperature
    Temperature 0.8 used when generating ten candidate implementations per task to diversify faults; affects which buggy programs enter the evaluation set.
  • fault_filter_threshold
    Implementations failing more than 50% of reference tests are excluded as “too easy”; this hand-chosen cutoff defines the hard-fault corpus the main results are measured on.
  • samples_per_task
    Ten generated solutions per task; together with the difficulty ranking, this determines which single faulty implementation is retained per task.
axioms (4)
  • domain assumption Effective testing requires independence between the system under test and the test oracle.
    Stated in the introduction as the classical SE premise the paper challenges for coupled LLM generation.
  • domain assumption Benchmark reference implementations and reference suites define correct expected behavior for fault detection.
    Fault detection requires fail-on-faulty and pass-on-reference; all primary metrics depend on this external ground truth.
  • domain assumption Autoregressive conditioning on prior tokens can systematically reuse incorrect assumptions across sequential artifacts.
    Used to motivate why code-before-test context would bias subsequent assertions (Introduction/Background).
  • ad hoc to paper LLM-Plain with compilation-only repair is a fair lightweight stand-in for modern iterative test-generation agents when isolating reasoning/workflow effects.
    Section IV adopts LLM-Plain and disables behavioral repair so oracles remain model-chosen; this is a design choice of the study, not a standard theorem.
invented entities (1)
  • error propagation (code-to-test aligned failures) independent evidence
    purpose: Name and study the phenomenon where faults in generated code are mirrored in generated tests so both are mutually consistent and defects are masked.
    Operationalized via lower fault detection under code-conditioned/agentic workflows versus independent prompt-only tests; not a physical entity, but a paper-introduced construct for the measured effect.

pith-pipeline@v1.1.0-grok45 · 22766 in / 3034 out tokens · 30312 ms · 2026-07-11T08:11:21.357763+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly used in software engineering workflows to generate both source code and test suites. This dual capability has enabled emerging development paradigms, including test-first and agentic workflows, where a single model is producing and validating implementations. However, these approaches assume that generated tests act as independent and reliable oracles - a fundamental requirement for effective software testing. In this paper, we challenge this assumption and investigate whether LLM-generated code biases the generation of subsequent tests. We introduce and empirically study the phenomenon of error propagation, where faults in generated code are systematically replicated in associated test artifacts. This leads to cases where incorrect implementations and tests are mutually consistent, masking defects rather than revealing them. We evaluate this effect across a range of programming tasks and agentic workflows, analyzing the consistency between generated code and test assertions, with particular focus on scenarios of aligned failures. Our study examines (i) whether erroneous code artifacts bias test generation, (ii) whether such bias persists under different prompting strategies, including chain-of-thought reasoning, and (iii) how errors propagate across multi-step workflows in which intermediate outputs are reused as context. The results show that error propagation is prevalent and impactful: generating tests after faulty code significantly reduces fault detection effectiveness compared to generating tests independently (14% vs. 25%). These findings highlight a fundamental limitation of current workflows, where lack of independence between generated artifacts undermines the reliability of automated testing. Furthermore, our results expose a previously underexplored threat to validity in empirical studies relying on coupled generation pipelines.

Figures

Figures reproduced from arXiv: 2607.05139 by Florian Tambon, Michael Konstantinou, Mike Papadakis.

Figure 1
Figure 1. Figure 1: Methodology. Faulty implementations are generated and filtered to construct the evaluation dataset. The resulting implementations are subsequently used to evaluate the effectiveness of the investigated LLM-based test generation techniques. due to runtime errors, exceptions, or other execution failures are excluded. Faults ranking and selection: The previous phase produces a set of candidate faulty implemen… view at source ↗
Figure 2
Figure 2. Figure 2: RQ1: Fault Detection across all models and bench￾marks. Generating test cases from the task description alone leads to better fault detection rate. conversational history throughout the workflow. Specifically, the model first receives the task description and generates an implementation. Since we are interested on what happens when the model generates faulty code, we only consider these cases, i.e., we dis… view at source ↗
Figure 3
Figure 3. Figure 3: illustrates the fault detection obtain for each prompt engineering technique used. We observe that generating test suites from the task description alone is still more effective than any approach contains the code under test. More pre￾cisely, Prompt-only achieves higher fault detection than the summarization-based workflow across all evaluated models, with improvements of 11.4%, 17.5%, 20.7%, 15.3%, and 12… view at source ↗
Figure 4
Figure 4. Figure 4: Fault Detection under different workflows Finally, we evaluate fault triggering to determine whether the generated test suites primarily validate the current im￾plementation, a phenomenon previously observed in LLM￾generated test oracles [7]. Table V reports the total number of tests generated by each model. Overall, generating tests directly from the task descrip￾tion (Prompt-only) does not produce the la… view at source ↗
Figure 5
Figure 5. Figure 5: illustrates the fault detection rates reported in our results, together with their corresponding fault triggering rates. We observe that fault triggering and fault detection are not consistently correlated. For example, generating test suites directly from the task description achieves the highest fault detection rates across models, yet it often exhibits the lowest fault triggering rates. However, the age… view at source ↗
Figure 6
Figure 6. Figure 6: Fault Detection on MBPP tasks using weak prompts expose the implementation under test. This observation raises a natural question: how does the fault detection effectiveness of LLM-generated test suites change when the task specification itself is under-specified? Prior study explored the influence of under-specified prompts [26] when it comes to the correctness of generated code. The authors introduced th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

    cs.SE 2026-07 conditional novelty 7.0

    By replaying each validation command on buggy, candidate, and gold-fix code, this study finds 46% of positive checks do not discriminate the bug, and a reminder alone explains part of the feedback benefit.

Reference graph

Works this paper leans on

57 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ammann and J

    P. Ammann and J. Offutt,Introduction to Software Testing. Cambridge University Press, 2008. [Online]. Available: https://doi.org/10.1017/CBO9780511809163

    work page doi:10.1017/cbo9780511809163 2008

  2. [2]

    Pezz `e and M

    M. Pezz `e and M. Young,Software testing and analysis - process, principles and techniques. Wiley, 2007

    2007

  3. [3]

    Context dependence and reliability in autoregressive language models,

    P. Sengupta, S. R. Pandey, S. Maharjan, and F. Eliassen, “Context dependence and reliability in autoregressive language models,” 2026. [Online]. Available: https://arxiv.org/abs/2602.01378

    arXiv 2026

  4. [4]

    Contextual drag: How errors in the context affect llm reasoning,

    Y . Cheng, X. Zhu, H. Zhao, and S. Arora, “Contextual drag: How errors in the context affect llm reasoning,” 2026. [Online]. Available: https://arxiv.org/abs/2602.04288

    arXiv 2026

  5. [5]

    Measuring the influence of incorrect code on test generation,

    D. Huang, J. M. Zhang, M. Harman, M. Du, and H. Cui, “Measuring the influence of incorrect code on test generation,”arXiv preprint arXiv:2409.09464, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    State of what art? a call for multi-prompt llm evaluation,

    M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky, “State of what art? a call for multi-prompt llm evaluation,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 933–949, 2024

    2024

  7. [7]

    Do llms generate test oracles that capture the actual or the expected program behaviour?

    M. Konstantinou, R. Degiovanni, and M. Papadakis, “Do llms generate test oracles that capture the actual or the expected program behaviour?”

  8. [8]

    Available: https://arxiv.org/abs/2410.21136

    [Online]. Available: https://arxiv.org/abs/2410.21136

    Pith/arXiv arXiv

  9. [9]

    Agentic large language models, a survey,

    A. Plaat, M. J. van Duijn, N. van Stein, M. Preuss, P. van der Putten, and K. J. Batenburg, “Agentic large language models, a survey,”J. Artif. Intell. Res., vol. 84, 2025. [Online]. Available: https://doi.org/10.1613/jair.1.18675

    doi:10.1613/jair.1.18675 2025

  10. [10]

    Understanding the planning of LLM agents: A survey,

    X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y . Wang, R. Tang, and E. Chen, “Understanding the planning of LLM agents: A survey,”CoRR, vol. abs/2402.02716, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.02716

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.02716 2024

  11. [11]

    Chatassert: Llm-based test oracle generation with external tools assistance,

    I. Hayet, A. Scott, and M. d’Amorim, “Chatassert: Llm-based test oracle generation with external tools assistance,”IEEE Transactions on Software Engineering, vol. 51, no. 1, pp. 305–319, 2024

    2024

  12. [12]

    Chatassert: Llm-based test oracle generation with external tools assistance,

    ——, “Chatassert: Llm-based test oracle generation with external tools assistance,”IEEE Trans. Software Eng., vol. 51, no. 1, pp. 305–319,

  13. [13]

    Available: https://doi.org/10.1109/TSE.2024.3519159

    [Online]. Available: https://doi.org/10.1109/TSE.2024.3519159

    work page doi:10.1109/tse.2024.3519159 2024

  14. [14]

    Synthesizing file-level data for unit test generation with chain-of-thoughts via self-debugging,

    Z. Hua, T. Chen, Y . Gong, S. Lu, P. Cheng, Q. Zhu, Y . He, Y . Fu, W. Jiao, W. Yang, and T. Xie, “Synthesizing file-level data for unit test generation with chain-of-thoughts via self-debugging,”CoRR, vol. abs/2602.03181,

    arXiv

  15. [15]

    Available: https://doi.org/10.48550/arXiv.2602.03181

    [Online]. Available: https://doi.org/10.48550/arXiv.2602.03181

    work page doi:10.48550/arxiv.2602.03181

  16. [16]

    How well llm-based test generation techniques perform with newer llm versions?

    M. Konstantinou, R. Degiovanni, and M. Papadakis, “How well llm-based test generation techniques perform with newer llm versions?”

  17. [17]

    Available: https://arxiv.org/abs/2601.09695

    [Online]. Available: https://arxiv.org/abs/2601.09695

    arXiv

  18. [18]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  19. [19]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  21. [21]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paulet al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 66 602–66 656

    2025

  22. [22]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY , USA: Curran Associates Inc., 2022

    2022

  23. [23]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–23, 2025

    2025

  24. [24]

    Chain-of-verification reduces hallucination in large language models,

    S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucination in large language models,” inFindings of the association for computational linguistics: ACL 2024, 2024, pp. 3563–3578

    2024

  25. [25]

    Yate: The role of test repair in llm-based unit test generation,

    M. Konstantinou, R. Degiovanni, J. M. Zhang, M. Harman, and M. Pa- padakis, “Yate: The role of test repair in llm-based unit test generation,” arXiv preprint arXiv:2507.18316, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Consistency meets verification: Enhancing test generation quality in large language models without ground-truth solutions,

    H. Taherkhani, A. DaghighFarsoodeh, M. Chowdhury, H. V . Pham, and H. Hemmati, “Consistency meets verification: Enhancing test generation quality in large language models without ground-truth solutions,”arXiv preprint arXiv:2602.10522, 2026

    arXiv 2026

  27. [27]

    Chain of targeted verification questions to improve the reliability of code generated by llms,

    S. Kouemo Ngassom, A. Moradi Dakhel, F. Tambon, and F. Khomh, “Chain of targeted verification questions to improve the reliability of code generated by llms,” inProceedings of the 1st ACM International Conference on AI-Powered Software, 2024, pp. 122–130

    2024

  28. [28]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The annals of mathematical statistics, pp. 50–60, 1947

    1947

  29. [29]

    An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption,

    T. T. Chekam, M. Papadakis, Y . Le Traon, and M. Harman, “An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption,” in2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 597–608

    2017

  30. [30]

    When prompt under-specification improves code correctness: An exploratory study of prompt wording and structure effects on llm-based code generation,

    A. Akli, M. Papadakis, M. Cordy, and Y . L. Traon, “When prompt under-specification improves code correctness: An exploratory study of prompt wording and structure effects on llm-based code generation,”

  31. [31]

    Available: https://arxiv.org/abs/2604.24712

    [Online]. Available: https://arxiv.org/abs/2604.24712

    Pith/arXiv arXiv

  32. [32]

    Measuring the influence of incorrect code on test generation,

    D. Huang, J. M. Zhang, M. Harman, M. Du, and H. Cui, “Measuring the influence of incorrect code on test generation,” in48th IEEE/ACM International Conference on Software Engineering, ICSE 2026, 2026

    2026

  33. [33]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024. [Online]. Available: https://doi.org/10.1145/3660783

    doi:10.1145/3660783 2024

  34. [34]

    HITS: high-coverage llm-based unit test generation via method slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: high-coverage llm-based unit test generation via method slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024, V . Filkov, B. Ray, and M. Zhou, Eds. ACM, 2024, pp. 1258–

    2024

  35. [35]

    Available: https://doi.org/10.1145/3691620.3695501

    [Online]. Available: https://doi.org/10.1145/3691620.3695501

    doi:10.1145/3691620.3695501

  36. [36]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643769

    doi:10.1145/3643769 2024

  37. [37]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V . Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 30–34

    2024

  38. [38]

    Coverup: Coverage-guided llm-based test generation,

    J. A. Pizzorno and E. D. Berger, “Coverup: Coverage-guided llm-based test generation,”CoRR, vol. abs/2403.16218, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.16218

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.16218 2024

  39. [39]

    Test wars: A comparative study of sbst, symbolic execution, and llm-based approaches to unit test generation,

    A. Abdullin, P. Derakhshanfar, and A. Panichella, “Test wars: A comparative study of sbst, symbolic execution, and llm-based approaches to unit test generation,” inIEEE Conference on Software Testing, Verification and Validation, ICST 2025, Napoli, Italy, March 31 - April 4, 2025. IEEE, 2025, pp. 221–232. [Online]. Available: https://doi.org/10.1109/ICST6...

    doi:10.1109/icst62969.2025.10989033 2025

  40. [40]

    Toga: a neural method for test oracle generation,

    E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, “Toga: a neural method for test oracle generation,” inProceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 2130–2141. [Online]. Available: https://doi.org/10.1145/3510003.3510141

    doi:10.1145/3510003.3510141 2022

  41. [41]

    Togll: Correct and strong test oracle generation with llms,

    S. B. Hossain and M. Dwyer, “Togll: Correct and strong test oracle generation with llms,” 2024. [Online]. Available: https://arxiv.org/abs/2405.03786

    Pith/arXiv arXiv 2024

  42. [42]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting,

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openr...

    2024

  43. [43]

    Evaluating the zero-shot robustness of instruction-tuned language models,

    J. Sun, C. Shaib, and B. C. Wallace, “Evaluating the zero-shot robustness of instruction-tuned language models,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=g9diuvxN6D

    2024

  44. [44]

    Promptception: How sensitive are large multimodal models to prompts?

    M. I. Ismithdeen, M. U. Khattak, and S. Khan, “Promptception: How sensitive are large multimodal models to prompts?” in Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Association for Computational Linguistics, 2025, pp. 23 950–23 9...

    doi:10.18653/v1/2025.findings-emnlp.1302 2025

  45. [45]

    State of what art? A call for multi-prompt LLM evaluation,

    M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky, “State of what art? A call for multi-prompt LLM evaluation,”Trans. Assoc. Comput. Linguistics, vol. 12, pp. 933–949,

  46. [46]

    Available: https://doi.org/10.1162/tacl a 00681

    [Online]. Available: https://doi.org/10.1162/tacl a 00681

    doi:10.1162/tacl

  47. [47]

    Prosa: Assessing and understanding the prompt sensitivity of llms,

    J. Zhuo, S. Zhang, X. Fang, H. Duan, D. Lin, and K. Chen, “Prosa: Assessing and understanding the prompt sensitivity of llms,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, ser. Findings of ACL, Y . Al-Onaizan, M. Bansal, and Y . Chen, Eds., vol. EMNLP

    2024

  48. [48]

    Association for Computational Linguistics, 2024, pp. 1950–

    2024

  49. [49]

    Available: https://doi.org/10.18653/v1/2024.findings- emnlp.108

    [Online]. Available: https://doi.org/10.18653/v1/2024.findings- emnlp.108

    doi:10.18653/v1/2024.findings- 2024

  50. [50]

    POSIX: A prompt sensitivity index for large language models,

    A. Chatterjee, H. S. V . N. S. K. Renduchintala, S. Bhatia, and T. Chakraborty, “POSIX: A prompt sensitivity index for large language models,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, ser. Findings of ACL, Y . Al-Onaizan, M. Bansal, and Y . Chen, Eds., vol. EMNLP 2024. Association ...

    doi:10.18653/v1/2024.findings-emnlp.852 2024

  51. [51]

    When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions,

    M. Larbi, A. Akli, M. Papadakis, R. Bouyousfi, M. Cordy, F. Sarro, and Y . Le Traon, “When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions,”arXiv preprint arXiv:2507.20439, 2025. [Online]. Available: https://arxiv.org/abs/2507.20439

    Pith/arXiv arXiv 2025

  52. [52]

    Defective task descriptions in LLM-based code generation: Detection and analysis,

    A. Akli, M. Papadakis, M. Cordy, and Y . Le Traon, “Defective task descriptions in LLM-based code generation: Detection and analysis,”arXiv preprint arXiv:2604.24703, 2026. [Online]. Available: https://arxiv.org/abs/2604.24703

    Pith/arXiv arXiv 2026

  53. [53]

    Analyzing prompt influence on automated method generation: An empirical study with Copilot,

    I. D. Fagadau, L. Mariani, D. Micucci, and O. Riganelli, “Analyzing prompt influence on automated method generation: An empirical study with Copilot,” inProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension (ICPC 2024), 2024, pp. 24– 34

    2024

  54. [54]

    Prompt stability in code LLMs: Measuring sensitivity across emotion- and personality-driven variations,

    W. Ma, Y . Yang, J. Ge, X. Xie, and L. Jiang, “Prompt stability in code LLMs: Measuring sensitivity across emotion- and personality-driven variations,”arXiv preprint arXiv:2509.13680, 2025

    arXiv 2025

  55. [55]

    Mind your format: Towards consistent evaluation of in-context learning improvements,

    A. V oronov, L. Wolf, and M. Ryabinin, “Mind your format: Towards consistent evaluation of in-context learning improvements,” inFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, ser. Findings of ACL, L. Ku, A. Martins, and V . Srikumar, Eds., vol. ACL 2024. Association for Compu...

    doi:10.18653/v1/2024.findings-acl.375 2024

  56. [56]

    On the evaluation of large language models in unit test generation,

    L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang, and J. Chen, “On the evaluation of large language models in unit test generation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18181

    Pith/arXiv arXiv 2024

  57. [57]

    Beck,Extreme programming explained: embrace change

    K. Beck,Extreme programming explained: embrace change. addison- wesley professional, 2000

    2000