Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In test-oracle generation, improving input context beats adding iterative self-repair.

desk verdict Useful empirical paper on non-iterative oracle generation, but the 'context beats repair' claim is not actually isolated by the experiments. read the letter →

arxiv 2608.05917 v1 pith:RQNXKP64 submitted 2026-08-06 cs.SE

Kefan Li , Hongyue Yu , Yuan Yuan This is my paper · ORCID
classification cs.SE
keywords testoraclegenerationregressiontestingself-repairtraplargelanguagemodelsmutationexecutionfeedbackstaticcontextdynamicstateextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the recent trend of using iterative self-repair with execution feedback to generate regression-test oracles is counterproductive: the feedback loop optimizes for assertions that pass, not assertions that detect regressions, a failure the authors call the Self-Repair Trap. As an alternative, the paper proposes DCAware, a non-iterative pipeline that feeds the model a 'folded' static context (keeping only invoked methods and class skeletons) plus runtime values the model itself chooses to inspect, then generates the oracle in one pass. On 500 TECO assertion-completion instances, DCAware raises pass rate and mutation-kill rate well above the ChatAssert baseline while using roughly a quarter of the generation tokens and 68-77% less time per instance. The authors' conclusion is that improving contextual quality is more effective than adding repair complexity in this setting.

What carries the argument

The load-bearing mechanism is DCAware's dual-context construction. Contextual Semantic Folding builds a static prompt with three pieces: the test class skeleton (lifecycle methods and utilities kept, other test methods folded), the focal class (imports, the full focal method body, signatures of sibling methods), and invoked callees (imports and full bodies of called methods only). Intent-Driven Dynamic State Extraction then treats the LLM as an active debugger: given the static context, the model emits queries (variables or side-effect-free expressions), which are evaluated by attaching the Java Debugger during a paused test run; the returned values are appended as dynamic context, and anti-overfitting guardrails instruct the model not to assert volatile memory references. The intended work of these components is to raise the signal-to-noise ratio of the context so a single generation can produce an assertion that both executes and kills mutants.

What would settle it

Run a fully non-iterative version of ChatAssert that keeps ChatAssert's original context but issues a single generation with no repair rounds; if this single-shot ChatAssert matches or beats DCAware, the claim that context quality drives the gains would be undercut, and the Self-Repair Trap would reduce to the cost of extra rounds. Alternatively, add an iterative repair loop to DCAware's own context: if repair further raises KG, then repair is not inherently trapping.

Watch

Extended reading notes

Core claim

The paper's central claim is that execution-guided iterative repair in LLM-based regression-oracle generation exhibits a feedback-driven degeneration: repairs increase the chance an assertion passes but weaken the assertion's fault-revealing ability. The evidence is a stage-wise analysis of ChatAssert showing later repair rounds recover oracles with lower kill rates, plus DCAware's performance: pass rate and KG of 80.8%/73.2% (Qwen3-Coder-30B) and 88.2%/78.17% (GPT-5-mini), surpassing ChatAssert's 68.4%/59.0% and 73.2%/61.56% with 68-77% lower per-instance time. The paper concludes that a single-pass pipeline grounded in high signal-to-noise static and dynamic context is sufficient for strong oracles, and that improving contextual quality is more effective than adding iterative repair complexity in the studied setting.

Load-bearing premise

The conclusion that improving context quality is more effective than iterative repair assumes that the performance gap between DCAware and ChatAssert comes from context construction rather than from ChatAssert's repair loop, since no variant in the paper separates the two factors.

Editorial extensions

If this is right

  • Regression-oracle completion can be done in a single generation without a repair loop, cutting per-instance time by 68-77% and generation tokens by roughly 90%.
  • Execution success is a proxy that, when used as a repair objective, can bias models toward weaker assertions; DCAware's Shared KG surpasses both Direct Prompting and ChatAssert on common passing-instance subsets.
  • Static context anchors dynamic queries: removing the static phase causes the largest drop in performance, showing the folded skeleton is what makes targeted runtime retrieval useful.
  • DCAware's gains hold across Qwen3-Coder-30B and GPT-5-mini and, in a supplementary run, on the larger GPT-5, suggesting the approach is not tied to one model's instruction-following strength.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Self-Repair Trap generalizes beyond oracle generation, then generate-execute-repair loops in other code-synthesis tasks (e.g., bug fixing, test generation) may also silently trade quality for passability; that is a testable hypothesis the paper does not pursue.
  • Because the paper never runs a non-iterative version of ChatAssert nor an iterative version of DCAware, its headline conclusion that context quality—rather than the mere absence of repair—causes the gains rests on an untested comparison; a two-by-two design would settle which factor matters.
  • The anti-overfitting guardrails (prohibiting memory addresses, toString, set-order assumptions) could plausibly be transferred to any execution-feedback test generator as a cheap robustness patch, independent of DCAware's context machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper targets regression-oracle completion, where an executable test prefix is provided and the current program version is treated as expected behavior. It proposes DCAware, a non-iterative pipeline that constructs a folded static context via JavaParser and an LLM-generated set of dynamic-state queries evaluated through a Java debugger, then produces a single oracle assertion. On 500 TECO assertion-completion instances, DCAware reports higher Pass Rate and KG than ChatAssert, Direct Prompting, and earlier training-based baselines, with lower per-instance time and cost. The paper also introduces the 'Self-Repair Trap' hypothesis, arguing that execution-guided iterative repair biases models toward passable but weaker assertions, and presents ablations and robustness checks for its context components.

Significance. If the reported results hold, the paper has clear practical value: it demonstrates a cheaper, non-agentic route to strong regression oracles and evaluates oracle quality with dynamic execution and mutation testing rather than static similarity metrics. The design has genuine strengths: the guardrail confound is explicitly checked (Section 5.2), repeated-run stability is reported (Figure 6), and the cost analysis distinguishes token, time, and API expense. However, the central conceptual claim—that contextual quality matters more than iterative repair—is not isolated by the current experimental design, and the evidence for the 'Self-Repair Trap' is correlational and limited to one system's behavior. The paper would be a solid empirical contribution if those gaps are addressed by additional crossed ablations and tightened statistical reporting; in its present form, the main conclusion overreaches the data.

major comments (3)
  1. [Section 3.3, Table 1, RQ5, and Conclusion] The abstract's closing claim—that improving contextual quality is more effective than adding iterative repair complexity—is not isolated by the experimental design. DCAware and ChatAssert differ simultaneously in context construction (folded static context plus selective dynamic queries vs. retrieval and execution feedback) and in whether an iterative repair loop is used. No configuration holds context fixed while toggling the repair loop: there is no non-iterative ChatAssert variant and no DCAware variant that runs a repair loop on its own context. The observed gains are therefore compatible with several alternative explanations, including the possibility that any non-iterative pipeline with these contexts would perform as well, or that adding repair to DCAware's context would improve results further. The stage-wise evidence in Figure 4 does not resolve this because it is drawn only from ChatAssert's behavior. I recommend adding crossed conditions (e.g., DCAware with and without a repair loop; ChatAssert's context without repair) or explicitly softening the causal claim to a comparative statement about the two implemented systems.
  2. [Section 3.3, Figure 4] The 'Self-Repair Trap' diagnosis rests on aggregate stage-grouped statistics with no reported number of instances per stage, no per-oracle repair trace, and no analysis of which assertions were modified between rounds. Figure 4 reports cumulative Pass Rate and 'Newly Passed KG' for stage groups 1, 2–5, and >5, but without stage-group sizes or confidence intervals it is not possible to assess whether later stages recover 'increasingly weaker oracles' or merely a small number of noisy outliers. Since the phenomenon is presented as a general objective-misalignment failure mode in Sections 1 and 4.1, the authors should report the distribution of repair round counts, the number of oracles in each stage group, and ideally a matched-oracle comparison of the same instances before and after repair. Without this, the causal narrative that execution feedback biases later-stage oracles toward weaker assertions is not established.
  3. [Section 3.2, Eq. (1), Tables 2, 4, 6, 10] The mutation-based comparisons on shared passed subsets carry the fault-revealing claims, but the reported differences are not accompanied by significance tests or effect sizes, and several differences are very small (e.g., Table 4a shows 92.56–93.34 over 176 instances; Table 10b shows 92.07 vs. 92.08 on 215 instances). With shared-set sizes of 152–273, one-percentage-point gaps are within sampling noise, and the paper's own interpretation of Table 10b as 'essentially identical' indicates that the reader cannot reliably tell which differences are meaningful. I request pairwise significance tests or bootstrap confidence intervals for all Shared KG comparisons. In addition, the KG metric is computed on a filtered set of 362 valid instances (Section 3.2); the paper should characterize which 138 problems were excluded and show that the filtering does not interact with the compared methods, since Pass Rate and KG are reported over different denominators.
minor comments (5)
  1. [Figure 4] The figure should state the number of oracles in each stage group, and the y-axis should clarify that KG is computed only over newly passed oracles, since the comparison between cumulative Pass Rate and non-cumulative KG is otherwise easy to misread.
  2. [Section 3.2] The ChatAssert baseline configuration is not fully specified in this paper (e.g., maximum repair rounds, retrieval parameters, and the exact way guardrails are injected). Adding this information would improve reproducibility, especially because the paper modifies ChatAssert by applying its guardrails.
  3. [Table 1] DCAware's Test Failed (TF) rate is considerably higher than ChatAssert's (11.8% vs. 0.8% for Qwen3-Coder-30B; 7.6% vs. 0.2% for GPT-5-mini) while the Runtime Error rate is much lower. This trade-off is not discussed and may be relevant to how the Pass Rate advantage should be interpreted in practice.
  4. [Section 3.2, Eq. (1)] The formula's numerator restricts credit to mutants in the union of golden-killed and golden-survived mutants, while the denominator counts only golden-killed mutants; this design choice is deliberate but deserves an explicit sentence in the text so that readers do not misinterpret KG as a plain mutation score.
  5. [Section 4.4 and Section 5.1] The GPT-5 result in Section 4.4 is reported without the repeated-run stability analysis that is used for the main models in Section 5.1; applying the same three-run protocol to the larger model would make the robustness claim uniform.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DCAware's gains are measured against external mutation testing, and no fitted parameter or self-citation is recycled into the evaluation.

full rationale

The paper's central comparison (DCAware vs. ChatAssert) is an empirical benchmark on 500 TECO instances with KG computed by PITest mutants; the method was not fitted to minimize KG, and no equation from the method is reused as its own evidence. The 'Self-Repair Trap' is introduced as a term for an observed pattern and then evidenced by stage-wise analysis of ChatAssert (Section 3.3, Figure 4); this is an empirical inference, not a constructional equivalence. The conclusion that contextual quality is more effective than iterative repair is not fully isolated because no ablation toggles the repair loop on identical context, but that is an experimental confound and a validity threat, not a circularity: the paper does not define DCAware's quality in terms of the outcome metric, nor does it import a load-bearing result from its own prior work. All citations to prior work are external (notably ChatAssert), and the only self-citation is the artifact availability statement. Therefore no circular step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on methodological assumptions rather than fitted constants. No numerical free parameters are introduced; temperature and mutator choice are standard hyperparameters. The main assumptions are the validity of mutation-kill rate as oracle strength, the regression-oracle definition of expected behavior, the side-effect-freedom of debugger queries, and the representativeness of the ChatAssert stage-wise analysis.

assumptions (4)
  • domain assumption Mutation kill rate, as computed by PITest with the DEFAULTS mutator and restricted to the golden oracle's mutant scope, is a valid measure of an oracle's fault-revealing capability.
    Used to define KG and Shared KG in Section 3.2; all fault-revealing conclusions inherit this assumption.
  • domain assumption The regression-oracle setting treats the current program version as the expected behavior, so supplying the focal method body and runtime state is legitimate context rather than leakage.
    Stated in Section 1; underpins both the static and dynamic context phases.
  • domain assumption LLM-generated debugger queries are side-effect-free and JDB evaluation does not perturb the state relevant to the final oracle.
    Section 2.2 relies on evaluating arbitrary expressions in the live JVM; Section 5.2 provides only a conservative name-based scan, not semantic verification.
  • domain assumption The ChatAssert baseline is faithfully reproduced and its stage-wise behavior (Figure 4) is representative of iterative repair approaches.
    The Self-Repair Trap evidence is drawn from one system on one dataset; no other repair-based methods are analyzed at stage granularity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness." pith.science (2026). https://pith.science/paper/RQNXKP64

@misc{pith2026260805917,
  author       = {Pith},
  title        = {Pith review of: Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQNXKP64}},
  note         = {Machine review of arXiv:2608.05917}
}
read the original abstract

Large Language Models (LLMs) have shown strong potential for regression-oracle completion, where a test prefix is given and the current program version is treated as expected behavior. Recent approaches increasingly rely on iterative self-repair and execution feedback, but optimizing execution success does not necessarily yield strong fault-revealing oracles. This objective, widely adopted in repair-based methods, serves only as a proxy and may be misaligned with the true goal of oracle generation. Such misalignment biases the repair process, giving rise to a feedback-driven degeneration that we term the Self-Repair Trap, where iterative repair progressively drives models toward assertions that are easier to satisfy but less effective at detecting faults. To address this issue, we propose DCAware, a computationally efficient, non-iterative framework that prioritizes high signal-to-noise contextual grounding over multi-round repair. DCAware integrates structured static context with selectively retrieved dynamic states, enabling precise and robust oracle generation without iterative feedback loops. Extensive experiments based on execution and mutation testing show that DCAware consistently improves fault-revealing effectiveness while maintaining high execution success, outperforming prior methods with substantially lower computational cost. Our results suggest that improving contextual quality is more effective than adding iterative repair complexity in the studied regression-oracle setting.

Figures

Figures reproduced from arXiv: 2608.05917 by the authors.

Figure 1
Figure 1. A motivating example illustrating the degeneration [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the DCAware framework. The pipeline constructs a folded static context, retrieves targeted runtime [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Anti-overfitting guardrails used during oracle gen [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Evolution of ChatAssert across generation and re [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Case study comparing Locals and DCAware. Tar [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Stability of Pass Rate and KG across three indepen [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 9 canonical work pages

  1. [1]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, Wannita Takerngsaksiri, Chetan Arora, and Aldeida Aleti. 2025. Enhancing large language models for text-to-testcase generation.Journal of Systems and Software230 (2025), 112531. doi:10.1016/j.jss.2025.112531

    arXiv 2025

  2. [2]

    Juan Altmayer Pizzorno and Emery D. Berger. 2025. CoverUp: Effective High Coverage Test Generation for Python.Proc. ACM Softw. Eng.2, FSE, Article FSE128 (June 2025), 23 pages. doi:10.1145/3729398

    doi:10.1145/3729398 2025

  3. [3]

    Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

    Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

  4. [4]

    Adam Bodicoat, Gunel Jahangirova, and Valerio Terragni. 2025. Understanding LLM-Driven Test Oracle Generation. In2025 2nd IEEE/ACM International Confer- ence on AI-powered Software (AIware). 29–39. doi:10.1109/AIware69974.2025.00011

    arXiv 2025

  5. [5]

    Mohamed Salah Bouafif, Mohammad Hamdaqa, and Edward Zulkoski. 2025. PRIMG: Efficient LLM-driven Test Generation Using Mutant Prioritization. In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE ’25). Association for Computing Machinery, New York, NY, USA, 1107–1116. doi:10.1145/3756681.3756991

    arXiv 2025

  6. [6]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. InCom- panion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering(Porto de Galinhas, Brazil)(FSE 2024). Association for Com- puting Machinery, New York, NY, USA, 572–576. ...

    arXiv 2024

  7. [7]

    2022.CodeParrot-Small-Multi

    CodeParrot. 2022.CodeParrot-Small-Multi. Hugging Face. Retrieved July 25, 2026 from https://huggingface.co/codeparrot/codeparrot-small-multi

    work page 2022

  8. [8]

    Henry Coles and PIT contributors. [n. d.].PIT: Mutation Testing for Java and the JVM. PIT Project. Retrieved July 25, 2026 from https://github.com/hcoles/pitest

    work page 2026

Show all 59 references
  1. [9]

    Ermira Daka and Gordon Fraser. 2014. A Survey on Unit Testing Practices and Problems. In2014 IEEE 25th International Symposium on Software Reliability Engineering. 201–211. doi:10.1109/ISSRE.2014.11

    2014 doi

  2. [10]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained Large Language Models and mutation testing.Information and Software Technology 171 (2024), 107468. doi:10.1016/j.infsof.2024.107468

    2024

  3. [11]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery,...

    2022

  4. [12]

    Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri

  5. [13]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2...

    2020

  6. [14]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering(Szeged, Hungary)(ESEC/FSE ’11). Association fo...

    2011

  7. [15]

    Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. InProceedings of the 2005 ACM SIGPLAN Confer- ence on Programming Language Design and Implementation(Chicago, IL, USA) (PLDI ’05). Association for Computing Machinery, New York, N...

    2005

  8. [17]

    Ishrak Hayet, Adam Scott, and Marcelo d’Amorim. 2025. ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance.IEEE Transactions on Software Engineering51, 1 (2025), 305–319. doi:10.1109/TSE.2024.3519159

    2025

  9. [19]

    Dwyer, Sebastian Elbaum, and Willem Visser

    Soneya Binta Hossain, Antonio Filieri, Matthew B. Dwyer, Sebastian Elbaum, and Willem Visser. 2023. Neural-Based Test Oracle Generation: A Large-Scale Evalua- tion and Lessons Learned. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium o...

    2023

  10. [20]

    Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Documentation on LLM-Based Test Oracle Generation. Proc. ACM Softw. Eng.2, FSE, Article FSE084 (June 2025), 22 pages. doi:10.1145/ 3729354

    2025

  11. [21]

    Chao Hu, Wenhao Zeng, Yuling Shi, Beijun Shen, and Xiaodong Gu. 2026. In Line with Context: Repository-Level Code Generation via Context Inlining.Proc. ACM Softw. Eng.3, FSE, Article FSE066 (June 2026), 23 pages. doi:10.1145/3797094

    2026 doi

  12. [22]

    JavaParser Project. [n. d.].JavaParser. JavaParser Project. Retrieved July 25, 2026 from https://github.com/javaparser/javaparser

    2026

  13. [23]

    Shaker Mahmud Khandaker, Fitsum Kifetew, Davide Prandi, and Angelo Susi

  14. [24]

    Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv:2410.21136 [cs.SE] https://arxiv.org/abs/2410.21136

    2024 arXiv

  15. [25]

    Rahul Krishna, Rangeet Pan, Saurabh Sinha, Srikanth Tamilselvam, Raju Pavuluri, and Maja Vukovic. 2025. Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights. InProceedings of the 33rd ACM In- ternational Conference on the Foundations of Soft...

    2025

  16. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principl...

    2023

  17. [27]

    Lahiri, and Siddhartha Sen

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen

  18. [28]

    Bolun Li, Zhihong Sun, Tao Huang, Hongyu Zhang, Yao Wan, Ge Li, Zhi Jin, and Chen Lyu. 2024. IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion.Proc. ACM Softw. Eng.1, FSE, Article 9 (July 2024), 22 pages. doi:10.1145/3643735

    2024 doi

  19. [29]

    Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. 2025. PatchPilot: A Cost- Efficient Software Engineering Agent with Early Attempts on Formal Verification. arXiv:2502.02747 [cs.RO] https://arxiv.org/abs/2502.02747

    2025 arXiv

  20. [30]

    2026.Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context A wareness (Artifact)

    Kefan Li, Hongyue Yu, and Yuan Yuan. 2026.Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context A wareness (Artifact). doi:10. 5281/zenodo.21357436

    2026

  21. [31]

    Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. 2024. A3A3-CodGen: A Repository-Level Code Generation Framework for Code Reuse With Local-Aware, Global-Aware, and Third-Party-Library-Aware.IEEE Transactions on Software ...

    2024

  22. [32]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/

    2004

  23. [33]

    Zhongxin Liu, Kui Liu, Xin Xia, and Xiaohu Yang. 2023. Towards More Realistic Evaluation for Neural Test Oracle Generation. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis(Seattle, WA, USA)(ISSTA 2023). Association for Computing M...

    2023

  24. [34]

    Qiuyang Mang, Runyuan He, Suyang Zhong, Xiaoxuan Liu, Huanchen Zhang, and Alvin Cheung. 2026. Automated Discovery of Test Oracles for Database Management Systems Using LLMs.Proc. ACM Manag. Data4, 3, Article 140 (May 2026), 28 pages. doi:10.1145/3802017

    2026 doi

  25. [35]

    Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. In2021 IEEE/ACM 43rd International Conference on Softw...

    2021

  26. [36]

    2007.xUnit Test Patterns: Refactoring Test Code

    Gerard Meszaros. 2007.xUnit Test Patterns: Refactoring Test Code. Addison- Wesley

    2007

  27. [37]

    Facundo Molina, Alessandra Gorla, and Marcelo d’Amorim. 2025. Test Oracle Automation in the Era of LLMs.ACM Trans. Softw. Eng. Methodol.34, 5, Article 150 (May 2025), 24 pages. doi:10.1145/3715107

    2025 doi

  28. [38]

    Ernst, and Mauro Pezzè

    Davide Molinelli, Luca Di Grazia, Alberto Martin-Lopez, Michael D. Ernst, and Mauro Pezzè. 2025. Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 278–290. d...

    2025

  29. [39]

    Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1026–1038. doi:10.1109/ICSE55347.2025.00243

    2025

  30. [40]

    Mooney, and Milos Gligoric

    Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023. Learning Deep Semantics for Test Completion. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2111–2123. doi:10. 1109/ICSE48619.2023.00178

    2023

  31. [41]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. arXiv:2203.13474 [cs.LG] https://arxiv.org/abs/2203.13474

    2023 arXiv

  32. [42]

    2025.GPT-5 System Card

    OpenAI. 2025.GPT-5 System Card. OpenAI. Retrieved July 25, 2026 from https://openai.com/index/gpt-5-system-card

    2025

  33. [43]

    Ravin Ravi, Dylan Bradshaw, Stefano Ruberto, Gunel Jahangirova, and Valerio Terragni. 2025. LLMLOOP: Improving LLM-Generated Code and Tests Through Automated Iterative Feedback Loops. In2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). 930–934. ...

    2025

  34. [44]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sun- daresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE] https://arxiv.org/abs/2009.10297

    2020 arXiv

  35. [45]

    P. Runeson. 2006. A survey of unit testing practices.IEEE Software23, 4 (2006), 22–29. doi:10.1109/MS.2006.91

    2006 doi

  36. [46]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proc. ACM Softw. Eng.1, FSE, Article 43 (July 2024), 21...

    2024 doi

  37. [47]

    Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Ko- valenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. InProceedings of the 2024 IEEE/ACM 46th Inter- national Conference on Software Engineering: Compan...

    2024

  38. [48]

    Zhihong Sun, Yao Wan, Jia Li, Hongyu Zhang, Zhi Jin, Ge Li, and Chen Lyu

  39. [49]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Un- derstanding and Generation. InProceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, Marie-Francin...

    2021 doi

  40. [50]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, Escaping the Self-Repair Trap: Improving Test O...

    2024

  41. [51]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshy- vanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing(Seoul, South Korea)(ICSE ’20). Associat...

    2020

  42. [52]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang

  43. [53]

    Qinghua Xu, Guancheng Wang, Lionel Briand, and Kui Liu. 2026. Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation.ACM Trans. Softw. Eng. Methodol.(March 2026). Just Accepted. doi:10.1145/3803418

    2026 doi

  44. [54]

    InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24)

    Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machinery, New York, NY,...

  45. [55]

    Quanjun Zhang, Weifeng Sun, Chunrong Fang, Bowen Yu, Hongyan Li, Meng Yan, Jianyi Zhou, and Zhenyu Chen. 2025. Exploring Automated Assertion Generation via Large Language Models.ACM Trans. Softw. Eng. Methodol.34, 3, Article 81 (Feb. 2025), 25 pages. doi:10.1145/3699598 Receiv...

    2025 doi

  46. [59]

    arXiv:2407.01489 [cs.SE] https://arxiv.org/abs/2407.01489

    Agentless: Demystifying LLM-based Software Engineering Agents. arXiv:2407.01489 [cs.SE] https://arxiv.org/abs/2407.01489

    arXiv

  47. [61]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388

    2025 arXiv

  48. [1547]

    doi:10.18653/v1/2020.findings-emnlp.139

    2020 doi

  49. [2015]

    doi:10.1109/TSE.2014.2372785

    The Oracle Problem in Software Testing: A Survey.IEEE Transactions on Software Engineering41, 5 (2015), 507–525. doi:10.1109/TSE.2014.2372785

    2015

  50. [2024]

    ACM Softw

    Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?Proc. ACM Softw. Eng.1, FSE, Article 84 (July 2024), 24 pages. doi:10.1145/3660791

    2024 doi

  51. [2025]

    In2025 IEEE Conference on Software Testing, Verification and Validation (ICST)

    AugmenTest: Enhancing Tests with LLM-Driven Oracles. In2025 IEEE Conference on Software Testing, Verification and Validation (ICST). 279–289. doi:10. 1109/ICST62969.2025.10988926

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.