The reviewed record of science sign in
Pith

arxiv: 2607.06195 · v1 · pith:XQGQIXOY · submitted 2026-07-07 · cs.SE

LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle

Minghui Long , Yanjie Zhao , Haoyu Wang This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 13:34 UTCglm-5.2pith:XQGQIXOYrecord.jsonopen to challenge →

classification cs.SE
keywords automated testingtest oracle problemLLM agent frameworksfuzzingagentic oracleLangChainLlamaIndexCrewAI
0
0 comments X

The pith

Active AI oracle finds 40 real bugs in LangChain, LlamaIndex, CrewAI

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the central obstacle in testing LLM agent frameworks is not generating test inputs but deciding whether a failing test reflects a genuine framework defect — the test oracle problem. In pure-Python frameworks like LangChain, LlamaIndex, and CrewAI, bugs rarely crash; they surface as ordinary exceptions (ValueError, KeyError) indistinguishable from API misuse, or as silent semantic failures where execution succeeds but behavior is wrong. The paper introduces LogicHunter, a two-phase system that pairs specification-driven test generation with an Agentic Oracle: an LLM-based agent that, rather than passively classifying a failure as bug-or-not, actively investigates by retrieving documentation, navigating source code via reflection, and executing reproduction scripts in a sandbox. The Agentic Oracle uses a structured reasoning loop (ReAct inside a finite state machine with three phases — initialization, investigation, verdict), pruned memory to avoid context saturation, and a high-confidence consensus mechanism requiring four independent sessions to agree before reporting a bug. The paper claims this design achieves 91.17% precision — meaning roughly 1 in 1.1 flagged failures is a real bug — compared to 29.27% for the best passive LLM judge, and that the system discovered 40 previously unknown bugs across the three frameworks (30 confirmed by developers, 26 fixed), while state-of-the-art baselines found zero.

Core claim

The paper's central claim is that replacing a passive test oracle (one that classifies a failure from a fixed snapshot of information) with an active agentic oracle (one that dynamically retrieves documentation, inspects source code, and runs reproduction code to gather evidence before rendering a verdict) transforms the precision of automated bug detection in LLM agent frameworks from impractical (~29%) to deployment-grade (~91%). The mechanism carrying this claim is the Agentic Oracle itself: a ReAct-based agent with dual-layer state management (an inner reasoning-acting loop inside an outer three-state finite state machine), dual-stream memory (persistent reasoning traces plus pruned tool

What carries the argument

Agentic Oracle: a ReAct-based LLM agent with three-phase state management (Initialization, Investigation, Verdict), dual-stream memory (persistent reasoning trace + pruned observation window), four introspection tools (code_search, doc_search, run_code, get_test_info), a six-dimensional defect taxonomy, and high-confidence consensus requiring four independent sessions to agree before flagging a bug. On the generation side: specification-driven seed synthesis fusing Pydantic schemas and type hints with mined real-world usage patterns, a Fixer Agent for seed repair, and a Mutator Agent producing valid-but-extreme inputs with behavioral probes (assertions encoding expected semantic properties).

If this is right

  • If the Agentic Oracle approach generalizes, the bottleneck in automated testing of complex Python libraries shifts from test generation to oracle design — and active, tool-using LLM agents become a standard component of fuzzing pipelines rather than a research curiosity.
  • The six-dimensional defect taxonomy and high-confidence consensus mechanism suggest a template for deploying LLM-based oracles in other domains where crash-based oracles fail (e.g., data processing libraries, configuration systems, DSL interpreters).
  • The finding that passive LLM judges plateau at ~29% precision even with strong models (GPT-5.2) implies that simply scaling model capability does not solve the oracle problem — structured investigation and evidence-gathering architecture matter more than raw model power.
  • The 32 silent-failure bugs (out of 40 total) indicate that the most dangerous defects in agent frameworks are invisible to crash-based testing, which has implications for how production AI systems should be validated before deployment.

Load-bearing premise

The oracle's precision and recall are measured on a corpus of 1,000 failures generated by LogicHunter's own test generator, with ground-truth labels assigned by two developers who are co-authors of the paper. This means the oracle is evaluated only on the distribution of failures that LogicHunter's generator produces — if the generator systematically misses certain bug classes, the oracle's measured accuracy does not reflect performance on the true bug distribution of thes

What would settle it

Run the Agentic Oracle on a failure corpus generated by an independent test generator (not LogicHunter) with ground-truth labels assigned by developers unaffiliated with the paper. If precision drops substantially below 91%, the self-referential evaluation loop is inflating the metric.

Figures

Figures reproduced from arXiv: 2607.06195 by Haoyu Wang, Minghui Long, Yanjie Zhao.

Figure 1
Figure 1. Figure 1: Comparison of test generation approaches. (a) Unit-Test Generation produces valid inputs with regression-oriented assertions. (b) Probe-Oriented Testing (our approach) generates spec-compliant inputs with expectation-oriented behavioral probes. (c) Fuzzing explores semantically blind inputs without assertions. as bugs. Second, the inference process remains a black box: even correct verdicts lack verifiable… view at source ↗
Figure 2
Figure 2. Figure 2: The workflow of LogicHunter. Phase I synthesizes specification-compliant tests with behavioral probes. Phase II filters anomalies and deploys the Agentic Oracle for evidence-based diagnosis. 3.1 System Overview LogicHunter implements a two-phase pipeline (see [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt templates for (a) generator agent, (b) fix agent, and (c) mutator agent. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The workflow of the Agentic Oracle. The system utilizes Dual-Layer State Management to guide [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of 𝑘𝑚𝑖𝑛 on oracle performance. We conducted a systematic ablation study on the key components and parameters of LogicHunter, cov￾ering both the generation side and the oracle side: usage mining in seed synthesis, the state manage￾ment strategy (minimum enforced reasoning rounds 𝑘𝑚𝑖𝑛) ( [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Case study of an internal deduplication bug in LlamaIndex. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
read the original abstract

Large Language Model (LLM) agent frameworks such as LangChain, LlamaIndex, and CrewAI have become critical infrastructure powering production AI systems, yet they remain severely under-tested due to fundamental challenges in automated testing. Unlike traditional software, where crashes serve as reliable oracles, defects in these pure Python frameworks manifest as ordinary exceptions or silent semantic failures, creating profound oracle ambiguity. This problem is exacerbated by strict type governance through Pydantic schemas and complex protocol requirements that cause existing fuzzers to generate overwhelming invalid inputs, while traditional test generators produce only trivial cases with weak regression assertions. We present LogicHunter, a fuzzing framework that addresses both the generation and oracle challenges through active specification-aware testing. LogicHunter employs specification-driven generation that systematically fuses formal type constraints with authentic usage patterns from real-world repositories, synthesizing inputs that are valid by construction yet semantically extreme, equipped with behavioral probes to expose silent failures. To resolve oracle ambiguity, we introduce the Agentic Oracle, which transcends passive classification by actively retrieving documentation, navigating source code, and inspecting runtime states through a ReAct-based architecture with Dual-Layer State Management and Dual-Stream Memory. Evaluated on three widely deployed frameworks, LogicHunter discovered 40 previously unknown bugs with 30 confirmed and 26 fixed by developers, while state-of-the-art baselines reported no bugs as final findings. The Agentic Oracle achieves 91.17% precision, surpassing the best passive approach at 29.27% by 61 percentage points.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 9 minor

Summary. This paper presents LogicHunter, a fuzzing framework for testing LLM agent frameworks (LangChain, LlamaIndex, CrewAI). The system has two phases: (1) specification-driven test generation that fuses formal type constraints with real-world usage patterns to produce valid-by-construction yet semantically extreme inputs with behavioral probes, and (2) an Agentic Oracle that uses a ReAct-based architecture with dual-layer state management and dual-stream memory to actively retrieve documentation, inspect source code, and execute reproduction scripts to diagnose whether anomalous executions constitute genuine bugs. The evaluation reports 40 previously unknown bugs (30 confirmed, 26 fixed by developers), with state-of-the-art baselines finding zero bugs. The Agentic Oracle achieves 91.17% precision on a Realistic Failure Corpus of 1,000 labeled failures, compared to 29.27% for the best passive baseline. An ablation study isolates the contributions of state management, memory pruning, tool access, and consensus mechanisms.

Significance. The paper addresses a genuine and important problem: the oracle problem for pure-Python agent frameworks where defects manifest as ordinary exceptions or silent semantic failures rather than crashes. The discovery of 40 real bugs with 30 developer confirmations provides strong external validation of practical utility. The ablation study (Table 8) systematically isolates component contributions, and the evaluation includes both GPT-5-mini and GPT-5.2 backbones for oracle baselines. The publicly available artifact (GitHub) and the inclusion of supplementary datasets for recall and false-positive-rate reference checks are commendable. The six-dimensional defect taxonomy and the high-confidence consensus mechanism are well-motivated design choices that address real problems in LLM-based triage.

major comments (4)
  1. §4.1.5, Table 7: The headline precision claim (91.17%) is measured on the Realistic Failure Corpus, which is constructed from LogicHunter's own generator output and labeled by two co-authors (Cohen's Kappa 0.77). This creates a self-referential evaluation loop: the generator, the oracle, and the ground truth are all produced by the same team. The two supplementary datasets partially address this but each tests only one dimension — the post-cutoff known-bug set (31 bugs) measures recall only (60.65%, no non-bugs to test FPR), and the hard-negative set (484 TitanFuzz failures) measures FPR only (0.04%, no genuine bugs to test recall). There is no single externally-sourced corpus where both precision and recall are measured simultaneously. This is load-bearing for the central precision claim and should be addressed by either (a) constructing a mixed external corpus with both bugs and non-bs
  2. §4.1.5: The labeling was performed by 'two developers' who are co-authors of the paper. The paper should explicitly disclose this conflict of interest in the evaluation section and discuss what safeguards were applied to prevent labeling bias (e.g., whether labeling was done blind to the oracle's verdicts, whether any independent reviewer was consulted). Without this disclosure, readers cannot assess the reliability of the ground-truth labels that underpin all oracle accuracy metrics.
  3. Table 7 vs Table 8: The oracle accuracy metrics differ between the two tables. Table 7 reports P=91.17%, R=72.14%, F1=80.49%, FPR=0.21% for the Agentic Oracle, while Table 8 reports P=90.91%, R=71.43%, F1=80.00%, FPR=0.21% for the 'Default Agentic Oracle.' The paper should clarify whether these are the same experiment with rounding differences, or different experimental setups (e.g., different corpus subsets). If the latter, the relationship between the two evaluations should be stated explicitly.
  4. §4.5: The paper states that the Agentic Oracle's recall on the post-cutoff known-bug set (60.65%) is lower than passive LLM judges (67-94%) because 'the Agentic Oracle requires stronger evidence from source-code inspection and executable validation before reporting a bug.' This is a plausible explanation, but it also means the oracle's high precision on the primary corpus may come at the cost of missing bugs that do not fit its evidence-gathering patterns. The paper should discuss whether this precision-recall tradeoff is inherent to the design or whether it reflects the self-referential corpus bias (i.e., the oracle is tuned to find exactly the kinds of bugs its generator produces).
minor comments (9)
  1. §3.2.3, Eq. for N_mut: The formula ⌊α·C(api)/M_max⌋·M_max simplifies to ⌊α·C(api)/M_max⌋·M_max, which is always a multiple of M_max. If the intent is to allocate budgets proportional to complexity, this quantization seems coarse. Please clarify the values of α and M_max used in the experiments.
  2. Table 6: The 'Cov.(%)' column header is ambiguous — it is unclear whether this is line coverage, branch coverage, or something else. §4.2 mentions 'Line coverage' but the table should state this explicitly.
  3. §4.1.4: TELPA is described as evaluated 'only [for] the Pynguin-based variant, as the CodaMOSA-based variant is incompatible.' This is a reasonable exclusion, but the paper should note whether this affects the fairness of the comparison (i.e., whether the CodaMOSA variant might perform better).
  4. Figure 5: The y-axis label is unclear ('Impact of k_min on oracle performance'). The figure shows multiple metrics but the legend or axis labels should explicitly state which metric each line represents.
  5. §5.2: The limitations section mentions that 'nondeterminism cannot be fully eliminated despite temperature-zero decoding and HCC.' It would be useful to quantify the residual nondeterminism — e.g., what fraction of oracle verdicts change across the five repeated runs reported in Table 7?
  6. Table 2: The 'Data Integrity' category description mentions 'LangChain splitters losing info' as an example — this is overly specific for a general taxonomy definition. Consider using a more abstract example.
  7. §4.3: The paper states '32 root-cause clusters are reported separately in our artifact' but Table 4 reports 40 bugs. The relationship between '40 unique bugs' and '32 root-cause clusters' should be clarified in the main text, not just deferred to the artifact.
  8. References [34] and [27]: Reference [34] (MirrorFuzz) is dated 2026, which appears to be a future date. Reference [27] duplicates [26] (both cite 'Lost in the Middle'). Please consolidate.
  9. §4.1.2: The paper uses 'GPT-5-mini' and 'GPT-5.2' as model names. If these are pseudonyms or placeholders for actual models, this should be clarified; if they are real model names, the access dates and API versions should be specified for reproducibility.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The comments identify legitimate concerns about evaluation methodology that we will address in revision. Below we respond point by point.

read point-by-point responses

  1. Referee: §4.1.5, Table 7: The headline precision claim (91.17%) is measured on the Realistic Failure Corpus, which is constructed from LogicHunter's own generator output and labeled by two co-authors. This creates a self-referential evaluation loop. The two supplementary datasets each test only one dimension. There is no single externally-sourced corpus where both precision and recall are measured simultaneously.

    Authors: We agree this is a genuine methodological gap. The Realistic Failure Corpus is indeed generated by LogicHunter's own pipeline and labeled by co-authors, and the two supplementary datasets each evaluate only one axis (recall or FPR) in isolation. No single external corpus simultaneously tests both precision and recall. We will address this in two ways. First, we will construct a mixed external corpus by combining the 31 post-cutoff known-bug set with a sample of the 484 hard-negative TitanFuzz failures, creating a single externally-sourced dataset containing both genuine bugs and non-bugs. We will report precision, recall, F1, and FPR on this mixed corpus for the Agentic Oracle and all baselines. Second, we will add an explicit discussion of the self-referential limitation in §4.1.5, acknowledging that the primary corpus reflects LogicHunter's generator distribution and that the mixed external corpus provides a complementary, independent evaluation. We note that the 30 developer-confirmed bugs (§4.3) already provide strong external validation of practical utility, but the referee is correct that the oracle accuracy metrics themselves need an independent mixed corpus. We will add this corpus and report results in the revised manuscript. revision: yes

  2. Referee: §4.1.5: The labeling was performed by 'two developers' who are co-authors. The paper should disclose this conflict of interest and discuss safeguards against labeling bias.

    Authors: We agree. The two labelers are indeed co-authors (the first two authors). We will revise §4.1.5 to explicitly disclose this and describe the safeguards we applied: (1) labeling was performed independently before any comparison or discussion, (2) labelers were given the test code, failure traceback, and official documentation but were not shown the Agentic Oracle's verdict during labeling, (3) disagreements were resolved through consensus discussion with the third author as arbiter when needed, and (4) the Cohen's Kappa of 0.77 indicates substantial inter-rater agreement despite independent labeling. We will also acknowledge that independent external labelers would further strengthen the evaluation and note this as a limitation. Additionally, as part of our response to the first comment, the mixed external corpus will include bugs with developer-confirmed ground truth (from merged PRs), removing the need for author labeling on that subset. revision: yes

  3. Referee: Table 7 vs Table 8: The oracle accuracy metrics differ between the two tables. Table 7 reports P=91.17%, R=72.14%, F1=80.49%, FPR=0.21% for the Agentic Oracle, while Table 8 reports P=90.91%, R=71.43%, F1=80.00%, FPR=0.21% for the 'Default Agentic Oracle.' Clarify whether these are the same experiment or different setups.

    Authors: We thank the referee for catching this. The two tables report the same configuration but under different experimental conditions. Table 7 reports the mean ± standard deviation across five independent runs on the full Realistic Failure Corpus (1,000 cases). Table 8 reports results from a single representative run used as the ablation baseline, conducted on the same corpus but without repeated trials. The small numerical differences (e.g., 91.17% vs 90.91% precision) fall within the reported standard deviation (±4.01%). We acknowledge this is insufficiently explained in the current manuscript. We will revise Table 8 to either report mean ± std across five runs for consistency with Table 7, or add a clear note stating that Table 8 uses a single representative run and that the values are within the variance reported in Table 7. We will also ensure the column headers and captions make the relationship explicit. revision: yes

  4. Referee: §4.5: The Agentic Oracle's recall on the post-cutoff known-bug set (60.65%) is lower than passive LLM judges (67-94%). The paper should discuss whether this precision-recall tradeoff is inherent to the design or reflects self-referential corpus bias.

    Authors: The referee raises an important point that we should address more thoroughly. We believe both factors contribute. First, the precision-recall tradeoff is partly inherent to the design: the Agentic Oracle requires source-code evidence and executable validation before reporting a bug, which makes it more conservative than passive judges that can classify based on surface-level patterns. Some real-world bugs in the post-cutoff set require complex system construction (e.g., multi-step agent workflows, external service mocking) that the oracle's evidence-gathering tools cannot fully reproduce, leading to missed bugs. Second, we cannot rule out that self-referential corpus bias contributes: the Realistic Failure Corpus reflects LogicHunter's generator distribution, and the oracle's investigation patterns may be better calibrated for the types of failures the generator produces. The fact that recall drops from 72.14% on the primary corpus to 60.65% on the external known-bug set is consistent with this interpretation. We will add a dedicated paragraph in §4.5 discussing both factors, and we will explicitly state that the precision-recall tradeoff on external bugs is an open question that the mixed external corpus (from our response to Comment 1) will help illuminate. We will also note that the 60.65% recall on externally-sourced real-world bugs, while lower than passive judges, still represents meaningful bug-finding capability, and that the Agentic Oracle's dramatically lower FPR (0.04% vs 1.03-2.07% for GPT-5.2 baselines on the hard-negative set) makes it more suitable for automated deployment where false-positive cost dominates. revision: yes

Circularity Check

0 steps flagged

No significant circularity; one minor self-citation that is not load-bearing for the central claim.

full rationale

The paper's central claims rest on two pillars: (1) 40 previously unknown bugs discovered, 30 confirmed and 26 fixed by external framework developers — this is externally validated by the framework maintainers themselves, not by the authors. (2) The Agentic Oracle achieves 91.17% precision on the Realistic Failure Corpus (§4.1.5, Table 7). While this corpus is sampled from LogicHunter's own generator output and labeled by two co-authors (Cohen's Kappa 0.77), this is not circular in the strict sense: the oracle is not defined in terms of the generator's output, no parameters are fitted to the evaluation corpus, and the oracle's classification is based on independent tool-based investigation (code_search, doc_search, run_code) rather than on any function of the generator. The precision is a genuine measurement against human labels, not a tautological consequence of the generation process. The paper also provides two supplementary datasets for external validation: a post-cutoff known-bug set (31 real-world bugs, recall 60.65%) and a hard-negative set (484 TitanFuzz failures, FPR 0.04%). Neither simultaneously measures both precision and recall, which is a generalizability limitation, but not a circularity defect. The one self-citation is reference [46] (Xue, Zhao, Wang et al.), co-authored by two of the present paper's authors, used in §2.2 for background motivation and in §4.1.5 for the methodology of mining bug-fixing PRs on the supplementary dataset only. This citation is not load-bearing for the central precision or bug-discovery claims. The derivation chain from generator → deterministic filtering → Agentic Oracle → human-labeled evaluation does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The free parameters (alpha, M_max, k_min, k_max, HCC threshold) are hyperparameters with stated defaults but some unspecified values. The domain assumptions are reasonable for the testing context but the labeler reliability axiom is load-bearing for the oracle evaluation. The invented entities are well-motivated and partially independently validated.

free parameters (6)
  • alpha (global scale factor) = not specified
    Controls mutation budget allocation N_mut = floor(alpha * C(api) / M_max) * M_max in §3.2.3. Value not reported.
  • M_max (max mutations per LLM call) = not specified
    Upper bound on mutations per API in §3.2.3. Value not reported.
  • C(api) (API complexity score) = 6 <= C(api) <= 30
    Per-API complexity metric used for resource allocation in §3.2.3. Range given but exact computation method not fully specified.
  • k_min (minimum reasoning rounds) = 5
    Minimum investigation rounds before verdict allowed in §3.3.2. Ablated in Figure 5; default is 5.
  • k_max (maximum reasoning rounds) = 8
    Maximum reasoning rounds in the Agentic Oracle FSM in §3.3.2.
  • HCC threshold (consensus sessions) = 4
    Number of independent Agentic Oracle sessions required to all agree before flagging a bug in §3.3.2.
axioms (5)
  • domain assumption LLM agent framework bugs manifest as conventional software defects (API misuse, logic errors, etc.) rather than novel defect classes
    Stated in §2.2, citing empirical study [46]. This motivates the entire approach — if framework bugs required fundamentally different testing techniques, the specification-driven approach would not apply.
  • domain assumption Pydantic schemas and type hints accurately encode the valid input space for framework APIs
    Assumed throughout §3.2. If schemas are incomplete or overly permissive, valid-by-construction generation may still produce semantically invalid inputs.
  • domain assumption Two developer labelers can reliably distinguish genuine bugs from non-bugs in the 1,000-case oracle corpus
    §4.1.5 reports Cohen's Kappa of 0.77. The entire oracle evaluation depends on these labels being correct.
  • domain assumption GPT-5-mini at temperature 0 produces sufficiently deterministic outputs for oracle use
    §4.1.2 states temperature=0 to 'minimize hallucinations and ensure deterministic judgments.' HCC with 4 sessions is used because determinism is not guaranteed.
  • domain assumption Real-world repository usage patterns represent correct API orchestration
    §3.2.1 mines 'high-quality repositories such as official integrations and top-rated framework projects' as usage templates. If these contain anti-patterns, the generator will propagate them.
invented entities (3)
  • Agentic Oracle independent evidence
    purpose: Active LLM-based test oracle that retrieves documentation, inspects source code, and executes reproduction scripts to diagnose failures
    The Agentic Oracle is the central contribution. Its effectiveness is independently validated by 30 developer-confirmed bugs and the supplementary post-cutoff known-bug set (60.65% recall on real-world bugs).
  • Behavioral Probes independent evidence
    purpose: Expectation-oriented assertions attached to generated tests to surface silent semantic failures
    Probes are validated by the 32 silent failure bugs discovered. The ablation in Table 8 (no_tools variant) shows probes alone maintain 90.91% precision but only 35.71% recall, confirming they surface candidates rather than serving as definitive oracles.
  • Six-Dimensional Defect Taxonomy no independent evidence
    purpose: Classification scheme (Internal Error, Doc Mismatch, Data Integrity, Robustness, Usability, Misuse) to mitigate overly conservative bias in LLM oracles
    The taxonomy is designed by the authors and aligned with ISO/IEC 25010. Its effectiveness is only measured within the paper's own evaluation corpus. No external validation of the taxonomy's completeness or correctness is provided.

pith-pipeline@v1.1.0-glm · 27842 in / 3538 out tokens · 457237 ms · 2026-07-08T13:34:13.004675+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 15 internal anchors

  1. [1]

    Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

    Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey.IEEE Transactions on Software Engineering41, 5 (2015), 507–525. doi:10.1109/TSE.2014.2372785

    work page doi:10.1109/tse.2014.2372785 2015

  2. [2]

    Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineering Agents: A Study of Thought-Action- Result Trajectories. arXiv:2506.18824 [cs.SE] https://arxiv.org/abs/2506.18824

    work page arXiv 2025

  3. [3]

    2022.LangChain

    Harrison Chase. 2022.LangChain. https://github.com/langchain-ai/langchain

    work page 2022

  4. [4]

    Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities.ACM Comput. Surv.51, 1, Article 4 (Jan. 2018), 27 pages. doi:10.1145/3143561

    work page doi:10.1145/3143561 2018

  5. [5]

    Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572–576

    work page 2024

  6. [6]

    2025.Pydantic

    Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, Alex Hall, and Victorien Plot. 2025.Pydantic. https: //github.com/pydantic/pydantic

    work page 2025

  7. [7]

    Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. 2023. Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. InProceedings of the 32nd ACM International Symposium on Software Testing and Analysis (ISSTA 2023). 423–435

    work page 2023

  8. [8]

    Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2024. Large Language Models are Edge-Case Generators: Crafting Unusual Programs for Fuzzing Deep Learning Libraries. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computin...

    work page doi:10.1145/3597503.3623343 2024

  9. [9]

    Google. [n. d.]. Atheris: Coverage -Guided Python Fuzzing Engine. https://github.com/google/atheris. Accessed: 2025-07-07

    work page 2025

  10. [10]

    Hayet, A

    I. Hayet, A. Scott, and M. d’Amorim. 2025. ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance.IEEE Transactions on Software Engineering51, 1 (2025), 305–319

    work page 2025

  11. [11]

    Y. He, J. Huang, H. Yu, and T. Xie. 2024. An Empirical Study on Focal Methods in Deep-Learning-Based Approaches for Assertion Generation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1750–1771

    work page 2024

  12. [12]

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber

    work page

  13. [13]

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv:2308.00352 [cs.AI] https: //arxiv.org/abs/2308.00352

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Documen- tation on LLM-based Test Oracle Generation. arXiv:2412.09360 [cs.SE] https://arxiv.org/abs/2412.09360

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, et al . 2024. Large Language Models for Software Engineering: A Systematic Literature Review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 220

    work page 2024

  16. [16]

    Zhang, Zheng Lin, Meng Luo, Qianru Zhang, and See-Kiong Ng

    Dong Huang, Mingzhe Du, Jie M. Zhang, Zheng Lin, Meng Luo, Qianru Zhang, and See-Kiong Ng. 2025. Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis. arXiv:2510.26423 [cs.SE] https://arxiv.org/abs/2510.26423

    work page arXiv 2025

  17. [17]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798 [cs.CL] https://arxiv.org/abs/2310.01798

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    CrewAI Inc. 2025. CrewAI: A lean, lightning-fast Python framework for autonomous AI agents. https://github.com/ crewAIInc/crewAI. Accessed: 2025-07-07

    work page 2025

  19. [19]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770 [cs.CL] https://arxiv.org/abs/ 2310.06770

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    Do LLMs generate test oracles that capture the actual or the expected program behaviour?

    M. Konstantinou, R. Degiovanni, and M. Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv:2410.21136

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Socio-technical anti-patterns in building ml-enabled software: Insights from leaders on the forefront,

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 919–931. doi:10.1109/ICSE48619.2023.00085

    work page doi:10.1109/icse48619.2023.00085 2023

  22. [22]

    W. Li, H. Yang, X. Luo, L. Cheng, and H. Cai. 2023. PyRTFuzz: Detecting Bugs in Python Runtimes via Two-Level Collaborative Fuzzing. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. 1645–1659. , Vol. 1, No. 1, Article . Publication date: July 2026. 22 M. Long, Y. Zhao, and H. Wang

    work page 2023

  23. [23]

    Xia Li, Jiajun Jiang, Samuel Benton, Yingfei Xiong, and Lingming Zhang. 2021. A Large-scale Study on API Misuses in the Wild. In2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 241–252. doi:10.1109/ ICST49551.2021.00034

    work page arXiv 2021

  24. [24]

    Zichuan Li, Jian Cui, Xiaojing Liao, and Luyi Xing. 2025. Les Dissonances: Cross-Tool Harvesting and Polluting in Multi-Tool Empowered LLM Agents. arXiv:2504.03111 [cs.CR] https://arxiv.org/abs/2504.03111

    work page arXiv 2025

  25. [25]

    2022.LlamaIndex

    Jerry Liu. 2022.LlamaIndex. https://github.com/jerryjliu/llama_index

    work page 2022

  26. [26]

    LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs

    Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. 2025. LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs. arXiv:2404.10304 [cs.SE] https://arxiv.org/abs/2404.10304

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Lost in the Middle: How Language Models Use Long Contexts

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172 [cs.CL] https://arxiv.org/abs/2307.03172

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts.Transactions of the Association for Computational Linguistics12 (2024), 157–173. doi:10.1162/tacl_a_00638

    work page doi:10.1162/tacl_a_00638 2024

  29. [29]

    Tong Liu, Zizhuang Deng, Guozhu Meng, Yuekang Li, and Kai Chen. 2024. Demystifying RCE Vulnerabilities in LLM-Integrated Apps. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security (CCS ’24). ACM, 1716–1730. doi:10.1145/3658644.3690338

    work page doi:10.1145/3658644.3690338 2024

  30. [30]

    Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: automated unit test generation for Python. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE ’22). ACM, 168–172. doi:10.1145/3510454.3516829

    work page doi:10.1145/3510454.3516829 2022

  31. [31]

    Large Language Model Agent: A Survey on Methodology, Applications and Challenges

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, and Ming Zhang. 2025. Large Language Model Agent: A...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    Rajesh Malviya, Vishal Javalkar, and Rano Malviya. 2024. Scalability and Performance Benchmarking of LangChain, LlamaIndex, and Haystack for Enterprise AI Customer Support Systems. doi:10.21428/e90189c8.43aeb06e

    work page doi:10.21428/e90189c8.43aeb06e 2024

  33. [33]

    Ivan Milev, Mislav Balunović, Maximilian Baader, and Martin Vechev. 2025. ToolFuzz – Automated Agent Tool Testing. arXiv:2503.04479 [cs.AI] https://arxiv.org/abs/2503.04479

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    Molina, A

    F. Molina, A. Gorla, and M. d’Amorim. 2025. Test Oracle Automation in the Era of LLMs.ACM Transactions on Software Engineering and Methodology34, 1 (2025), 1

    work page 2025

  35. [35]

    Shiwen Ou, Yuwei Li, Lu Yu, Chengkun Wei, Tingke Wen, Qiangpu Chen, Yu Chen, Haizhi Tang, and Zulie Pan. 2026. MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing.IEEE Transactions on Software Engineering52, 1 (Jan. 2026), 360–375. doi:10.1109/tse.2025.3619966

    work page doi:10.1109/tse.2025.3619966 2026

  36. [36]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789 [cs.AI] https://arxiv.org/a...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    Alfin Wijaya Rahardja, Junwei Liu, Weitong Chen, Zhenpeng Chen, and Yiling Lou. 2025. Can Agents Fix Agent Issues? arXiv:2505.20749 [cs.AI] https://arxiv.org/abs/2505.20749

    work page arXiv 2025

  38. [38]

    Sebastián Ramírez. [n. d.].FastAPI. https://github.com/fastapi/fastapi

    work page

  39. [39]

    Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter. 2015. Tricorder: building a program analysis ecosystem. InProceedings of the 37th International Conference on Software Engineering - Volume 1 (Florence, Italy)(ICSE ’15). IEEE Press, 598–608

    work page 2015

  40. [40]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessí, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: language models can teach themselves to use tools. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates ...

    work page 2023

  41. [41]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (March 2024). doi:10.1007/s11704-024-40231-1

    work page doi:10.1007/s11704-024-40231-1 2024

  42. [42]

    Yuxuan Wang, Jingshu Chen, and Qingyang Wang. 2025. Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects. arXiv:2505.15088 [cs.SE] https://arxiv.org/abs/2505.15088

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. 2024. StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows. arXiv:2403.11322 [cs.CL] https://arxiv.org/abs/2403.11322 , Vol. 1, No. 1, Article . Publication date: July 2026. LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle 23

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, a...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. 2024. Fuzz4ALL: Universal Fuzzing with Large Language Models. In2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). 1547–1559

    work page 2024

  46. [46]

    Qinghua Xu, Guancheng Wang, Lionel Briand, and Kui Liu. 2025. Hallucination to Consensus: Multi-Agent LLMs for End-to-End Test Generation with Accurate Oracles. arXiv:2506.02943 [cs.SE] https://arxiv.org/abs/2506.02943

    work page arXiv 2025

  47. [47]

    Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)(Seoul, Korea, Republic of). IEEE Press, 3369–3380. doi:10.1109/ASE63991.2025.00278

    work page doi:10.1109/ase63991.2025.00278 2025

  48. [48]

    Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2025. Advancing Code Coverage: Incorporating Program Analysis with Large Language Models.ACM Trans. Softw. Eng. Methodol.(July 2025). doi:10.1145/3748505 Just Accepted

    work page doi:10.1145/3748505 2025

  49. [49]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering(NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 1601, 125 pages

    work page 2024

  50. [50]

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. InterCode: standardizing and benchmarking interactive coding with execution feedback. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 1035, 29 pages

    work page 2023

  51. [51]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.03629

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  52. [52]

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2024. No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. arXiv:2305.04207 [cs.SE] https://arxiv.org/abs/ 2305.04207

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  53. [53]

    Zhang, S

    K. Zhang, S. Wang, J. Han, X. Zhu, X. Li, S. Wang, et al . 2025. Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 508–520

    work page 2025

  54. [54]

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. 2025. Which Agent Causes Task Failures and When? on Automated Failure Attribution of LLM Multi-Agent Systems. InProceedings of the 42nd International Conference on Machine Learning, Vol. 267. 76583–76599. https://m...

    work page 2025

  55. [55]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(N...

    work page 2023