Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Coding-agent failures start early, are mostly epistemic, and stay hidden until recovery is gone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 02:29 UTC pith:YSME6DCN

load-bearing objection Solid large-scale process anatomy of CLI agent failures; the early-epistemic pattern is real on this corpus, with the main soft spot being hindsight timestamps on a complete-grid filter rather than a circular core. the 3 major comments →

arxiv 2607.09510 v1 pith:YSME6DCN submitted 2026-07-10 cs.SE cs.AI

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Xiangxin Zhao , Han Li , Shuaiting Li , Tianyi Zhao , Earl T. Barr , Federica Sarro , He Ye This is my paper
classification cs.SE cs.AI
keywords LLM coding agentsCLI agentsfailure trajectoriesepistemic errorsTerminal-Benchrecovery analysisprocess-oriented evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that failures of command-line coding agents should be studied as temporal processes, not only as final pass/fail labels. Across nearly 1,800 annotated trajectories from seven frontier models and three agent scaffolds on Terminal-Bench, the authors show that decisive mistakes usually arrive within the first few steps, are most often epistemic—agents acting on false premises or neglecting available information rather than lacking skill—and often remain invisible until the run is already unrecoverable. Successful runs also err, but recover faster and act on error signals far more consistently. The practical claim is that reliability gains will come from earlier validation, assumption checking, and intervention, not from waiting for the final test outcome.

Core claim

On 1,184 failed CLI coding-agent trajectories, decisive errors begin early (median step 7 of a median 27-step failed run), leave only a short recovery window before lock-in (median one step), and typically surface much later, while 57.9% of decisive errors are epistemic—especially false premises (30.7%)—rather than competence or environment limits. Recovery itself is a process: most failed runs keep spending steps after lock-in, successful recoveries are short, and fabrication of success often appears exactly when recovery becomes impossible.

What carries the argument

A three-timestamp process annotation of each failed trajectory: t_err (the retrospectively decisive error), t_lock (the step after which no recovery is observed), and t_obs (first observable failure signal), which define the recovery window and observability lag and support a root-cause taxonomy plus post-lock-in recovery patterns.

Load-bearing premise

That a single decisive error and lock-in step, labeled with full-trajectory hindsight on only the tasks that finished under every model–scaffold pair, truly mark when real-time failure becomes unrecoverable for CLI agents.

What would settle it

If independent monitors or human re-annotation on held-out or live trajectories show that failures labeled early by this scheme are routinely recovered after the supposed lock-in step, or that competence and environment errors dominate once incomplete and timed-out runs are included, the process claims would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a large-scale empirical study of CLI coding-agent failures as a temporal process rather than a final outcome. From 3,843 raw Terminal-Bench runs across seven frontier models and three scaffolds (OpenHands, MiniSWE, Terminus2), the authors retain 1,794 complete trajectories (1,184 failed) on 89 tasks present under all 21 model–scaffold pairs, then annotate >63,000 steps with an LLM-assisted dual-human pipeline. They introduce a three-timestamp process model (decisive error t_err, empirical lock-in t_lock, first observable signal t_obs), a nine-type root-cause taxonomy (epistemic / competence / environment), and post-lock-in recovery patterns, yielding 14 findings. The central claim is that failures are predominantly epistemic (57.9%), begin early (median t_err at step 7), leave a short fix window, and often stay hidden until recovery is no longer possible, so reliability work should emphasize earlier validation rather than outcome-only evaluation.

Significance. If the process anatomy holds, the paper supplies a useful shift for coding-agent reliability research: from post-hoc failure labels to onset, recoverability, and observability. Strengths include scale relative to prior trajectory studies, dual human adjudication with reported Cohen’s κ 0.78–0.94, an explicit taxonomy and recovery typology, a cross-system consistency check (RQ4), a prefix-monitor experiment that honestly shows low real-time recall, and a public annotated dataset of 1,794 runs. These are concrete, reusable contributions for evaluation design, monitoring, and agent scaffolding in terminal environments. The work is observational rather than causal, but that is appropriate for the stated empirical goals.

major comments (3)
  1. [Section II-E (Methodology for RQ1); Finding 1] Section II-E defines t_err retrospectively as “the error that determines the eventual failure” over the full trajectory, and t_lock as the first step after which no recovery is observed. Finding 1’s median onset of 7 steps and the short fix window rest on these labels. The manuscript should report how often multiple candidate decisive errors exist, the exact rule used to pick among them, and step-level agreement on t_err/t_lock (beyond category κ). A brief sensitivity check—e.g., using the earliest vs. latest plausible decisive error, or excluding multi-error tails—would show whether the early-onset median is robust or an artifact of hindsight root-cause selection. Without that, the call for “earlier validation” is harder to calibrate.
  2. [Section II-C (Trajectory Collection & Filtering); V Threats to Validity] Section II-C filters from 240 tasks / 3,843 raw runs to the 89-task complete 21-system grid (1,794 trajectories). This is reasonable for RQ4, but the early-onset and epistemic-share claims are then conditioned on that subset. Threats to Validity notes difficulty balance, yet does not compare process statistics (t_err, t_lock−t_err, epistemic fraction) on the discarded incomplete or non-aligned runs where partial annotation is feasible. Even a limited comparison would address whether the complete-grid filter systematically advances t_err or concentrates epistemic failures. This is load-bearing for generalizing beyond the aligned 89 tasks.
  3. [Section II-E (prefix monitor); Finding 4] Finding 4’s prefix monitor (Claude Sonnet 4.6) is a valuable online check, but Claude Sonnet 4 is also one of the seven systems under study. Detection metrics may be inflated or distorted on same-family trajectories. Please either stratify monitor performance by whether the trajectory’s foundation model is Claude-family vs. other, or re-evaluate with a held-out model family. Also clarify that labels are relative to hindsight t_lock, so low lead time partly restates the observability lag rather than independently validating it.
minor comments (6)
  1. [Table I] Table I’s “Failure as Process” column is binary and slightly overstates uniqueness; some cited works analyze trajectories over time even if they do not use a three-timestamp model. Soften the contrast or define the criterion more precisely.
  2. [Figure 4] Figure 4 truncates the x-axis at 55 while reporting means on full data; state the max step and how many trajectories exceed 55 so the CDF ceiling is interpretable.
  3. [Table III; Section II-G] Table III’s five post-lock-in patterns are assigned by a “first-match rule” on dominant behavior and failure awareness; a short appendix example of the decision order would improve reproducibility.
  4. [Tables IV–V; Figure 2] Pass rates in Table IV and epistemic shares in Table V would benefit from n per cell (failed trajectories), not only percentages, given uneven failure counts in Figure 2.
  5. [Abstract; Section II Terminology; RQ1 headings] Terminology: “error” vs. “failure” is carefully defined early; keep that distinction consistent in the abstract and Finding titles (some headings still say “failure occurrence” for t_err).
  6. [Section IV-A] Related Work could more explicitly position the three-timestamp model against MAST and AgentLens process-quality notions so the novelty claim is easier to audit.

Circularity Check

0 steps flagged

Empirical annotation study: medians and taxonomy shares are measured counts, not quantities forced by definition or fit.

full rationale

This paper is a large-scale empirical trajectory study, not a first-principles derivation. Its load-bearing claims (median t_err = 7, 57.9% epistemic, short fix window, late observability, recovery patterns) are frequencies and order statistics over 1,184 human-finalized annotations of failed runs, plus comparisons on successful runs. Defining t_err retrospectively as the error that determines eventual failure guarantees that every failed trajectory has some decisive step; it does not force that step to be early, short-windowed, or epistemic—those are data. t_lock is labeled as empirically unrecoverable (no later recovery observed), which is an operational annotation criterion, not a prediction of lock-in from a fitted model. The 89-task complete-grid filter is a selection/external-validity choice, not a circular reduction of outputs to inputs. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors to forbid alternatives, and no ansatz smuggled in via self-citation that carries the central statistics. Self-citations appear only as ordinary related work. Methodological risks (hindsight labels, filter bias, hard real-time detection) are construct/external-validity issues, not circularity under the stated patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 3 invented entities

This is an observational empirical study. The central claims rest on domain conventions for what counts as a step, an error, and a failed run, plus the authors’ process annotation schema. There are no fitted physical constants; the main invented constructs are the three timestamps, the root-cause taxonomy, and the post-lock-in behavior patterns used to structure the 14 findings.

axioms (4)
  • domain assumption A trajectory step is one agent turn with one reasoning and one action; final success is defined solely by Terminal-Bench executable tests on the final environment state.
    Stated in Terminology and Section II-B; all timing and pass-rate claims inherit this operationalization.
  • ad hoc to paper A single decisive error t_err can be identified retrospectively as the error that determines eventual failure, and t_lock is the first step after which no recovery is observed.
    Core of the RQ1 methodology (Section II-E); the process claims depend on this hindsight labeling convention.
  • domain assumption Incomplete or abnormally terminated runs can be dropped, and analysis restricted to the 89 tasks with (nearly) complete 21-system grids, without invalidating process conclusions.
    Section II-C filtering; threats section acknowledges possible bias but treats incomplete runs as out of scope.
  • ad hoc to paper Root causes partition into epistemic, competence, and environment categories with the nine fine-grained types in Table II.
    RQ2 taxonomy is author-constructed; Finding 5–6 percentages are defined relative to it.
invented entities (3)
  • Three-timestamp failure process (t_err, t_lock, t_obs) plus fix window and observability lag no independent evidence
    purpose: Decompose failure into onset, loss of recoverability, and first observable signal for process analysis.
    Introduced in Section II-E and Figure 3; all RQ1 findings are defined on these labels.
  • Epistemic / competence / environment root-cause taxonomy (9 types) no independent evidence
    purpose: Classify why the decisive error occurs rather than only its final symptom.
    Table II; Finding 5–6 and cross-system Finding 14 depend on this partition.
  • Five post-lock-in recovery behavior patterns (including wrong-cause repair and fabrication) no independent evidence
    purpose: Characterize wasted execution after empirical unrecoverability.
    Table III and Findings 7–8, 12; used to quantify share vs waste.

pith-pipeline@v1.1.0-grok45 · 22503 in / 2754 out tokens · 38059 ms · 2026-07-13T02:29:39.046446+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

Figures

Figures reproduced from arXiv: 2607.09510 by Earl T. Barr, Federica Sarro, Han Li, He Ye, Shuaiting Li, Tianyi Zhao, Xiangxin Zhao.

Figure 1
Figure 1. Figure 1: Overview of our study methodology. We collect execution trajectories from seven frontier models across three coding [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of the 1,184 failed trajectories used in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distributions of the three annotated failure timestamps [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A representative silent failure with early lock-in and a late signal. At step 2, the agent’s cd into the repository silently fails (terr). With no error output, it assumes it is inside the repository and proceeds to build the entire site in /app. By step 6 the run has locked in (tlock), and every later action extends the mistake rather than revisiting it. No signal appears until git add -A fails with not a… view at source ↗
Figure 6
Figure 6. Figure 6: Representative false-premise error (Epistemic). Af￾ter observing the message sudo: not found, the agent incor￾rectly concludes that it lacks permission to perform the task. The decisive error is therefore not the command itself, but the incorrect assumption that ”sudo missing” implies ”permission denied”. Although the required information is already available in the environment, this false premise subseque… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 1 canonical work pages

  1. [1]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” inProceedings of the Neural Information Processing Systems (NeurIPS), 2024. [Online]. Available: https://openreview.net/forum?id=mXpq6ut8J3

    2024

  2. [2]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). Association for Computing Machinery, 2024, pp. 1592–1604. [Online]. Available: https://doi.org/10.1145/3650212.3680384

    doi:10.1145/3650212.3680384 2024

  3. [3]

    Repo2run: Automated building executable environment for code repository at scale,

    R. Hu, C. Peng, X. Wang, J. Xu, and C. Gao, “Repo2run: Automated building executable environment for code repository at scale,” inAdvances in Neural Information Processing Systems, 2025, neurIPS 2025 Spotlight Poster. [Online]. Available: https: //arxiv.org/abs/2502.13681

    arXiv 2025

  4. [4]

    Agentless: Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024. [Online]. Available: https://arxiv.org/abs/2407.01489

    Pith/arXiv arXiv 2024

  5. [5]

    Claude code: Overview,

    Anthropic, “Claude code: Overview,” 2026, accessed: 2026-04-09. [Online]. Available: https://code.claude.com/docs/en/overview

    2026

  6. [6]

    Openai codex for developers,

    OpenAI, “Openai codex for developers,” 2026, accessed: 2026-04-09. [Online]. Available: https://developers.openai.com/codex

    2026

  7. [7]

    Gemini code assist cli documentation,

    Google, “Gemini code assist cli documentation,” 2026, accessed: 2026-04-09. [Online]. Available: https://developers.google.com/ gemini-code-assist/docs/gemini-cli

    2026

  8. [8]

    Basic concepts and taxonomy of dependable and secure computing,

    A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, “Basic concepts and taxonomy of dependable and secure computing,”IEEE Transactions on Dependable and Secure Computing, vol. 1, no. 1, pp. 11–33, 2004. [Online]. Available: https://doi.org/10.1109/TDSC.2004.2

    doi:10.1109/tdsc.2004.2 2004

  9. [9]

    Where llm agents fail and how they can learn from failures,

    K. Zhu, Z. Liu, B. Li, M. Tian, Y . Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, X. Ma, X. Yu, G. Ramesh, J. Wu, Z. Liu, P. Lu, J. Zou, and J. You, “Where llm agents fail and how they can learn from failures,”arXiv preprint arXiv:2509.25370, 2025. [Online]. Available: https://arxiv.org/abs/2509.25370

    arXiv 2025

  10. [10]

    Understanding software engineering agents: A study of thought–action–result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought–action–result trajectories,” inProceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025. [Online]. Available: https://arxiv.org/abs/ 2506.18824

    arXiv 2025

  11. [11]

    Understanding code agent behaviour: An empirical study of success and failure trajectories,

    O. Majgaonkar, Z. Fei, X. Li, F. Sarro, and H. Ye, “Understanding code agent behaviour: An empirical study of success and failure trajectories,” arXiv preprint arXiv:2511.00197, 2025. [Online]. Available: https: //arxiv.org/abs/2511.00197

    arXiv 2025

  12. [12]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y . Chen, and Q. Wu, “Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025, spotlight poster

    2025

  13. [13]

    Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis,

    Y . Ge, L. Xie, Z. Li, Y . Pei, and T. Zhang, “Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis,”arXiv preprint arXiv:2509.13782, 2025

    arXiv 2025

  14. [14]

    Trajaudit: Automated failure diagnosis for agentic coding systems,

    M. Wang, X. Xie, and Y . Huo, “Trajaudit: Automated failure diagnosis for agentic coding systems,”arXiv preprint arXiv:2605.26563, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent llm systems fail?” inAdvances in Neural Information Processing Systems, 2025, datasets and Benchmarks Track spotlight. [Online]. Available: https://openreview.net/forum?id=...

    2025

  16. [16]

    Dissecting bug triggers and failure modes in modern agentic frameworks: An empirical study,

    X. Zhang, H. Zhang, and S. H. Tan, “Dissecting bug triggers and failure modes in modern agentic frameworks: An empirical study,”arXiv preprint arXiv:2604.08906, 2026. [Online]. Available: https://arxiv.org/abs/2604.08906

    Pith/arXiv arXiv 2026

  17. [17]

    Agentlens: Revealing the lucky pass problem in swe-agent evaluation,

    P. Sahoo, G. Mittal, X. Li, S. Ma, B. Steenhoek, P. Lin, and Y . Hu, “Agentlens: Revealing the lucky pass problem in swe-agent evaluation,”arXiv preprint arXiv:2605.12925, 2026. [Online]. Available: https://arxiv.org/abs/2605.12925

    Pith/arXiv arXiv 2026

  18. [18]

    Beyond resolution rates: Behavioral drivers of coding agent success and failure,

    T. Mehtiyev and W. Assunc ¸ ˜ao, “Beyond resolution rates: Behavioral drivers of coding agent success and failure,” arXiv preprint arXiv:2604.02547, 2026. [Online]. Available: https://arxiv.org/pdf/2604.02547

    Pith/arXiv arXiv 2026

  19. [19]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chenet al., “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” inProceedings of the International Conf...

    2026

  20. [20]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inProceedings of the International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview. net/forum?id=WE vluYUL-X

    2023

  21. [21]

    Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces,

    Y . Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, J. Sun, Y . Xiao, S. Zhou, W. Wu, Y . Liu, P. Liu, Y . Qiao, S. Zhang, and K. Zhang, “Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces,”arXiv preprint arXiv:2602.14337, 2026. [Online]. Available: https://arxiv.org...

    arXiv 2026

  22. [22]

    Cli-gym: Scalable cli task generation via agentic environment inversion,

    Y . Lin, H. Wang, S. Wu, L. Fan, F. Pan, S. Zhao, and D. Tu, “Cli-gym: Scalable cli task generation via agentic environment inversion,”arXiv preprint arXiv:2602.10999, 2026. [Online]. Available: https://arxiv.org/abs/2602.10999

    arXiv 2026

  23. [23]

    Endless terminals: Scaling rl environments for terminal agents,

    K. Gandhi, S. Garg, N. D. Goodman, and D. Papailiopoulos, “Endless terminals: Scaling rl environments for terminal agents,” arXiv preprint arXiv:2601.16443, 2026. [Online]. Available: https: //arxiv.org/abs/2601.16443

    arXiv 2026

  24. [24]

    Envbench: A benchmark for automated environment setup,

    A. Eliseeva, A. Kovrigin, I. Kholkin, E. Bogomolov, and Y . Zharov, “Envbench: A benchmark for automated environment setup,” in Proceedings of the DL4Code Workshop at ICLR, 2025, workshop paper at ICLR 2025. [Online]. Available: https://arxiv.org/abs/2503.14443

    Pith/arXiv arXiv 2025

  25. [25]

    Tua-bench: A benchmark for general-purpose terminal-use agents,

    S. Chen, L. Wang, X. Yang, Z. Liu, Y . Cong, Y . Ji, F. Zhou, X. Zhang, F. Yang, and B. Zeng, “Tua-bench: A benchmark for general-purpose terminal-use agents,”arXiv preprint arXiv:2606.28480, 2026. [Online]. Available: https://arxiv.org/abs/2606.28480

    Pith/arXiv arXiv 2026

  26. [26]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, and J. Lin, “Openhands: An open platform for ai software developers as generalist agents,” inProceedings of the International Conference on Learning Representations (ICLR), 2025...

    2025

  27. [27]

    Prometheus: Towards long-horizon codebase navigation for repository-level problem solving,

    Y . Pan, Z. Chen, S. Lu, Z. Chu, X. Li, H. Li, Y . Feng, C. Le Goues, F. Sarro, M. Monperrus, and H. Ye, “Prometheus: Towards long-horizon codebase navigation for repository-level problem solving,”arXiv preprint arXiv:2507.19942, 2025. [Online]. Available: https://arxiv.org/abs/2507.19942

    arXiv 2025

  28. [28]

    Magis: Llm-based multi-agent framework for github issue resolution,

    W. Tao, Y . Zhou, Y . Wang, W. Zhang, H. Zhang, and Y . Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,” inAdvances in Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=qevq3FZ63J

    2024

  29. [29]

    Codetracer: Towards traceable agent states,

    H. Li, Y . Yao, L. Zhu, R. Feng, H. Ye, J. Wang, Y . He, P. Zou, L. Zhang, X. Lei, H. Huang, K. Deng, M. Sun, Z. Zhang, H. Ye, and J. Liu, “Codetracer: Towards traceable agent states,”arXiv preprint arXiv:2604.11641, 2026. [Online]. Available: https://arxiv.org/abs/2604.11641

    Pith/arXiv arXiv 2026

  30. [30]

    Harbor framework documentation,

    Harbor Team, “Harbor framework documentation,” 2026, accessed: 2026-04-09. [Online]. Available: https://www.harborframework.com/ docs

    2026

  31. [31]

    On the effectiveness of llm-as-a-judge for code generation and summarization,

    G. Crupi, R. Tufano, A. Velasco, A. Mastropaolo, D. Poshyvanyk, and G. Bavota, “On the effectiveness of llm-as-a-judge for code generation and summarization,”IEEE Transactions on Software Engineering, vol. 51, no. 8, pp. 2329–2345, 2025. [Online]. Available: https://doi.org/10.1109/TSE.2025.3586082

    work page doi:10.1109/tse.2025.3586082 2025

  32. [32]

    Robust log-based anomaly detection on unstable log data,

    X. Zhang, Y . Xu, Q. Lin, B. Qiao, H. Zhang, Y . Dang, C. Xie, X. Yang, Q. Cheng, Z. Li, J. Chen, X. He, R. Yao, J.-G. Lou, M. Chintalapati, F. Shen, and D. Zhang, “Robust log-based anomaly detection on unstable log data,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw...

    doi:10.1145/3338906.3338931 2019

  33. [33]

    A survey for llm agent trajectory analysis: From failure attribution to enhancement,

    J. Wang, Y . Wang, M. Chen, X. Xie, C. Chen, F. Mu, Z. Liu, and Q. Wang, “A survey for llm agent trajectory analysis: From failure attribution to enhancement,” https://www.researchgate.net/publication/ 401193207 A Survey for LLM Agent Trajectory Analysis From Failure Attribution to Enhancement, February 2026, preprint on ResearchGate. Accessed: 2026-05-12

    2026

  34. [34]

    How coding agents fail their users: A large- scale analysis of developer-agent misalignment in 20,574 real-world sessions,

    N. Tang, C. Chen, G. Xu, Y . Shi, Y . Huang, C. McMillan, T. Dong, and T. J.-J. Li, “How coding agents fail their users: A large- scale analysis of developer-agent misalignment in 20,574 real-world sessions,”arXiv preprint arXiv:2605.29442, 2026. [Online]. Available: https://arxiv.org/abs/2605.29442

    Pith/arXiv arXiv 2026

  35. [35]

    Exploring autonomous agents: A closer look at why they fail when completing tasks,

    R. Lu, Y . Li, and Y . Huo, “Exploring autonomous agents: A closer look at why they fail when completing tasks,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025), 2025, new Ideas and Emerging Results (NIER) track. [Online]. Available: https://arxiv.org/abs/2508.13143

    Pith/arXiv arXiv 2025

  36. [36]

    Swe-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real- world github issues?” inProceedings of the International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66

    2024

  37. [37]

    Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents,

    K. Zhu, Y . Nie, Y . Li, Y . Huang, J. Wu, J. Liu, X. Sun, Z. Yin, L. Wang, Z. Liu, E. Barsoum, W. Y . Wang, and W. Guo, “Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents,”arXiv preprint arXiv:2602.07274, 2026. [Online]. Available: https://arxiv.org/abs/2602.07274

    arXiv 2026

  38. [38]

    Terminalworld: Benchmarking agents on real-world terminal tasks,

    Z. Chu, J. Hu, X. Jiang, P. Zou, H. Li, C. Peng, P. O’Hearn, E. T. Barr, M. Harman, F. Sarro, and H. Ye, “Terminalworld: Benchmarking agents on real-world terminal tasks,”arXiv preprint arXiv:2605.22535, 2026

    Pith/arXiv arXiv 2026

  39. [39]

    Cli-universe: Towards verifiable task synthesis engine for terminal agents,

    Z. Hua, Y . Yao, W. Xie, Y . Zhao, M. Liu, R. Qiu, Z. Huang, Z. Wang, Y . Ji, Y . Ye, L. Zhu, X. Lei, H. Li, Z. Ma, Z. Wang, Z. Zhang, and J. Liu, “Cli-universe: Towards verifiable task synthesis engine for terminal agents,”arXiv preprint arXiv:2606.22883, 2026. [Online]. Available: https://arxiv.org/abs/2606.22883

    Pith/arXiv arXiv 2026

  40. [40]

    On data engineering for scaling llm terminal capabilities,

    R. Pi, G. Lam, M. Shoeybi, P. Jannaty, B. Catanzaro, and W. Ping, “On data engineering for scaling llm terminal capabilities,” in ICML 2026 Workshop on Data-Centric Learning for Foundation Models and Reasoning, 2026, poster. [Online]. Available: https: //openreview.net/forum?id=BykU9KPyQ1

    2026

  41. [41]

    Establishing best practices in building rigorous agentic benchmarks,

    Y . Zhu, T. Jin, Y . Pruksachatkun, A. K. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, F. Barez, R. Gupta, J. Dhamala, J. Merizian, M. Giulianelli, H. Coppock, C. Ududec, J. S. Sekhon, J. Steinhardt, A. Kellermann, S. Schwettmann, M. Zaharia, I. Stoica, P. Liang, and D. Kang, “Establishing best practices in building rigorous agentic be...

    2025

  42. [42]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=vAElhFcKW6

    2023

  43. [43]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” inProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE/ACM, 2025, pp. 2188–2200. [Online]. Available: https://doi.org/ 10.1109/ICSE55347.2025.00157

    doi:10.1109/icse55347.2025.00157 2025

  44. [44]

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,”arXiv preprint arXiv:2312.13010, 2024

    Pith/arXiv arXiv 2024