Pith. sign in

REVIEW 2 major objections 6 minor 66 references

The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tool architecture alone — how capabilities are organized and exposed to a coding agent — shifts consistency, exploration, and efficiency even when the underlying information and actions are kept equivalent.

desk verdict A genuinely controlled study of coding-agent tool interfaces with credible consistency and efficiency results; the exploration claim, however, is confounded by NLSearch's extra LLM subagent. read the letter →

arxiv 2608.11386 v1 pith:IM27INHB submitted 2026-08-11 cs.SE

Xiangzhe Xu , Hamidreza Saghir , Qianhui Wu , Marc-Alexandre Côté , Tong Wang , Kiran Lakkaraju , Kexin Pei , Xiangyu Zhang This is my paper · ORCID
classification cs.SE
keywords toolarchitecturecodingagentsagentconsistencyrepositoryexplorationCodeActnatural-languagesearchefficiencycognitivescaffolding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the interface through which a coding agent reaches its tools changes behavior once the underlying information and actions are held fixed. It answers yes: comparing six capability-matched tool architectures across three actor models and 11,700 trajectories on repository-level issue fixing, the authors find reliable differences in consistency, exploration, and efficiency. Structured low-level tools improve repeated-attempt consistency by up to 4.7×; a natural-language search interface broadens repository exploration and raises access to relevant files by more than 11%; and a Python code-execution interface keeps task performance roughly equal while using 41.6% fewer steps and 56.3% fewer tokens. Lightweight text-based cognitive scaffolding has little effect. The paper's broader point is that tool architecture is a design variable distinct from tool capability, worth measuring in its own right.

What carries the argument

The machinery is a controlled taxonomy of six tool architectures built on two axes: level of abstraction (raw bash; Atomic low-level primitives; NLSearch natural-language retrieval; Python executable-code blocks) and cognitive scaffolding (HypoTrack hypothesis records; Scratchpad free-form notes). To make architecture the only variable, each setup is deliberately implemented so that its actions can be reproduced in BashOnly: NLSearch uses no embeddings and runs grep-style commands through a subagent, atomic tools map onto documented shell equivalents, and 97 of 100 sampled Python actions map to bash-only operations. The argument then runs through trajectory-level measurements on repeated rollouts: $\text{pass}^k$ for consistency, Jaccard distance over read files and CodeBLEU distance over patches for exploration, and steps plus input/output tokens for efficiency. The repeated-rollout protocol is what lets the paper separate architecture effects from aggregate task performance.

What would settle it

A decisive test is to log the exact effect of every tool call in the Atomic, NLSearch, and Python setups and replay the same effect through BashOnly with identical output text; if $\text{pass}^k$, read-diversity, and step-count gaps disappear when the underlying effects are identical, the architecture explanation is wrong. A simpler version replaces the NLSearch subagent with a deterministic grep wrapper and checks whether the 11% relevant-file recall gain survives.

Watch

Extended reading notes

Core claim

The central discovery is that the same repository information and the same action capabilities, delivered through different interfaces, produce reproducible differences in how coding agents behave. The authors report that Atomic, a small set of constrained tools for search, viewing, targeted string replacement, and file creation layered on bash, is the only architecture that improves $\text{pass}^k$ consistency for all three actor models; for the weakest actor the gain is 0.059–0.074 across $\text{pass}^5$ to $\text{pass}^9$, and the mechanism they identify is a sharp drop in malformed commands and broken edits. NLSearch, a natural-language query interface implemented as a subagent over grep-style search without embeddings, is the only architecture that raises file-read diversity for all three actors, increasing access to high-relevance files by more than 11% while also lowering precision. Python, a CodeAct-style interface in which the agent writes executable code blocks instead of issuing tool calls, achieves similar task performance to BashOnly with 41.6% fewer steps and 56.3% lower token usage, mainly because actors bundle several operations into each interaction. HypoTrack and Scratchpad, lightweight text tools for recording hypotheses and reasoning, do not change actor behavior in this setting, because actors mostly restate reasoning they already produce.

Load-bearing premise

The case rests on the assumption that the six tool setups give the agent the same information and the same actions, so that any behavioral change is caused by the interface rather than by a hidden difference in capability.

Editorial extensions

If this is right

  • If the results hold, coding-agent builders can improve repeated-run reliability by exporting common shell actions as constrained tools, with the largest gains for weaker models whose failures are dominated by malformed commands and broken edits.
  • A natural-language search interface is a reliable way to broaden repository coverage: it raises recall of relevant files for every actor tested, though deployers should expect lower precision and little change in final patch diversity.
  • CodeAct-style interfaces offer an efficiency win without a resolve-rate penalty: the same task is solved in fewer steps and with lower cumulative input cost across all three actors.
  • Lightweight text-based scratchpads and hypothesis trackers are not sufficient to change reasoning behavior on their own, since actors tend to project existing reasoning into them.
  • Overall resolve rate stays broadly similar across architectures, so the effects of tool architecture appear mainly in non-functional properties rather than in simple success rates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the Atomic consistency gain tracks interaction-error rates, the same effect should appear for any tool-using agent whose failures are dominated by command-syntax mistakes; a cheap test is to repackage a general-purpose shell as constrained primitives in a non-coding agent benchmark and compare rerun consistency.
  • Editorial inference: if NLSearch's exploration gain comes from query diversity rather than from embeddings, then a cheaper interface that diversifies the agent's own search formulations could reproduce part of the relevant-file recall increase without a subagent.
  • Editorial inference: the paper's matched-capability recipe — require every non-baseline tool to have a documented baseline equivalent, then spot-check the mapping — is itself a reusable test for deciding whether any agent-tool improvement is caused by architecture or by added capability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies how the organization and exposure of tool capabilities ('tool architecture') affect coding-agent behavior while attempting to hold underlying capabilities fixed. Six architectures are compared on repository-level issue fixing: BashOnly as baseline; Atomic (structured low-level tools); NLSearch (natural-language search via a subagent); Python (CodeAct-style executable code); and two lightweight cognitive scaffolds, HypoTrack and Scratchpad. Using 65 SWE-bench Live instances, three actor models, and 10 rollouts per actor–setup pair (11,700 trajectories total), the paper reports that Atomic improves repeated-attempt consistency (pass^k) across all three actors, NLSearch increases read diversity and relevant-file recall, Python achieves similar resolve rates with fewer steps and lower token cost, and the cognitive scaffolds have little effect. A smaller extension study on additional SWE-bench tasks is reported in Appendix B.6.

Significance. If the central isolation claim holds, this is a valuable empirical contribution: it provides one of the first large-scale, multi-actor controlled comparisons of tool-interface architecture per se, and it makes the effects concrete through repeated-rollout consistency, exploration, and efficiency metrics. The study is unusually careful in several respects: the Python-to-bash capability mapping is manually audited (97/100 actions, Table 11); the relevance proxy is human-validated (Table 6); and the main consistency result is checked with 30 repeats for the two open-weight actors (Table 4). The paper also states and tests falsifiable predictions (e.g., that Atomic reduces low-level interaction errors, that NLSearch increases early-search diversity). The principal weakness is that the capability-equivalence premise is not actually achieved for NLSearch, and the headline deltas are reported without uncertainty quantification; both issues bear directly on the paper's causal claims.

major comments (2)
  1. [§2.1 (NLSearch); §3.3; Table 7] The capability-equivalence premise stated in §2.1 (“we design these setups to keep their capabilities as similar as possible”) is not secured for NLSearch. The implementation is “a subagent with the same actor model, which takes a natural-language query and uses bash commands such as grep to iteratively search for and return potentially relevant snippets.” This gives the NLSearch agent an additional LLM-backed, multi-turn retrieval component whose search reasoning is executed outside the actor's context; BashOnly actors have no comparable resource. Because the abstract and Finding 2 attribute the >11% relevant-file recall gain and the read-diversity gains to natural-language interface architecture rather than to added retrieval compute, this is a load-bearing confound. Please add a control condition that provides the BashOnly actor with a similarly capable search subagent under a structured or non-NL interface, or re-implement NLSearch without an LLM subagent (e.g., a deterministic NL-to-grep mapper), and re-run the exploration analysis.
  2. [§3.2 (Table 2); §3.3 (Fig. 3)] The headline consistency and exploration comparisons are reported without confidence intervals or significance tests. In Table 2, the Atomic gains for Kimi-K2.5 are +0.013–0.014 and for Sonnet-4.5 +0.017–0.031 on absolute pass^k values around 0.25–0.31; Fig. 3 shows read-diversity changes of +13% to +28% without error bars. With 65 instances and 10 rollouts per cell, these differences may be within sampling noise. Table 4 provides a 30-repeat robustness check only for the two open-weight actors, not for Sonnet-4.5. Please add per-instance paired bootstrap confidence intervals (or an equivalent test) for the pass^k deltas and read-diversity deltas, and state which claimed effects survive once uncertainty is accounted for.
minor comments (6)
  1. [Abstract; §3.4] The abstract's “56.3% lower token usage” should specify that the large reduction is in input tokens; Table 8 shows output and observation tokens are not uniformly lower under Python, so “token usage” is ambiguous.
  2. [Appendix B.4, Table 7] The definition of “high-relevant” files (“read by at least three successful trajectories from every setup”) is ambiguous about whether the count is pooled across setups or required within each setup; please clarify, since this affects how the recall comparison across setups should be interpreted.
  3. [Appendix B.4, Table 6] The human validation of the relevance proxy is described as LLM-assisted, but the number of annotators and inter-annotator agreement are not reported; adding these would strengthen the claim that the proxy is independent of the compared setups.
  4. [§3.3, Fig. 3] The y-axis label “% Difference from BashOnly” is easy to misread as percentage points; please clarify in the caption and add error bars or confidence intervals so the reader can judge the stability of the 13–28% values.
  5. [§2.2] The sampling procedure for the 65-instance subset (“randomly sampling 25 of its 100 repositories... yielding 65 problem instances”) should state the random seed or a reproducible sampling protocol so that the exact instance set can be reconstructed.
  6. [Table 2 caption] Please state explicitly that the reported pass^k values are averages over instances, since the formal definition in Appendix A is per-instance and the aggregation step is only described in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical architecture comparisons do not reduce to fitted parameters, self-citations, or definitions of the outcomes.

full rationale

The paper's central claims are observational comparisons across six tool architectures, measured on SWE-bench Live and additional coding tasks. No equation or fitted parameter is used to derive the reported consistency, exploration, or efficiency results: pass^k, read diversity, CodeBLEU distance, token usage, and step counts are all computed directly from recorded trajectories and benchmark outcomes, and the paper compares these metrics across setups rather than predicting them from a model. The capability-equivalence premise in Section 2.1 is an experimental design assumption, not a circular input; the NLSearch subagent concern raised by the reader is a potential confound about whether capability is truly held fixed, which is a validity threat, not a circularity. The high-relevant-file proxy in Appendix B.4 is defined using successful trajectories from every setup, which creates a mild coupling between the proxy and the compared setups, but the paper independently validates the proxy against human judgments (Table 6, 94.4% precision and 97.6% recall), so the relevant-file comparison has external support. Self-citations such as Yang et al. (2024) and Jimenez et al. (2023) are used as benchmark sources and representative-agent design references, not as load-bearing justifications for the paper's conclusions. Overall, the derivation chain is self-contained and empirically grounded; no claim reduces to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on design choices and measurement assumptions rather than fitted parameters or invented entities. The main hand-chosen quantities are the high-relevant-file threshold and the early-search window. The domain assumptions about model and benchmark representativeness are load-bearing for generalization but do not threaten the internal comparisons.

free parameters (2)
  • high_relevant_file_threshold = 3 successful trajectories from every setup
    Hand-chosen threshold defining the 'high-relevant' file ground truth in Appendix B.4; affects the recall and precision numbers in Table 7 that support the NLSearch exploration claim.
  • early_search_horizon = first 20 steps
    Hand-chosen window used to proxy initial repository search in Table 5; the diversity comparisons for NLSearch vs BashOnly depend on this window.
assumptions (4)
  • domain assumption Three actor models (Qwen3Coder-30B, Kimi K2.5, Claude Sonnet 4.5) are representative of coding agents in general.
    All headline claims are measured on these three models; the paper does not test other model families or scales.
  • domain assumption The sampled SWE-bench Live subset (25 of 100 repositories, at most 5 issues each, 65 instances) is representative of repository-level issue fixing.
    Sampling is random but capped, and no power or sensitivity analysis is provided for behavioral metrics.
  • domain assumption Ten repeated rollouts per instance are sufficient to estimate pass^k for k up to 9, read diversity, and solution diversity.
    pass^9 is computed from only 10 attempts, giving coarse resolution; the paper reports no variance estimates.
  • domain assumption BLEU and CodeBLEU distances are valid measures of reasoning similarity and patch diversity.
    Used in Appendix C (scratchpad BLEU vs BashOnly reasoning) and Section 3.3 (solution diversity); these metrics are approximate for code, which the paper does not discuss in detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior." pith.science (2026). https://pith.science/paper/IM27INHB

@misc{pith2026260811386,
  author       = {Pith},
  title        = {Pith review of: The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IM27INHB}},
  note         = {Machine review of arXiv:2608.11386}
}
abstract

As large language models continue to improve, agentic systems are becoming increasingly important, and tools are a key design dimension because they determine how agents access information and take action in their environments. Prior work on agent tooling has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model. We refer to this latter design dimension as tool architecture. We study tool architecture in coding agents through controlled experiments on repository-level issue fixing, comparing six tool architectures that hold the underlying information and actions similar while varying how they are organized and exposed to the model, across three actors and a total of 11,700 trajectories. Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior: Compared to a basic architecture where the agent has only the bash tool, more structured low-level interfaces improve consistency across repeated attempts by up to 4.7 $\times$; natural-language search broadens repository exploration and increases access to relevant files by more than 11%; and Python CodeAct-style interfaces achieve similar task performance with 41.6% fewer steps and 56.3% lower token usage. By contrast, lightweight text-based cognitive-scaffolding tools, such as tools that let the agent record intermediate reasoning, have limited effect on actor behavior.

Figures

Figures reproduced from arXiv: 2608.11386 by the authors.

Figure 1
Figure 1. Analogy between software engineering (SE) and agent tool design. In SE, software [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of tool architectures evaluated in this work. The panels illustrate how [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Effect of tool architecture on exploration and final-solution diversity. Bars show [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Efficiency comparison across tool architectures. Each point represents one actor– [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Overall resolve rates across actors and tool architectures. Results are broadly simi [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Atomic is associated with fewer environment-interaction errors, especially for weaker actors. The figure shows the average number of environment-interaction errors per trajectory under BashOnly and Atomic, both in total and broken down by error category for each actor.…
Figure 7
Figure 7. Figure 7: Illustration of the token metrics used in Table [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Wasted tokens from speculative revision under [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Scratchpad entries are highly similar to reasoning content that already appears in [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Hypothesis tracking rarely induces genuine branching behavior. We categorize [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Distribution of tool-call frequency per trajectory under each tool architecture. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Distribution of tool-call locations within trajectories. For each tool call, we [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages

  1. [1]

    Demystifying evals for ai agents, 2026 a

    Anthropic. Demystifying evals for ai agents, 2026 a . URL https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

    work page 2026

  2. [2]

    Harness design for long-running application development, 2026 b

    Anthropic. Harness design for long-running application development, 2026 b . URL https://www.anthropic.com/engineering/harness-design-long-running-apps

    work page 2026

  3. [3]

    Claude code overview, 2026 c

    Anthropic. Claude code overview, 2026 c . URL https://code.claude.com/docs/en/overview

    work page 2026

  4. [4]

    Claude sonnet, 2026 d

    Anthropic. Claude sonnet, 2026 d . URL https://www.anthropic.com/claude/sonnet

    work page 2026

  5. [5]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement

    Antonis Antoniades, Albert \"O rwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285, 2024

    arXiv 2024

  6. [6]

    Basic concepts and taxonomy of dependable and secure computing

    Algirdas Avizienis, J-C Laprie, Brian Randell, and Carl Landwehr. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing, 1 0 (1): 0 11--33, 2004

    work page 2004

  7. [7]

    Momentum builds, but gaps remain, 2025

    BCG. Momentum builds, but gaps remain, 2025. URL https://web-assets.bcg.com/fd/0d/bcc5dfae4cbaa08c718b95b16cf5/ai-at-work-2025-slideshow-june-2025-edit-02.pdf

    work page 2025

  8. [8]

    Building ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned

    Nghi DQ Bui. Building ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned. arXiv preprint arXiv:2603.05344, 2026

    arXiv 2026

Show all 66 references
  1. [9]

    Understanding automated program repair agents through the lens of traceability: An empirical study, 2026

    Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, and Baishakhi Ray. Understanding automated program repair agents through the lens of traceability: An empirical study, 2026. URL https://arxiv.org/abs/2506.08311

    2026 arXiv

  2. [10]

    Hassan, and Hajimu Iida

    Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjanasith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E. Hassan, and Hajimu Iida. Agent readmes: An empirical study of context files for agentic coding, 2025. URL...

    2025 arXiv

  3. [11]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  4. [12]

    Automatic evaluation of language generation technology based on structure alignment

    Katsuki Chousa and Tsutomu Hirao. Automatic evaluation of language generation technology based on structure alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 7663--7670, 2025

    2025

  5. [13]

    Augment code, 2026 a

    Augment Code. Augment code, 2026 a . URL https://www.augmentcode.com

    2026

  6. [14]

    Augment code context engine, 2026 b

    Augment Code. Augment code context engine, 2026 b . URL https://www.augmentcode.com/context-engine

    2026

  7. [15]

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah J...

    2025 arXiv

  8. [16]

    Mcptoolbench++: A large scale ai agent model context protocol mcp tool use benchmark, 2025

    Shiqing Fan, Xichen Ding, Liang Zhang, and Linjian Mo. Mcptoolbench++: A large scale ai agent model context protocol mcp tool use benchmark, 2025. URL https://arxiv.org/abs/2508.07575

    2025 arXiv

  9. [17]

    Configuring agentic ai coding tools: An exploratory study, 2026

    Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. Configuring agentic ai coding tools: An exploratory study, 2026. URL https://arxiv.org/abs/2602.14690

    2026 arXiv

  10. [18]

    More with less: An empirical study of turn-control strategies for efficient coding agents, 2025

    Pengfei Gao and Chao Peng. More with less: An empirical study of turn-control strategies for efficient coding agents, 2025. URL https://arxiv.org/abs/2510.16786

    2025

  11. [19]

    Trae agent: An llm-based agent for software engineering with test-time scaling

    Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al. Trae agent: An llm-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370, 2025

    2025 arXiv

  12. [20]

    Debug2fix: Supercharging coding agents with interactive debugging capabilities

    Spandan Garg and Yufan Huang. Debug2fix: Supercharging coding agents with interactive debugging capabilities. arXiv preprint arXiv:2602.18571, 2026

    2026 arXiv

  13. [21]

    u ndler, Mark M \

    Thibaud Gloaguen, Niels M \"u ndler, Mark M \"u ller, Veselin Raychev, and Martin Vechev. Evaluating agents. md: Are repository-level context files helpful for coding agents? arXiv preprint arXiv:2602.11988, 2026

    2026 arXiv

  14. [22]

    Reliabilitybench: Evaluating llm agent reliability under production-like stress conditions

    Aayush Gupta. Reliabilitybench: Evaluating llm agent reliability under production-like stress conditions. arXiv preprint arXiv:2601.06112, 2026

    2026

  15. [23]

    Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, and Ahmed E. Hassan. Agentic refactoring: An empirical study of ai coding agents, 2025. URL https://arxiv.org/abs/2511.04824

    2025

  16. [24]

    \'E tude comparative de la distribution florale dans une portion des alpes et des jura

    Paul Jaccard. \'E tude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37: 0 547--579, 1901

    1901

  17. [25]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  18. [26]

    Toward deriving software architectures from quality attributes

    Rick Kazman and Len Bass. Toward deriving software architectures from quality attributes. Technical Report CMU/SEI-94-TR-010 , Software Engineering Institute, Carnegie Mellon University, Aug 1994. URL https://www.sei.cmu.edu/library/toward-deriving-software-architectures-from-...

    1994

  19. [27]

    Evaluating the output of machine translation systems

    Alon Lavie. Evaluating the output of machine translation systems. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Tutorials, 2010

    2010

  20. [28]

    Contextbench: A benchmark for context retrieval in coding agents

    Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T Barr, Federica Sarro, Zhaoyang Chu, and He Ye. Contextbench: A benchmark for context retrieval in coding agents. arXiv preprint arXiv:2602.05892, 2026

    2026

  21. [29]

    The swe-bench illusion: When state-of-the-art llms remember instead of reason, 2025

    Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The swe-bench illusion: When state-of-the-art llms remember instead of reason, 2025. URL https://arxiv.org/abs/2506.12286

    2025

  22. [30]

    An empirical study on failures in automated issue solving, 2025 a

    Simiao Liu, Fang Liu, Liehao Li, Xin Tan, Yinghao Zhu, Xiaoli Lian, and Li Zhang. An empirical study on failures in automated issue solving, 2025 a . URL https://arxiv.org/abs/2509.13941

    2025

  23. [31]

    Reposcope: Leveraging call chain-aware multi-view context for repository-level code generation

    Yang Liu, Li Zhang, Fang Liu, Zhuohang Wang, Donglin Wei, Zhishuo Yang, Kechi Zhang, Jia Li, and Lin Shi. Reposcope: Leveraging call chain-aware multi-view context for repository-level code generation. arXiv preprint arXiv:2507.14791, 2025 b

    2025

  24. [32]

    On the impact of agents

    Jai Lal Lulla, Seyedmoein Mohsenimofidi, Matthias Galster, Jie M Zhang, Sebastian Baltes, and Christoph Treude. On the impact of agents. md files on the efficiency of ai coding agents. arXiv preprint arXiv:2601.20404, 2026

    2026

  25. [33]

    How to understand whole software repository

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. How to understand whole software repository. arXiv preprint arXiv:2406.01422, 2024

    2024 arXiv

  26. [34]

    Understanding code agent behaviour: An empirical study of success and failure trajectories, 2025

    Oorja Majgaonkar, Zhiwei Fei, Xiang Li, Federica Sarro, and He Ye. Understanding code agent behaviour: An empirical study of success and failure trajectories, 2025. URL https://arxiv.org/abs/2511.00197

    2025

  27. [35]

    Superagency in the workplace: Empowering people to unlock ai’s full potential, January 2025 a

    McKinsey. Superagency in the workplace: Empowering people to unlock ai’s full potential, January 2025 a . URL https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/superagency-in-the-workplace-empowering-people-to-unlock-ais-full-potential-at-work

    2025

  28. [36]

    The state of ai, 2025 b

    McKinsey. The state of ai, 2025 b . URL https://www.mckinsey.com/ /media/mckinsey/business

    2025

  29. [37]

    2025: The year the frontier firm is born, 2025

    Microsoft. 2025: The year the frontier firm is born, 2025. URL https://www.microsoft.com/en-us/worklab/work-trend-index/2025-the-year-the-frontier-firm-is-born

    2025

  30. [38]

    Harness engineering: leveraging codex in an agent-first world, 2026

    OpenAI. Harness engineering: leveraging codex in an agent-first world, 2026. URL https://openai.com/index/harness-engineering/

    2026

  31. [39]

    The open platform for cloud coding agent

    OpenHands. The open platform for cloud coding agent. https://openhands.dev/, 2026

    2026

  32. [40]

    Repograph: Enhancing ai software engineering with repository-level code graph

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. Repograph: Enhancing ai software engineering with repository-level code graph. arXiv preprint arXiv:2410.14684, 2024

    arXiv

  33. [41]

    The hidden cost of readability: How code formatting silently consumes your llm budget

    Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, and Xiaoning Du. The hidden cost of readability: How code formatting silently consumes your llm budget. arXiv preprint arXiv:2508.13666, 2025

    2025 arXiv

  34. [42]

    Prometheus: Towards long-horizon codebase navigation for repository-level problem solving, 2026

    Yue Pan, Zimin Chen, Siyu Lu, Zhaoyang Chu, Xiang Li, Han Li, Yang Feng, Claire Le Goues, Federica Sarro, Martin Monperrus, and He Ye. Prometheus: Towards long-horizon codebase navigation for repository-level problem solving, 2026. URL https://arxiv.org/abs/2507.19942

    2026

  35. [43]

    On the criteria to be used in decomposing systems into modules

    David Lorge Parnas. On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15 0 (12): 0 1053--1058, 1972

    1972

  36. [44]

    Foundations for the study of software architecture

    Dewayne E Perry and Alexander L Wolf. Foundations for the study of software architecture. ACM SIGSOFT Software engineering notes, 17 0 (4): 0 40--52, 1992

    1992

  37. [45]

    Towards a science of ai agent reliability

    Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a science of ai agent reliability. arXiv preprint arXiv:2602.16666, 2026

    2026 arXiv

  38. [46]

    System structure for software fault tolerance

    Brian Randell. System structure for software fault tolerance. In Proceedings of the international conference on Reliable software, pp.\ 437--449, 1975

    1975

  39. [47]

    Codebleu: a method for automatic evaluation of code synthesis

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020

    2009 arXiv

  40. [48]

    Specrover: Code intent extraction via llms

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. Specrover: Code intent extraction via llms. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp.\ 963--974. IEEE, 2025

    2025

  41. [49]

    Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of mt quality

    Scarton Scarton, Mikel L Forcada, Miquel Espla-Gomis, and Lucia Specia. Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of mt quality. In Proceedings of the 16th International Conference on Spoken Language Translation, 2019

    2019

  42. [50]

    Hugging face smolagents, 2026

    Smolagents. Hugging face smolagents, 2026. URL https://huggingface.co/docs/smolagents/index

    2026

  43. [51]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, Yanru Chen, Yanxu Chen, Yicun Chen, Yimin Chen,...

    2026 arXiv

  44. [52]

    Improving code localization with repository memory, 2026 a

    Boshi Wang, Weijian Xu, Yunsheng Li, Mei Gao, Yujia Xie, Huan Sun, and Dongdong Chen. Improving code localization with repository memory, 2026 a . URL https://arxiv.org/abs/2510.01003

    2026

  45. [53]

    Gta: A benchmark for general tool agents

    Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: A benchmark for general tool agents. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volum...

    2024 doi

  46. [54]

    Agentnoisebench: Benchmarking robustness of tool-using llm agents under noisy condition, 2026 b

    Ruipeng Wang, Yuxin Chen, Yukai Wang, Chang Wu, Junfeng Fang, Xiaodong Cai, Qi Gu, Hui Su, An Zhang, Xiang Wang, Xunliang Cai, and Tat-Seng Chua. Agentnoisebench: Benchmarking robustness of tool-using llm agents under noisy condition, 2026 b . URL https://arxiv.org/abs/2602.11348

    2026

  47. [55]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  48. [56]

    An empirical study of agent developer practices in ai agent frameworks, 2025 a

    Yanlin Wang, Xinyi Xu, Jiachi Chen, Tingting Bi, Wenchao Gu, and Zibin Zheng. An empirical study of agent developer practices in ai agent frameworks, 2025 a . URL https://arxiv.org/abs/2512.01939

    2025

  49. [57]

    Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers, 2025 b

    Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and Eugene Siow. Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers, 2025 b . URL https://arxiv.org/abs/2...

    2025 arXiv

  50. [58]

    Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, and Ahmed E. Hassan. On the use of agentic coding: An empirical study of pull requests on github, 2026. URL https://arxiv.org/abs/2509.14745

    2026

  51. [59]

    Performance evaluation of software architectures

    Lloyd G Williams and Connie U Smith. Performance evaluation of software architectures. In Proceedings of the 1st international workshop on Software and performance, pp.\ 164--177, 1998

    1998

  52. [60]

    Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025

    Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025

    2025

  53. [61]

    Reducing cost of llm agents with trajectory reduction

    Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. Reducing cost of llm agents with trajectory reduction. Proceedings of the ACM on Software Engineering, 3 0 (FSE): 0 1241–1263, 2026. ISSN 2994-970X. doi:10.1145/3797084. URL http://dx.doi.org/10.1145/3797084

    2026 doi

  54. [62]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  55. [63]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37: 0 50528--50652, 2024

    2024

  56. [64]

    -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045

    2024 arXiv

  57. [65]

    Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025

    2025 arXiv

  58. [66]

    Autocoderover: Autonomous program improvement

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp.\ 1592--1604, 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.