Pith. sign in

REVIEW 3 major objections 4 cited by

Grounding multi-agent coding in git worktrees, commits, and merges lifts long-horizon accuracy by large margins over single agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 20:48 UTC pith:OE4HH6Y7

load-bearing objection Solid, practical multi-agent SWE design that grounds coordination in git worktrees and merge/test gates; headline gains look real in direction but the manuscript as given has incomplete main tables and abstract/body number drift, so the causal story is only half-shown. the 3 major comments →

arxiv 2603.21489 v2 pith:OE4HH6Y7 submitted 2026-03-23 cs.CL cs.AI

Effective Strategies for Asynchronous Software Engineering Agents

Jiayi Geng , Graham Neubig This is my paper
classification cs.CL cs.AI
keywords multi-agent systemssoftware engineering agentsasynchronous collaborationgit worktreebranch-and-mergeCAIDCommit0PaperBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon software tasks such as implementing a library from a skeleton or reproducing a research paper still defeat single coding agents on accuracy and wall-clock time. Running several agents at once seems like the natural fix, but concurrent edits collide, dependencies drift, and partial results refuse to recombine. This paper argues that the remedy is not freer dialogue among agents but the same coordination tools human teams already use: a central manager that builds a dependency-aware plan, engineers that work asynchronously in isolated git worktrees, and integration only through commit-and-merge gated by executable tests. The resulting system, CAID, raises success rates by roughly a quarter absolute on paper-reproduction tasks and about fourteen points on library-generation tasks. The deeper claim is that branch-and-merge is the load-bearing coordination pattern for multi-agent work on a shared codebase, and that ordinary SWE primitives make that pattern reliable and machine-executable.

Core claim

CAID—centralized task delegation, asynchronous execution in isolated workspaces, and structured test-gated integration—consistently improves long-horizon shared-artifact software engineering over single-agent baselines, and systematic analysis identifies branch-and-merge realized via git worktree, commit, and merge as the central coordination mechanism that makes multi-agent collaboration workable.

What carries the argument

Centralized Asynchronous Isolated Delegation (CAID): a manager builds a dependency graph and assigns units; each engineer works in its own git worktree; progress re-enters the shared main branch only through git commit and git merge after local self-verification with tests.

Load-bearing premise

A prompt-engineered central manager can keep producing dependency-aware, parallelizable task decompositions and reassignments that are good enough for isolation and merge to finish the job.

What would settle it

On the same Commit0-Lite and PaperBench suites, replace the manager’s dependency-aware plan and reassignment loop with coarse or random task splits while keeping worktrees and merge gates fixed; if the reported absolute gains vanish, the claim that structured branch-and-merge coordination is the decisive mechanism fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes CAID (Centralized Asynchronous Isolated Delegation), a multi-agent architecture for long-horizon software engineering over shared repositories. A central manager builds dependency-aware task plans, delegates subtasks to engineer agents that run asynchronously in isolated git worktrees, and integrates progress via git commit/merge with test-based self-verification and structured JSON communication. The central empirical claim is that CAID improves accuracy over single-agent baselines by roughly 25–27 absolute points on PaperBench and ~14 points on Commit0, and that branch-and-merge realized through SWE primitives (worktree, commit, merge) is the key coordination mechanism. Supporting analysis includes a verification-intensity vs. efficiency trade-off on a nine-repository Commit0 subset (Figure 5) and discussion of limitations around cost, prompt-based delegation, and generalization.

Significance. If the reported gains hold under complete, reproducible evaluation, the work would be a useful systems contribution: it maps mature human SWE coordination primitives onto LLM multi-agent design and argues that physical isolation plus explicit merge/test integration, rather than free-form dialogue, is load-bearing for shared-artifact tasks. The design is coherent, the failure modes of concurrent edits are well motivated, and the paper is candid about cost/runtime overhead and prompt-engineered delegation (§6). Strengths include an executable integration protocol, self-verification before commit, and an open implementation pointer. The significance is currently limited by incomplete main-result presentation and abstract/body metric inconsistency, which prevent independent verification of the headline numbers and of the causal role of branch-and-merge.

major comments (3)
  1. Section 3 (Main Results) is truncated after the Commit0 setup (§3.2) and never presents complete main-result tables for either Commit0-Lite or PaperBench. The abstract claims +25.6% (PaperBench) / +14.7% (Commit0), while the body abstract-style claim states +26.7% / +14.3%. Without full single-agent vs CAID pass rates, model breakdowns, run counts, and variance, the central accuracy claim cannot be verified. Please restore complete tables (and reconcile the abstract numbers) before the claim can be treated as established.
  2. The claim that “branch-and-merge is a central coordination mechanism” (abstract; §7) is not supported by ablations that remove isolation or merge while holding the manager, model, and compute fixed. Figure 5 only varies verification intensity vs efficiency on a 9-repo subset; it does not isolate worktree/merge from extra agents, extra test calls, or better prompting. A minimal ablation (shared workspace vs worktree; free-form integration vs git merge) is needed to make the mechanism claim load-bearing rather than correlational.
  3. §6 and §2.3–2.4 acknowledge that task decomposition and dynamic reassignment rely on prompt engineering heuristics. Given that the architecture’s value depends on the manager producing dependency-aware, parallelizable units, the paper should report failure modes of bad decompositions (e.g., rate of idle engineers, merge-conflict frequency, fraction of tasks reassigned after failed integration) and, where possible, sensitivity to manager prompt variants. Without this, the weakest assumption remains unquantified.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper; gains measured on external benchmarks, not forced by definition or self-citation.

full rationale

CAID is an engineering design (central manager + async engineers in git worktrees + merge/test integration) whose load-bearing claims are absolute accuracy gains on Commit0 and PaperBench versus single-agent baselines, plus the analysis claim that branch-and-merge via SWE primitives is central. Those claims are not derived from equations that redefine the target, nor from parameters fitted to the same quantities then re-labeled as predictions. Performance is scored by external executable tests (library unit tests; paper-reproduction rubrics), which are independent of how CAID is defined. Related-work citations (MetaGPT, ChatDev, OpenHands, etc.) position the design; they do not supply a uniqueness theorem or ansatz that forces the reported pass rates. Self-positioning and prompt-engineered manager heuristics are limitations of the method, not circular reductions of the result. Abstract/body number mismatches and incomplete main tables (if present) are evidence-quality issues, not circularity. Honest finding: derivation chain is design + external evaluation; score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

This is an empirical multi-agent systems paper. The central claim rests on engineering design choices and benchmark outcomes, not on a mathematical derivation. Load-bearing background includes standard git semantics, the adequacy of unit/entry-point tests as integration oracles, and the manager’s ability to decompose tasks from prompts. Free parameters are operational (concurrency caps, budgets, models, prompt variants). The main invented construct is the CAID coordination paradigm itself.

free parameters (4)
  • max concurrent engineers / active coroutines
    Caps parallelism; affects both interference risk and wall-clock behavior; chosen by system design rather than derived.
  • iteration / round budgets and termination limits
    Tasks incomplete if budgets exhaust; directly affects reported success rates.
  • underlying LLM choice and prompt variants
    Results reported across models and three coordination prompt styles (manager review vs self-verify vs efficiency); performance depends on these choices.
  • restricted shared files policy (e.g., __init__.py)
    Hand-specified isolation rule that shapes what engineers may commit.
axioms (4)
  • domain assumption Git worktree isolation plus merge conflict handling prevents silent concurrent corruption of the shared codebase.
    Core of §2.3; maps human SWE practice onto agents without independent proof that all interference modes are covered.
  • domain assumption Executable tests (or minimal entry points) are a sufficient verification signal for whether a commit should be integrated.
    §2.5 self-verification and termination; pass/fail of available tests stands in for global correctness.
  • ad hoc to paper A central LLM manager can build useful dependency-aware task graphs and dynamic reassignments from structured prompts.
    §2 and Limitations §6 admit reliance on prompt heuristics rather than learned delegation policies.
  • domain assumption Structured JSON + git commits avoid the primary multi-agent misalignment failure mode of free-form dialog.
    Stated in intro/§2.4 citing Cemri et al.; treated as design premise.
invented entities (2)
  • CAID (Centralized Asynchronous Isolated Delegation) independent evidence
    purpose: Name and package the three-primitive coordination paradigm (centralized delegation, async execution, isolated workspaces) as the default multi-agent SWE architecture.
    New system construct evaluated on Commit0 and PaperBench; independent evidence is the empirical gains and ablations, not external theory.
  • Manager–engineer structured JSON task protocol no independent evidence
    purpose: Machine-parsable task boundaries (files, functions, dependencies) replacing free-form multi-agent chat.
    Paper-specific interface design in §2.4 / Appendix A.1; evidence is internal system behavior.

pith-pipeline@v1.1.0-grok45 · 15233 in / 3021 out tokens · 40609 ms · 2026-07-13T20:48:28.681092+00:00 · methodology

0 comments
read the original abstract

AI agents have become increasingly capable at isolated software engineering (SWE) tasks such as resolving issues on Github. Yet long-horizon tasks involving multiple interdependent subtasks still pose challenges both with respect to accuracy, and with respect to timely completion. A natural approach to solving these long-horizon tasks in a timely manner is asynchronous multi-agent collaboration, where multiple agents work on different parts of the task at the same time. But effective application of multi-agent systems has proven surprisingly difficult: concurrent edits by multiple agents interfere with each other, dependencies are difficult to synchronize, and combining partial progress into a coherent whole is challenging. On the other hand, human developers have long relied on mature collaboration infrastructure to manage these challenges in large software projects. Inspired by these collaboration primitives, we introduce Centralized Asynchronous Isolated Delegation (CAID), a structured multi-agent coordination paradigm grounded in three core SWE primitives: centralized task delegation, asynchronous execution, and isolated workspaces. CAID constructs dependency-aware task plans through a central manager, executes subtasks concurrently in isolated workspaces, and consolidates progress via structured integration with executable test-based verification. In empirical evaluation, we find that CAID improves accuracy over single-agent baselines by 25.6% absolute on paper reproduction tasks (PaperBench) and 14.7% on Python library development tasks (Commit0). Through systematic analysis, we find that branch-and-merge is a central coordination mechanism for multi-agent collaboration, and that SWE primitives such as git worktree, git commit, and git merge enable it to be realized in a reliable and executable manner.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ProgramBench: Can Language Models Rebuild Programs From Scratch?

    cs.SE 2026-05 unverdicted novelty 7.0

    ProgramBench introduces 200 tasks where models must reconstruct full programs like FFmpeg or SQLite from docs alone; none of 9 evaluated LMs fully solve any task and the best passes 95% tests on only 3% of tasks while...

  2. Multi-Agent Computer Use

    cs.MA 2026-06 unverdicted novelty 6.0

    A manager-driven DAG decomposition with parallel subagents improves computer use agent success rates by 3.4-25.5% and reduces wall-clock time on long-horizon benchmarks.

  3. ATM: CID-Brokered Pre-Write Admission for Multi-Agent Code Co-Synthesis

    cs.SE 2026-06 unverdicted novelty 5.0

    ATM is a CID-brokered governance framework that maps write intents to semantic atoms for pre-admission control, validation, and neutral-steward application in single-domain multi-agent code synthesis.

  4. Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents

    cs.SE 2026-04 unverdicted novelty 5.0

    Formal architecture descriptors reduce AI coding agent navigation steps by 33-44% and behavioral variance by 52% in controlled and observational studies.

Reference graph

Works this paper leans on

26 extracted references · 17 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    Pith/arXiv arXiv

  2. [2]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741,

    Pith/arXiv arXiv

  3. [3]

    Commit0: Library generation from scratch.arXiv preprint arXiv:2412.01769,

    Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. Commit0: Library generation from scratch.arXiv preprint arXiv:2412.01769,

    Pith/arXiv arXiv

  4. [4]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Pith/arXiv arXiv

  5. [5]

    Measuring ai ability to complete long tasks.arXiv preprint arXiv:2503.14499,

    Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney V on Arx, et al. Measuring ai ability to complete long tasks.arXiv preprint arXiv:2503.14499,

    arXiv

  6. [6]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 15174–15186, 2024a. Nikita Benkovich and Vitalii Valkov. Ag...

    arXiv

  7. [7]

    Mas-prove: Understanding the process verification of multi-agent systems.arXiv preprint arXiv:2602.03053,

    Vishal Venkataramani, Haizhou Shi, Zixuan Ke, Austin Xu, Xiaoxiao He, Yingbo Zhou, Semih Yavuz, Hao Wang, and Shafiq Joty. Mas-prove: Understanding the process verification of multi-agent systems.arXiv preprint arXiv:2602.03053,

    arXiv

  8. [8]

    Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025a

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025a. Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J Ryan, Jiaxin Pei, et al. Cooperbench: Why coding agents can...

    Pith/arXiv arXiv

  9. [9]

    Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657,

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657,

    Pith/arXiv arXiv

  10. [10]

    Accessed: 2026-02-20. OpenAI. Gpt-5-mini.https://www.openai.com,

    2026

  11. [11]

    Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, et al

    Large language model developed by OpenAI. Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, et al. The openhands software agent sdk: A composable and extensible foundation for production agents.arXiv preprint arXiv:2511.03690, 2025a. Aohan Zeng, Xin Lv, Qinkai Zhe...

    Pith/arXiv arXiv

  12. [12]

    Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, and Shangding Gu

    Large language model developed by Anthropic. Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, and Shangding Gu. Understanding agent scaling in llm-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794,

    arXiv

  13. [13]

    Self-evolving multi-agent collaboration networks for software development.arXiv preprint arXiv:2410.16946,

    Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development.arXiv preprint arXiv:2410.16946,

    Pith/arXiv arXiv

  14. [14]

    Autoagents: A framework for automatic agent generation.arXiv preprint arXiv:2309.17288,

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation.arXiv preprint arXiv:2309.17288,

    Pith/arXiv arXiv

  15. [15]

    Agentorchestra: Orchestrating hierarchical multi-agent intelligence with the tool-environment-agent (tea) protocol

    Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Yilei Zhao, Ce Cui, Yang Liu, and Bo An. Agentorchestra: Orchestrating hierarchical multi-agent intelligence with the tool-environment-agent (tea) protocol. 2025b. Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vuli´c, Anna Korhonen, and Sercan Ö Arık. Multi-agent design: Optimizing...

    arXiv

  16. [16]

    Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.arXiv preprint arXiv:2310.02170,

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.arXiv preprint arXiv:2310.02170,

    Pith/arXiv arXiv

  17. [17]

    Megaagent: A large-scale autonomous llm-based multi-agent system without predefined sops

    Qian Wang, Tianyu Wang, Zhenheng Tang, Qinbin Li, Nuo Chen, Jingsheng Liang, and Bingsheng He. Megaagent: A large-scale autonomous llm-based multi-agent system without predefined sops. InFindings of the Association for Computational Linguistics: ACL 2025, pages 4998–5036, 2025b. Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dan...

    Pith/arXiv arXiv 2025

  18. [18]

    Multi-agent cooperation through learning-aware policy gradients

    Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake Richards, Guillaume Lajoie, João Sacramento, et al. Multi-agent cooperation through learning-aware policy gradients. arXiv preprint arXiv:2410.18636,

    Pith/arXiv arXiv

  19. [19]

    Agent-as-a-judge: Evaluate agents with agents

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934,

    Pith/arXiv arXiv

  20. [20]

    Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923,

    Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al. Coact-1: Computer-using agents with coding as actions.arXiv preprint arXiv:2508.03923,

    arXiv

  21. [21]

    Boad: Discovering hierarchical software engineering agents via bandit optimization.arXiv preprint arXiv:2512.23631,

    Iris Xu, Guangtao Zeng, Zexue He, Charles Jin, Aldo Pareja, Dan Gutfreund, Chuang Gan, and Zhang-Wei Hong. Boad: Discovering hierarchical software engineering agents via bandit optimization.arXiv preprint arXiv:2512.23631,

    arXiv

  22. [22]

    More agents is all you need.arXiv preprint arXiv:2402.05120, 2024a

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need.arXiv preprint arXiv:2402.05120, 2024a. Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li. Agent-oriented planning in multi-agent systems.arXiv preprint arXiv:2410.02189, 2024b. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherbur...

    Pith/arXiv arXiv

  23. [23]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al

    URL https://openai.com/index/ introducing-swe-bench-verified/. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941,

    Pith/arXiv arXiv

  24. [24]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868,

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868,

    Pith/arXiv arXiv

  25. [25]

    Devbench: A comprehensive benchmark for software development.arXiv preprint arXiv:2403.08604, 3, 2024c

    Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. Devbench: A comprehensive benchmark for software development.arXiv preprint arXiv:2403.08604, 3, 2024c. Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et ...

    Pith/arXiv arXiv

  26. [26]

    Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces.arXiv preprint arXiv:2602.14337,

    Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, et al. Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces.arXiv preprint arXiv:2602.14337,

    arXiv