Pith. sign in

REVIEW 4 major objections 5 minor 53 references

LLM multi-agent systems gain accuracy from architectural scaling only above a model-capability threshold, peak at intermediate complexity, and stay inconsistent at every scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 22:41 UTC pith:LP2RPV7P

load-bearing objection Useful practitioner guidance on when MAS scaling helps, with a real consistency finding, but the intermediate-peak story is partly a timeout artifact. the 4 major comments →

arxiv 2607.27942 v1 pith:LP2RPV7P submitted 2026-07-30 cs.MA

Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis

Linus Sander , Fengjunjie Pan , Vahid Zolfaghari , Andre Schamschurko , Nenad Petrovic , Alois Knoll This is my paper
classification cs.MA
keywords LLM multi-agent systemsarchitectural scalingdesign principlessummary-based communicationelastic feedbackworkflow topologyterminal-benchrun-to-run consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that LLM multi-agent systems can be scaled usefully if they follow four design principles: keep the architecture simple, let agents decide when feedback is worth another pass, run mostly sequential workflows with optional loops, and pass only short natural-language summaries between agents. The authors turn those principles into a reference architecture—a constrained directed workflow graph plus a shared group chat of summaries—and test four setups of rising complexity on terminal-based system-engineering tasks with two models of different strength. Scaling raises accuracy with roughly linear cost growth, but only when the base model is capable enough; a weaker model spends tokens and triggers loops without real gains. Performance tops out at a mid-sized setup (five agents, three loops) and then falls, largely from timeouts and evaluation limits, while run-to-run consistency stays poor at every level. The practical message is that practitioners should establish model adequacy first, grow architecture only when the cost–accuracy tradeoff still improves, and treat consistency—not mean accuracy—as the main barrier to production use.

Core claim

Architectural scaling of an LLM multi-agent system, under a principle-guided reference design, yields measurable accuracy gains with approximately linear cost growth only when the underlying LLM exceeds a minimum capability threshold; performance peaks at intermediate complexity and then degrades, while run-to-run consistency remains a central failure mode at every scale.

What carries the argument

A reference architecture that pairs a shared group chat of natural-language agent summaries with a constrained directed workflow graph Gw = (A, E, s, T): agents choose the next successor (at most three), optional feedback loops are elastic and agent-triggered, and only concise summaries—not full traces—are passed forward.

Load-bearing premise

The claim rests on treating four hand-built topologies that each add two agents and two loops as a fair ladder of architectural complexity, rather than a mix of role design, longer paths, and timeout exposure.

What would settle it

Rerun the same four configurations on the same terminal benchmark with a strong model, but equalize minimum path length and timeout budget (or remove timeout-driven failures) and check whether accuracy still peaks at the mid-complexity setup and still fails to improve consistency.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Establish base-model adequacy before spending effort on multi-agent topology.
  • Prefer simple sequential workflows and expand complexity only while the measured cost–accuracy tradeoff still improves.
  • Design quality-assurance agents to challenge upstream framing rather than rubber-stamp it.
  • Report run-to-run consistency alongside mean accuracy when claiming MAS gains.
  • Treat summary-based group communication and agent-chosen elastic loops as default building blocks for scalable MAS in tool-rich domains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If consistency is the binding constraint, ensemble or majority-vote wrappers over the same topology may buy reliability cheaper than adding more specialized agents.
  • The capability-threshold result suggests weak models may need stronger single-agent tool interfaces before multi-agent orchestration pays off.
  • Standardized timeout and partial-credit scoring on interactive benchmarks would change how the peak-then-drop pattern is interpreted across labs.
  • Adversarial QA prompts that force loop triggers could be a direct, low-cost test of whether the consistency bottleneck is framing cascade rather than model noise alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper distills four design principles for scalable LLM multi-agent systems (simplicity, elastic feedback, sequential workflows with optional loops, summary-based communication) and operationalizes them in a reference architecture formalized as a constrained directed workflow graph with a shared group-chat of agent summaries. It instantiates four topologies of increasing complexity (Singleton, MAS-S, MAS-M, MAS-L) and evaluates them on terminal-bench with GPT-5-nano, GPT-5-mini, and a limited GPT-5.3-Codex check. The central empirical claim is that architectural scaling improves accuracy with roughly linear cost growth only above a base-model capability threshold; performance peaks at intermediate complexity (MAS-M) then degrades, while run-to-run consistency remains poor at every scale.

Significance. If the results hold under cleaner controls, the paper offers useful practitioner guidance: establish LLM adequacy before adding agents, scale architectures incrementally, and treat consistency—not mean accuracy—as the production bottleneck. Strengths include an external automated benchmark, multi-metric reporting (accuracy, test-weighted accuracy, cost, timeouts, loop utilization, consistency segments), explicit acknowledgment of consistency failures, and a competitive MAS-M + GPT-5.3-Codex placement on the public leaderboard. The contribution is primarily empirical systems evidence and design synthesis rather than a new theoretical scaling law; that is appropriate for cs.MA but makes the isolation of architectural effects load-bearing for the claimed intermediate peak.

major comments (4)
  1. [§VI.A–C, §VII.b, §V.D] The intermediate-peak claim (Abstract; §VI.A; §VII.b) is not cleanly isolated from benchmark artifacts. Each scaling step adds two agents and two loops under a ≤3-successor rule (§IV.B, §V.D), which simultaneously lengthens the shortest path, multiplies handoffs, and drives AgentTimeout counts from µ≈3.3 (Singleton) to µ=50 (MAS-L) for GPT-5-mini (§VI.C). The paper itself attributes the MAS-L drop largely to timeouts and evaluation limits. Without a control that holds wall-clock/timeout budget and minimum path length fixed while varying only role/loop structure—or an equal-budget re-run of MAS-L—the claim that performance peaks at intermediate architectural complexity remains confounded. Please either add such a control or substantially qualify the peak claim in Abstract/Discussion.
  2. [Table I, Fig. 3–4, §VI.A–B] The Singleton→MAS-S gain for GPT-5-mini is material (+15.2% relative), but the MAS-S→MAS-M step that defines the reported peak is thin: +2.2% relative accuracy and test-weighted 0.543 vs 0.530 (§VI.A, Table I), with only 3–5 runs per configuration (§V.F). Consistency segments (Fig. 4) further show that fewer than half of solved tasks are solved in all runs. Please report uncertainty (e.g., bootstrap CIs or per-task McNemar/binomial tests across runs) for accuracy and test-weighted accuracy, and state whether MAS-M reliably outperforms MAS-S or only ranks highest on the point estimate.
  3. [§V.C, §VI.A, §VII.a] Model-family generality is a load-bearing limitation for the capability-threshold claim (§VII.a). All primary scaling curves use GPT-5-nano/mini from one family; Codex is only evaluated on MAS-M (§V.C, §V.F). The nano-flat / mini-rising contrast could reflect family-specific tool-use or instruction-following quirks rather than a general capability threshold. At minimum, add one non-OpenAI model tier on Singleton and MAS-M (or clearly scope the threshold claim to this family and mark cross-family generality as untested).
  4. [§IV.B, §VI.D, P2 in §III] The architecture assumes agents can reliably self-assess completion and choose successors via a [<next_agent>] tag in a natural-language summary (§IV.B). Loop-utilization results (§VI.D) show Approver-originated loops are rarely used and several MAS-L loops are almost never triggered, which undercuts P2 (elastic feedback) as operationalized. Please analyze mis-routing / premature termination / tag-parse failures, or provide ablation evidence that agent-controlled routing outperforms a fixed schedule under the same roles—otherwise the elastic-feedback principle is asserted more than demonstrated.
minor comments (5)
  1. [Fig. 3–4] Figure labels in the manuscript text use 'spoox-s/m/l' in places (Fig. 3–4 captions in the source) while the body uses MAS-S/M/L; unify naming for camera-ready clarity.
  2. [§V.A, §IV.B, §V.C, §II] Typos: 'succeass criteria' (§V.A), 'mechansim' (§IV.B), 'terminl-bench' (§V.C), 'spoox' artifacts. Also 'BACKGROUND ANDRELATEDWORK' spacing in the heading.
  3. [Table I] Table I 'Cost trend' as '+1×, +1.4×, +3.2×' is underspecified (relative to what baseline quantity—completion tokens only?). Define the exact cost normalizer in the caption.
  4. [§IV.B] The formal graph constraints (§IV.B) are clear, but the paper never states whether Gw must be acyclic aside from optional loops, nor how multiple terminal agents interact with evaluation stop conditions. A short clarification would help reproducibility.
  5. [§VI.A] Leaderboard comparison dates (21 July 2026) and model names (GPT-5.3-Codex) should be version-pinned with snapshot IDs if available, since leaderboards move.

Circularity Check

0 steps flagged

No significant circularity: empirical systems study with literature-derived principles and external-benchmark measurements.

full rationale

The paper’s load-bearing claims are empirical outcomes (accuracy, cost, consistency, loop utilization) measured on terminal-bench with automated unit tests and two external LLM tiers, not quantities derived from fitted laws or self-defined identities. The four design principles (P1–P4) are explicitly distilled from a survey of prior MAS work and then operationalized in a hand-designed reference topology; that is ordinary systems methodology, not a prediction forced by construction. The constrained directed workflow graph and the Singleton→MAS-S→MAS-M→MAS-L ladder are author-chosen configurations whose performance is reported as measured fact, including the MAS-L degradation attributed to timeouts—there is no equation in which a fitted parameter is renamed a prediction, no uniqueness theorem imported from the authors’ prior work, and no self-citation that alone forces the central scaling result. Confounding of path length with architectural complexity is an experimental-validity concern, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly methodological choice plus domain assumptions, not free physical constants. The central claims rest on the authors’ topology ladder, agent-controlled routing, summary-only communication, Terminal-Bench as the task distribution, and treating same-family model tiers as a capability axis. No graviton-like entities; the ‘reference architecture’ is an engineered artifact under test.

free parameters (3)
  • max_successors_per_agent = 3
    Hard cap ∀u: out-degree ≤ 3 in G_w is a hand-chosen structural limit that shapes all scalable topologies.
  • scaling_ladder_agent_and_loop_increments = +2 agents, +2 loops per step
    Complexity steps are defined as +2 agents and +2 loops (1→3→5→7 agents; 0→1→3→5 loops), which directly defines the x-axis of the scaling claims.
  • run_counts_per_configuration = 3–5 runs (MAS-M: 5)
    Consistency and mean accuracy depend on 3–5 runs per pairing, with MAS-M preferentially run five times for leaderboard submission.
axioms (5)
  • domain assumption Terminal-Bench unit-test success is a valid primary measure of system-engineering agent quality for comparing architectures.
    Entire evaluation and leaderboard comparisons rest on this benchmark’s tasks and timeout rules (§V.A, §VI).
  • domain assumption Holding model family fixed while varying size/tier isolates ‘capability’ from vendor-specific behavior.
    Stated motivation for GPT-5-nano/mini/Codex comparisons (§V.C); underpins the capability-threshold claim.
  • ad hoc to paper Agents can reliably self-assess sub-task completion and choose successors via a [<next_agent>] tag in a natural-language summary.
    Core control mechanism of elastic feedback and optional loops (§IV.B); if routing is noisy, topology benefits are misattributed.
  • domain assumption Passing only concise natural-language summaries (not full traces) preserves enough information for downstream agents while controlling context growth.
    Principle P4 and group-chat design (§III–IV.A); foundational to claimed linear context growth and noise suppression.
  • ad hoc to paper Directed graph reachability/start/terminal constraints plus sequential backbone are sufficient topology rules for scalable MAS in this domain.
    Formalization of G_w=(A,E,s,T) and three structural constraints (§IV.B) define the reference architecture under test.
invented entities (2)
  • Reference MAS architecture G_w with group-chat-of-summaries and agent-tagged optional loops independent evidence
    purpose: Operationalize P1–P4 into a single scalable topology pattern instantiated as Singleton/MAS-S/M/L.
    Engineered composite of known mechanisms; evaluated empirically rather than derived. Independent evidence is the Terminal-Bench runs, not an external physical signature.
  • Four packaged design principles P1–P4 no independent evidence
    purpose: Claim a compact, practitioner-facing basis for scalable LLM-MAS design.
    Synthesized from survey citations rather than newly discovered laws; usefulness is argued via the experiments.

pith-pipeline@v1.2.0-daily-grok45 · 17689 in / 3457 out tokens · 76504 ms · 2026-07-31T22:41:20.951782+00:00 · methodology

0 comments
read the original abstract

LLM-based multi-agent systems have the potential to enable collective intelligence and scale toward solving highly complex tasks through coordinated ensembles of specialized agents. However, despite their theoretical potential, the architectural design space remains largely non-systematized and lacks broadly established design principles. Furthermore, the scalability characteristics of such systems are only partially understood so far. This paper makes two contributions. We first distill four design principles for scalable MAS architectures from a structured analysis of prior work: simplicity, elastic feedback, sequential workflows with optional loops, and summary-based communication. We operationalize these principles in a reference architecture whose topology is formalized as a constrained directed workflow graph, and we evaluate four configurations of increasing complexity on a standardized benchmark of terminal-based system engineering tasks using two LLMs of differing capability. Our findings show that scaling yields measurable accuracy improvements with approximately linear cost growth, but only when the underlying LLM exceeds a minimum capability threshold. Performance peaks at intermediate complexity, then degrades due to timeouts and evaluation limitations. In addition, persistent consistency issues emerge as a central challenge across all scaling levels. These results provide concrete design guidance for practitioners and highlight consistency and evaluation standardization as key targets for future research.

Figures

Figures reproduced from arXiv: 2607.27942 by Alois Knoll, Andre Schamschurko, Fengjunjie Pan, Linus Sander, Nenad Petrovic, Vahid Zolfaghari.

Figure 1
Figure 1. Figure 1: Example of a multi-agent system lifecycle. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Setups of reference MAS systems MAS-S (3 agents, 1 loop): The Solver handles the full task, and the Tester evaluates the result, deciding whether to return control to the Solver for revision or pass the solution to the Summarizer. This loop exemplifies a guided yet elastic workflow, repeated as often as the Tester seems necessary. MAS-M (5 agents, 3 loops): An Explorer conducts struc￾tured environment inve… view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy across agent-system scaling steps. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task-solving consistency across agent runs. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 24 linked inside Pith

  1. [1]

    Learning agents for uncertain environments,

    S. Russell, “Learning agents for uncertain environments,” inProceedings of the eleventh annual conference on Computational learning theory, 1998, pp. 101–103

    1998

  2. [2]

    A roadmap of agent research and development,

    N. R. Jennings, K. Sycara, and M. Wooldridge, “A roadmap of agent research and development,”Autonomous agents and multi-agent systems, vol. 1, no. 1, pp. 7–38, 1998

    1998

  3. [3]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”Science China Information Sciences, vol. 68, no. 2, p. 121101, 2025

    2025

  4. [4]

    Kola: Carefully benchmarking world knowledge of large language models,

    J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Liet al., “Kola: Carefully benchmarking world knowledge of large language models,”arXiv preprint arXiv:2306.09296, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  7. [7]

    A survey of llm-based agents in medicine: How far are we from baymax?

    W. Wang, Z. Ma, Z. Wang, C. Wu, J. Ji, W. Chen, X. Li, and Y . Yuan, “A survey of llm-based agents in medicine: How far are we from baymax?” arXiv preprint arXiv:2502.11211, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inThe eleventh international conference on learning representations, 2022

    2022

  9. [9]

    The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey,

    T. Masterman, S. Besen, M. Sawtell, and A. Chao, “The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey,”arXiv preprint arXiv:2404.11584, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Ruler: What’s the real context size of your long-context language models?

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y . Zhang, and B. Ginsburg, “Ruler: What’s the real context size of your long-context language models?”arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–30, 2025

    2025

  12. [12]

    Agentbench: Evaluating llms as agents,

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yanget al., “Agentbench: Evaluating llms as agents,”arXiv preprint arXiv:2308.03688, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Multi-agent computer use,

    J. Y . Koh, R. Salakhutdinov, and D. Fried, “Multi-agent computer use,” arXiv preprint arXiv:2606.01533, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,

    W. Chen, Y . Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y . Lu, Y .-H. Hung, C. Qianet al., “Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  15. [15]

    Large language model based multi-agents: A survey of progress and challenges,

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,”arXiv preprint arXiv:2402.01680, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Camel: Communicative agents for

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for” mind” exploration of large language model society,”Advances in Neural Information Processing Systems, vol. 36, pp. 51 991–52 008, 2023

    2023

  17. [17]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandranet al., “Why do multi-agent llm systems fail?”arXiv preprint arXiv:2503.13657, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    A survey on llm- based multi-agent systems: workflow, infrastructure, and challenges,

    X. Li, S. Wang, S. Zeng, Y . Wu, and Y . Yang, “A survey on llm- based multi-agent systems: workflow, infrastructure, and challenges,” Vicinagearth, vol. 1, no. 1, p. 9, 2024

    2024

  19. [19]

    Llm multi-agent systems: Challenges and open problems,

    S. Han, Q. Zhang, Y . Yao, W. Jin, and Z. Xu, “Llm multi-agent systems: Challenges and open problems,”arXiv preprint arXiv:2402.03578, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Beyond self-talk: A communication-centric survey of llm-based multi-agent systems,

    B. Yan, Z. Zhou, L. Zhang, L. Zhang, Z. Zhou, D. Miao, Z. Li, C. Li, and X. Zhang, “Beyond self-talk: A communication-centric survey of llm-based multi-agent systems,”arXiv preprint arXiv:2502.14321, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Stay focused: Problem drift in multi-agent debate,

    J. Becker, L. B. Kaesberg, A. Stephan, J. P. Wahle, T. Ruas, and B. Gipp, “Stay focused: Problem drift in multi-agent debate,”arXiv preprint arXiv:2502.19559, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Linet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  23. [23]

    Scaling large language model-based multi-agent collaboration,

    C. Qian, Z. Xie, Y . Wang, W. Liu, K. Zhu, H. Xia, Y . Dang, Z. Du, W. Chen, C. Yanget al., “Scaling large language model-based multi-agent collaboration,”arXiv preprint arXiv:2406.07155, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Chateval: Towards better llm-based evaluators through multi- agent debate,

    C.-M. Chan, W. Chen, Y . Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu, “Chateval: Towards better llm-based evaluators through multi- agent debate,”arXiv preprint arXiv:2308.07201, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  26. [26]

    Reflex- ion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflex- ion: Language agents with verbal reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 8634–8652, 2023

    2023

  27. [27]

    Multi-agent collaboration: Harnessing the power of intelligent llm agents,

    Y . Talebirad and A. Nadiri, “Multi-agent collaboration: Harnessing the power of intelligent llm agents,”arXiv preprint arXiv:2306.03314, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Improving factuality and reasoning in language models through multiagent debate,

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” inForty-first International Conference on Machine Learning, 2023

    2023

  29. [29]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,”arXiv preprint arXiv:2304.05128, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 801–824, 2025

    2025

  31. [31]

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,”arXiv preprint arXiv:2312.13010, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Conget al., “Chatdev: Communicative agents for software development,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 174–15 186

    2024

  33. [33]

    Agilecoder: Dynamic collaborative agents for software development based on agile methodology,

    M. H. Nguyen, T. P. Chau, P. X. Nguyen, and N. D. Bui, “Agilecoder: Dynamic collaborative agents for software development based on agile methodology,” in2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 156–167

    2025

  34. [34]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  35. [35]

    (2025) Claude code

    Anthropic. (2025) Claude code. Accessed: Nov. 14, 2025. [Online]. Available: https://github.com/anthropics/claude-code

    2025

  36. [36]

    (2025) Openai codex

    OpenAI. (2025) Openai codex. Accessed: Nov. 14, 2025. [Online]. Available: https://github.com/openai/codex

    2025

  37. [37]

    Literature review of multi-agent debate for problem- solving,

    A. Tillmann, “Literature review of multi-agent debate for problem- solving,”arXiv preprint arXiv:2506.00066, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion,

    T. Liu, X. Wang, W. Huang, W. Xu, Y . Zeng, L. Jiang, H. Yang, and J. Li, “Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion,”arXiv preprint arXiv:2409.14051, 2024

    arXiv 2024

  39. [39]

    Auto-scaling llm-based multi-agent systems through dynamic integration of agents,

    R. R. Perera, A. Basnayake, and M. Wickramasinghe, “Auto-scaling llm-based multi-agent systems through dynamic integration of agents,” Frontiers in Artificial Intelligence, vol. 8, p. 1638227, 2025

    2025

  40. [40]

    Ai agents that matter,

    S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, “Ai agents that matter,”arXiv preprint arXiv:2407.01502, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    M. S. A. Team. (2025) The 100 line ai agent that solves github issues and more. Accessed: Nov. 24, 2025. [Online]. Available: https://github.com/SWE-agent/mini-swe-agent

    2025

  42. [42]

    Embodied llm agents learn to cooperate in organized teams,

    X. Guo, K. Huang, J. Liu, W. Fan, N. V ´elez, Q. Wu, H. Wang, T. L. Griffiths, and M. Wang, “Embodied llm agents learn to cooperate in organized teams,”arXiv preprint arXiv:2403.12482, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    2024

  44. [44]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchananet al., “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  45. [45]

    Tptu: large language model-based ai agents for task planning and tool usage,

    J. Ruan, Y . Chen, B. Zhang, Z. Xu, T. Bao, G. Du, S. Shi, H. Mao, Z. Li, X. Zenget al., “Tptu: large language model-based ai agents for task planning and tool usage,”arXiv preprint arXiv:2308.03427, 2023

    arXiv 2023

  46. [46]

    Ai agents for computer use: A review of instructionbased computer control, gui automation, and operator assistants,

    P. J. Sager, B. Meyer, P. Yan, R. von Wartburg-Kottler, L. Etaiwi, A. Enayati, G. Nobel, A. Abdulkadir, B. F. Grewe, and T. Stadelmann, “Ai agents for computer use: A review of instructionbased computer control, gui automation, and operator assistants,”arXiv preprint arXiv:2501.16150, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Posix: A prompt sensitivity index for large language models,

    A. Chatterjee, H. K. Renduchintala, S. Bhatia, and T. Chakraborty, “Posix: A prompt sensitivity index for large language models,”arXiv preprint arXiv:2410.02185, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  49. [49]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  50. [50]

    (2025) Gpt-5 system card

    OpenAI. (2025) Gpt-5 system card. Accessed: Dec. 1, 2025. [Online]. Available: https://cdn.openai.com/gpt-5-system-card.pdf

    2025

  51. [51]

    Large language models are not fair evaluators,

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y . Cao, L. Kong, Q. Liu, T. Liuet al., “Large language models are not fair evaluators,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 9440– 9450

    2024

  52. [52]

    When agents disagree with themselves: Measuring behavioral consistency in llm-based agents,

    A. Mehta, “When agents disagree with themselves: Measuring behavioral consistency in llm-based agents,” 2026. [Online]. Available: https://arxiv.org/abs/2602.11619

    Pith/arXiv arXiv 2026

  53. [53]

    Mas-prove: Understanding the process verification of multi-agent systems,

    V . Venkataramani, H. Shi, Z. Ke, A. Xu, X. He, Y . Zhou, S. Yavuz, H. Wang, and S. Joty, “Mas-prove: Understanding the process verification of multi-agent systems,” 2026. [Online]. Available: https://arxiv.org/abs/2602.03053

    arXiv 2026