Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

Single-agent or Multi-agent Systems? Why Not Both?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-agent LLM systems lose most of their accuracy advantage over a single agent as models improve, and a hybrid that routes requests by difficulty outperforms both while cutting cost.

desk verdict The headline claim about diminishing MAS advantage is contradicted by the paper's own within-family comparisons in the appendix; the confounded Table 2 can't carry the load. read the letter →

arxiv 2505.18286 v1 pith:VCGMAFFK submitted 2025-05-23 cs.MA cs.AIcs.LG

Mingyan Gao , Yanzi Li , Banruo Liu , Yifan Yu , Phillip Wang , Ching-Yu Lin , Fan Lai This is my paper · ORCID
classification cs.MAcs.AIcs.LG
keywords multi-agentsystemssingle-agentlargelanguagemodelsagentroutingcascadecost-efficientdeploymentconfidence-guidedtracingMASdefectanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model systems built from multiple specialized agents have been promoted as more accurate than a single agent, but this paper argues that the advantage is largely a property of weaker base models. Across 15 tasks spanning code generation, math reasoning, software engineering, travel planning, finance, retrieval-augmented QA, and scientific experimentation, the authors find that multi-agent systems (MAS) and single-agent systems (SAS) tie on roughly 80% of examples, and MAS wins shrink as the underlying LLM gets stronger. MAS still helps on the hardest problems, but typical deployments pay 4 to 220 times more input tokens and 2 to 12 times more response tokens for small or negative accuracy gains. The authors attribute the gap to three defect classes, node, edge, and path, and use a confidence-guided tracing method to locate the bottleneck agent. They then propose routing requests to SAS or MAS by difficulty and a cascade that uses SAS first and escalates to MAS only when verification fails, reporting 1.1-12% better accuracy at up to 20% lower deployment cost.

What carries the argument

The paper models agentic execution as a directed graph $G = (V, E)$ with LLM agents and tools as nodes and inter-agent messages as edges, defining SAS as $|V_{llm}| = 1$ and MAS as $|V_{llm}| > 1$, with cost as the token-weighted sum over messages. The named diagnostic mechanism is confidence-guided critical path tracing: each agent reports a confidence $c_i$ on a 1-10 scale, and after each request the importance score $$I_i = \sum_{r \in R} Q_r c_i + (s - c_i)(1 - Q_r)$$ is accumulated, where $s = 10$ and $Q_r$ is 1 if the final output is correct and 0 otherwise. The lowest-scoring agent is flagged as the bottleneck. The optimization mechanism is the hybrid paradigm: an LLM rater assigns difficulty scores and routes requests between SAS and MAS, plus a cascade that runs SAS first, verifies the output, and escalates to MAS only on failure.

What would settle it

Run the same MAS/SAS benchmark pair on the next frontier model generation: if the MAS-win share stops declining as capability improves, the diminishing-benefit trend fails. Separately, compare importance-score rankings against full one-agent-at-a-time ablations on a held-out set, since the tracing method fails if self-reported confidence is uncorrelated with actual contribution.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the accuracy premium of MAS over SAS is not intrinsic: it shrinks as LLM capability improves and can invert on simple tasks where MAS overthinking hurts. Using the same frameworks that originally reported roughly 9-16% gains for ChatGPT, the authors measure gains of about 0.8-3.0% with Gemini-2.0-Flash, while MAS consumes 4-220 times more prefill tokens. Datapoint-level analysis shows most examples are ties, and MAS wins are outweighed by SAS wins on several datasets. The paper also introduces a hybrid paradigm: an LLM rater routes requests by difficulty, and a cascade verifies SAS outputs and escalates failures to MAS; on evaluated tasks this improves accuracy by 1.1-12% while cutting deployment cost by up to 20%.

Load-bearing premise

The load-bearing premise is that an LLM agent's self-reported confidence score is informative about how much that agent caused the final output to be correct; the paper gives no calibration evidence for this, and without it the importance-score ranking that identifies critical agents is not trustworthy.

Editorial extensions

If this is right

  • Deploying MAS as the default for a new agentic task is hard to justify on accuracy alone for frontier models; the workloads where MAS wins are concentrated on very hard inputs.
  • Upgrading only the identified critical agent captures much of the accuracy gain of upgrading all agents, so selective augmentation is a cost-effective lever.
  • Routing easy requests to SAS and hard requests to MAS can beat either paradigm alone on accuracy at lower cost, without fine-tuning a difficulty model.
  • Cascade designs are only applicable when the final output can be verified cheaply; for open-ended or subjective outputs the extra verification cost must be added.
  • The three defect classes give a practical checklist: check whether a single agent caps performance, whether downstream agents are overloaded by upstream messages, and whether summaries lose information that then propagates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the confidence-guided tracing result generalizes, MAS frameworks could autotune themselves by replacing one agent at a time; a natural test is whether importance scores correlate with ablation gains across more frameworks and models.
  • The diminishing advantage suggests a moving target: each MAS design is benchmarked against the SAS of its day, so MAS research may need to focus on tasks where decomposition genuinely adds capability, such as long-horizon tool use, rather than reasoning alone.
  • The paper's stated scope, general-purpose models and task-solving agentic applications, leaves open whether fine-tuned or domain-specialized agents would restore a larger MAS advantage.
  • The cascade's gain comes from SAS being cheap and often right; as verification becomes cheaper or SAS accuracy rises, the cascade's benefit should increase, which is testable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a broad empirical comparison of single-agent (SAS) and multi-agent (MAS) LLM systems across 15 datasets and 9 frameworks, concluding that MAS accuracy advantages over SAS shrink as base models become more capable while MAS cost penalties remain. It then proposes a confidence-guided method for identifying critical agents in a MAS, and two hybrid mechanisms—agent routing and agent cascade—that alternate between SAS and MAS to improve the accuracy-cost tradeoff. The manuscript includes extensive appendix tables, ablations, case studies, and a human-annotated software-engineering evaluation.

Significance. If the central empirical claim were established, the paper would provide timely evidence for a reassessment of MAS deployment as frontier LLMs improve. The breadth of the evaluation—multiple model families (GPT, Gemini, Llama), multiple MAS frameworks, and both discrete and continuous metrics—is a genuine strength, and the graph-based defect taxonomy (node, edge, path) is a useful organizing framework. The routing and cascade designs are practical engineering contributions, and the cost analyses with prefill/decode token accounting are valuable. However, the headline claim is not currently supported by controlled comparisons, and the cascade accuracy improvement is a construction-level guarantee rather than an empirical discovery, so the paper's contributions need to be re-scoped.

major comments (4)
  1. [§3.2, Tables 2, 6-9, 11-12] The central claim that 'the benefits of MAS over SAS diminish as LLM capabilities improve' is not established by the paper's own controlled data. Table 2 compares original ChatGPT-era published results with Gemini-2.0-Flash runs, changing model family, prompt version, and—per Appendix B.1—also changing the SelfCol architecture from three agents to two with revised prompts. Within-family comparisons in Appendix C go in the opposite direction: on SelfCol-DS1000, Llama-3-8B (Table 7) shows MAS 25.3% vs SAS 14.1% (+11.2pp), while Llama-3-70B (Table 6) shows 47.7% vs 25.0% (+22.7pp). Similar increases appear for AIME for GPT-3.5→GPT-4o (+1.7pp in Table 9 to +5.0pp in Table 8) and for Gemini-1.5-Flash→Gemini-2.0-Flash (+5.0pp in Table 12 to +13.3pp in Table 11), and for HumanEval (+0.6pp to +3.1pp). The paper acknowledges AIME as a possible exception but not the DS1000/Llama cells or the GPT/Gemini math cells. The diminishing-benefits claim should be replaced by a per-cell analysis and a statistical test across the controlled within-family comparisons, or qualified to specific datasets and families.
  2. [§4.2, Table 4, Appendix B.4] The headline that agent cascade 'improves accuracy by 1.1-12%' is a mathematical guarantee rather than an empirical finding. Under exact verification, the cascade output is correct whenever either the SAS or MAS output is correct, so cascade accuracy is the union of the two accuracies. The paper itself states in Appendix B.4 that 'the agent cascade uniformly has higher accuracy than SAS and MAS, because as long as either SAS or MAS is correct, it would be correct in the agent cascade.' Table 4 should therefore be presented as an oracle-style upper bound, with the empirical contribution centered on the realized cost savings and the escalation rate. As reported, the accuracy comparison is internally valid but not a discovery about MAS/SAS behavior. The range also appears inconsistent: Table 4 shows a 13.3pp gain on Debate-AIME (38.3 vs 25.0), which exceeds the stated 'up to 12%' and the abstract's '1.1-12%'.
  3. [§4.1, Appendix B.2, Figure 4a] The confidence-guided critical-path method hinges on the assumption that the self-reported confidence c_i is informative about an agent's contribution, but no calibration evidence is provided. The importance score I_i = Σ_r Q_r c_i + (s−c_i)(1−Q_r) is a linear combination of the binary final outcome and the self-report; if c_i is uninformative, the critical-agent ranking in Figure 4a is not reliable. The paper should report a calibration analysis (e.g., reliability of confidence as a predictor of per-agent correctness across rounds) or an alternative robustness check, such as comparing the selected critical agent against an exhaustive leave-one-out upgrade/downgrade evaluation, which the current single math-reasoning experiment only partially covers.
  4. [All tables; abstract; §4.2] The quantitative comparisons are presented as point estimates without error bars, confidence intervals, or significance tests, despite small sample sizes in several datasets (AIME has 60 problems; BigCodeBench-Hard has 148). Claims such as a 1.1-12% accuracy improvement or a 4-220x token ratio should be accompanied by uncertainty estimates and a statement of the number of independent runs (e.g., temperature variations are reported but used as evidence of consistency rather than as repeated trials). The paper also reports different cost-savings figures in the abstract (up to 20%), the introduction (up to 88.1%), and the conclusion (88.1%); these need to be reconciled and attributed to routing versus cascade.
minor comments (6)
  1. [Abstract / Introduction / Conclusion] The cost-savings numbers are inconsistent: the abstract says 'reducing deployment costs by up to 20%,' the introduction says 'up to 88.1%,' and the conclusion repeats 88.1%; please state which mechanism produces which saving and make the figures consistent.
  2. [§3.2 vs Appendix B.1] The main text says the evaluation covers 'Gemini-2.5-Pro, Gemini-2.5-Flash,' but Appendix B.1 lists Gemini-2.0-Flash-001, Gemini-2.5-Pro, GPT-3.5-Turbo, GPT-4o, and LLaMA-3.1 models; Gemini-2.5-Flash does not appear in the experiment setup.
  3. [Table 3] The column header contains a typo ('Deocde' should be 'Decode'); also, the reasons for the very large spread in prefill token ratios (from 1.2x on HoVer to 220x on AIME) are not discussed.
  4. [Figure 4b/c] The two panels use different threshold axis ranges (0-5 and 0-10), and neither reports the fraction of requests actually routed to MAS or the escalation rate; reporting p or the escalation ratio would make the cost-accuracy tradeoff interpretable.
  5. [§4.2 cost formula] The cascade cost formula omits the verification cost; Appendix B.4 acknowledges this cannot be ignored when verification uses humans or LLM judges. Please state the exact-match or test-based verification assumption in the main text.
  6. [Appendix D, Figure 10] The early-stopping ablation with intermediate message truncation is an interesting experiment, but the text should clarify whether the 25% figure refers to the fraction of messages that are necessary for the final output, or to the fraction of length that suffices; the caption and text use both phrasings.

Circularity Check

1 steps flagged · score 6.0 of 10

Agent cascade accuracy gain is guaranteed by construction; the MAS-vs-SAS comparison itself is empirical and non-circular.

  1. self definitional [Section 4.2 (Agent Cascade), Table 4, Appendix B.4]
    "It should be noted that the agent cascade uniformly has higher accuracy than SAS and MAS, because as long as either SAS or MAS is correct, it would be correct in the agent cascade."

    The cascade is defined as: run SAS first; if the verifier accepts the SAS output, return it; otherwise escalate to MAS. Therefore, for every request, the cascade is correct exactly when SAS is correct OR MAS is correct, so its accuracy is the union of the two systems' correct sets. This union is always at least as large as each individual set, so Table 4's reported accuracy improvements of 1.1-12% follow by elementary set theory from the cascade's own definition plus the measured SAS/MAS accuracies. The paper acknowledges this in Appendix B.4, making the 'accuracy improvement' a logical consequence of the construction rather than an empirical discovery. Only the cost reduction (up to 20%, or 88.1% elsewhere) is empirically determined and independent of the construction.

full rationale

The paper's primary empirical claim — that MAS advantages over SAS diminish as LLM capabilities improve — is supported by a comparative study across tasks and models (Section 3, Tables 2-3, Figures 2). This comparison is data-driven and does not reduce to any fitted parameter or self-citation; no load-bearing self-citations or imported uniqueness theorems appear. The one genuine reduction-by-construction is in the Agent Cascade design (Section 4.2, Table 4). Because the cascade routes to MAS only when SAS fails verification, its correctness is the union of SAS and MAS correctness, so its accuracy is guaranteed to be at least as high as both. The paper itself states this explicitly in Appendix B.4. Thus the headline accuracy gains of 1.1-12% are tautological consequences of the design definition, not novel empirical evidence; the cost savings are the empirically meaningful component. This partial circularity affects a central design claim, warranting a score of 6, while the MAS-vs-SAS analysis remains independent and non-circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or computational entities; its designs are compositions of existing components. The main hidden load is the uncalibrated use of self-reported confidence and the exact-verification assumption for cascades.

free parameters (2)
  • Confidence scale s = 10 (scale 1-10)
    In Section B.2, the paper sets s=10 because LLMs cannot output calibrated float confidences; the importance score I_i depends on this arbitrary scale.
  • Routing difficulty threshold = user-defined (swept 0-10 in Figure 4b)
    Agent routing accuracy and cost depend on a threshold that is not learned or set by a principled rule; the paper treats it as a user choice.
assumptions (4)
  • domain assumption Self-reported confidence correlates with per-agent contribution to correctness.
    Invoked in Section 4.1 to compute importance scores and identify critical agents; no calibration evidence is provided.
  • domain assumption Cascade verification is exact for the studied tasks.
    Section B.4 notes the cascade works only when outputs can be accurately evaluated; the cost and accuracy numbers assume perfect verification via pass@1 or exact match.
  • domain assumption Historical MAS results are comparable to re-runs on Gemini-2.0-Flash.
    Table 2 contrasts original ChatGPT-era reported numbers with new Gemini runs while SelfCol prompts and agent counts were modified, as described in Appendix B.1.
  • domain assumption The token cost model prefill + 2*decode approximates real API cost.
    Sections B.3 and B.4 use this fixed weighting to compute all cost savings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Single-agent or Multi-agent Systems? Why Not Both?." pith.science (2026). https://pith.science/paper/VCGMAFFK

@misc{pith2026250518286,
  author       = {Pith},
  title        = {Pith review of: Single-agent or Multi-agent Systems? Why Not Both?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VCGMAFFK}},
  note         = {Machine review of arXiv:2505.18286}
}
read the original abstract

Multi-agent systems (MAS) decompose complex tasks and delegate subtasks to different large language model (LLM) agents and tools. Prior studies have reported the superior accuracy performance of MAS across diverse domains, enabled by long-horizon context tracking and error correction through role-specific agents. However, the design and deployment of MAS incur higher complexity and runtime cost compared to single-agent systems (SAS). Meanwhile, frontier LLMs, such as OpenAI-o3 and Gemini-2.5-Pro, have rapidly advanced in long-context reasoning, memory retention, and tool usage, mitigating many limitations that originally motivated MAS designs. In this paper, we conduct an extensive empirical study comparing MAS and SAS across various popular agentic applications. We find that the benefits of MAS over SAS diminish as LLM capabilities improve, and we propose efficient mechanisms to pinpoint the error-prone agent in MAS. Furthermore, the performance discrepancy between MAS and SAS motivates our design of a hybrid agentic paradigm, request cascading between MAS and SAS, to improve both efficiency and capability. Our design improves accuracy by 1.1-12% while reducing deployment costs by up to 20% across various agentic applications.

Figures

Figures reproduced from arXiv: 2505.18286 by the authors.

Figure 1
Figure 1. Overview of the paper. We present a comprehensive empirical comparison of MAS and SAS [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SAS can achieve comparable accuracy performance to MAS. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. We propose a lightweight, confidence-guided probing method to identify critical agents for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: (a) Our confidence-guided tracing method can identify the correct critical agent to augment. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Experiment result of TDAG. 1 50 100 Problem ID 0.0 0.2 0.4 0.6 0.8 1.0 Score SAS MAS (a) Gemini-1.5-Flash 1 50 100 Problem ID 0.0 0.2 0.4 0.6 0.8 1.0 Score SAS MAS (b) Gemini-2.0-Flash 1 50 100 Problem ID 0.0 0.2 0.4 0.6 0.8 1.0 Score SAS MAS (c) GPT-4o [PITH_FULL_IMA…
Figure 6
Figure 6. Figure 6: Experiment result of HoVer [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Experiment result of FinRobot. 1 2 3 4 5 6 # Agents 0 20 40 60 Accuracy (%) T = 0.0 T = 0.4 T = 0.8 (a) Gemini-2.0-Flash (AIME) 1 2 3 4 5 6 7 # Agents 90 95 100 Accuracy (%) Concatenate Summarize (b) Gemini-2.0-Flash (GSM8K) 1 2 3 4 5 6 7 # Agents 90 95 100 Accuracy (%…
Figure 8
Figure 8. Figure 8: Impact of the number of solvers in the Math Debate task. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Impact of the number of rounds of conversation in the Math Debate task. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Impact of early-exit during message passing. “Inter. Early-Stop Length” refers to [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Multi-agent LLM systems help exactly when the context removed by compressed relays outweighs the downstream-relevant information those relays discard.

  2. CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

    cs.MA 2025-09 conditional novelty 6.0 of 10

    CORRECT distills recurring multi-agent failure patterns into reusable error schemas and retrieves them at inference time to localize the decisive error step more accurately than judging or fine-tuning baselines.

  3. Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference

    cs.AI 2025-09 reject novelty 5.0 of 10

    A hierarchical causal attribution framework (performance causal inversion, Shapley values, CDC-MAS step discovery) localizes failure agents and steps in LLM multi-agent systems, reporting up to 36.2% step accuracy.

  4. Rethinking the Illusion of Thinking

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Reasoning models' Towers of Hanoi failures persist under stepwise prompting, while River Crossing failures mostly vanish when tests are restricted to solvable configurations.

  5. BetaWeb: Towards a Blockchain-enabled Trustworthy Agentic Web

    cs.MA 2025-08 unverdicted novelty 4.0 of 10

    BetaWeb promises a blockchain-enabled trustworthy agentic web, but the submitted manuscript body is a different mining-robot paper, leaving the proposal without supporting evidence.

Reference graph

Works this paper leans on

147 extracted references · 66 canonical work pages · cited by 5 Pith papers

  1. [1]

    https://artofproblemsolving.com/ wiki/index.php/2025_AIME_I, 2025

    American Invitational Mathematics Examination. https://artofproblemsolving.com/ wiki/index.php/2025_AIME_I, 2025

    2025

  2. [2]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent llm systems fail?, 2025

    2025

  4. [4]

    Gamegpt: Multi-agent collaborative framework for game development, 2023

    Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. Gamegpt: Multi-agent collaborative framework for game development, 2023

    2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  6. [6]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors, 2023

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors, 2023

    2023

  7. [7]

    Enhancing diagnostic capability with multi-agents conversational large language models

    Xi Chen, Huahui Yi, Mingke You, WeiZhi Liu, Li Wang, Hairui Li, Xue Zhang, Yingman Guo, Lei Fan, Gang Chen, Qicheng Lao, Weili Fu, Kang Li, and Jian Li. Enhancing diagnostic capability with multi-agents conversational large language models. npj Digital Medicine, 8:159, 2025

    2025

  8. [8]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    2025

Show all 147 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  2. [10]

    Gemini 2.5 pro: Our most intelligent ai model

    Google DeepMind. Gemini 2.5 pro: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ ,

    2025

  3. [11]

    Self-collaboration code generation via chatgpt

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590, 2023

    2023 arXiv

  4. [12]

    Self-collaboration code generation via chatgpt, 2024

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt, 2024

    2024

  5. [13]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023

    2023

  6. [14]

    Improv- ing factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. 10

    2023 arXiv

  7. [15]

    Alireza Ghafarollahi and Markus J. Buehler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning, 2024

    2024

  8. [16]

    Cognify: Supercharging gen-ai workflows with hierarchical autotuning, 2025

    Zijian He, Reyna Abhyankar, Vikranth Srivatsa, and Yiying Zhang. Cognify: Supercharging gen-ai workflows with hierarchical autotuning, 2025

    2025

  9. [17]

    Cognify: Supercharging genai workflows with hierarchical autotuning

    Zijian He, Reyna Abhyankar, Vikranth Srivatsa, and Yiying Zhang. Cognify: Supercharging genai workflows with hierarchical autotuning. arXiv preprint arXiv:2502.08056, 2025

    2025 arXiv

  10. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  11. [19]

    Metagpt: Meta programming for a multi-agent collaborative framework, 2024

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative frame...

    2024

  12. [20]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative fr...

    2024

  13. [21]

    Hover: A dataset for many-hop fact extraction and claim verification

    Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. Hover: A dataset for many-hop fact extraction and claim verification. arXiv preprint arXiv:2011.03088, 2020

    2011 arXiv

  14. [22]

    SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Curie: Toward rigorous and automated scientific experimentation with ai agents, 2025

    Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, and Ang Chen. Curie: Toward rigorous and automated scientific experimentation with ai agents, 2025

    2025

  16. [24]

    Ds-1000: A natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. ...

    2023

  17. [25]

    Paperqa: Retrieval-augmented generative agent for scientific research

    Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559, 2023

    2023 arXiv

  18. [26]

    Improving llm classification of logical errors by integrating error relationship into prompts

    Yanggyu Lee, Suchae Jeong, and Jihie Kim. Improving llm classification of logical errors by integrating error relationship into prompts. In Angelo Sifaleras and Fuhua Lin, editors, Generative Intelligence and Intelligent Tutoring Systems, pages 91–103, Cham, 2024. Springer Nat...

    2024

  19. [27]

    Macm: Utilizing a multi-agent system for condition mining in solving complex mathematical problems, 2024

    Bin Lei, Yi Zhang, Shan Zuo, Ali Payani, and Caiwen Ding. Macm: Utilizing a multi-agent system for condition mining in solving complex mathematical problems, 2024

    2024

  20. [28]

    Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems, 2025

    Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, ...

    2025

  21. [29]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/ , 2025. Accessed: 2025-05-15

    2025

  22. [30]

    Hyperagent: Generalist software engineering agents to solve coding tasks at scale

    Huy Nhat Phan, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2406.11912, 2024

    2024 arXiv

  23. [31]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  24. [32]

    Chatdev: Communicative agents for software development, 2024

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development, 2024

    2024

  25. [33]

    Detecting hallucinations in large language model generation: A token probability approach, 2024

    Ernesto Quevedo, Jorge Yero, Rachel Koerner, Pablo Rivas, and Tomas Cerny. Detecting hallucinations in large language model generation: A token probability approach, 2024

    2024

  26. [34]

    Teola: Towards end-to-end optimization of llm-based applications, 2025

    Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. Teola: Towards end-to-end optimization of llm-based applications, 2025

    2025

  27. [35]

    Towards end-to-end optimization of llm- based applications with ayo

    Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. Towards end-to-end optimization of llm- based applications with ayo. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, page 1302...

    2025

  28. [36]

    Tdag: A multi-agent framework based on dynamic task decomposition and agent generation

    Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. Tdag: A multi-agent framework based on dynamic task decomposition and agent generation. arXiv preprint arXiv:2402.10178, 2024

    2024 arXiv

  29. [37]

    Tdag: A multi-agent framework based on dynamic task decomposition and agent generation, 2025

    Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. Tdag: A multi-agent framework based on dynamic task decomposition and agent generation, 2025

    2025

  30. [38]

    Agent workflow memory, 2024

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory, 2024

    2024

  31. [39]

    Stateflow: Enhancing llm task-solving through state-driven workflows, 2024

    Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing llm task-solving through state-driven workflows, 2024

    2024

  32. [40]

    Tradingagents: Multi-agents llm financial trading framework, 2025

    Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. Tradingagents: Multi-agents llm financial trading framework, 2025

    2025

  33. [41]

    Exploring large language models for communication games: An empirical study on werewolf, 2024

    Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf, 2024

    2024

  34. [42]

    Finrobot: An open- source ai agent platform for financial applications using large language models

    Hongyang Yang, Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Junlin Wang, Tianyu Zhou, Mao Guan, Runjia Zhang, and Christina Dan Wang. Finrobot: An open- source ai agent platform for financial applications using large language models. arXiv preprint arXiv:2405.14767, 2024

    2024 arXiv

  35. [43]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212, 2025

    2025 arXiv

  36. [44]

    Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024

    2024

  37. [45]

    Dpmac: Differentially private communication for cooperative multi-agent reinforcement learning, 2023

    Canzhe Zhao, Yanjie Ze, Jing Dong, Baoxiang Wang, and Shuai Li. Dpmac: Differentially private communication for cooperative multi-agent reinforcement learning, 2023

    2023

  38. [46]

    TicTacToe

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15...

    2024 arXiv

  39. [48]

    Agent gpt -4 o1 - - - - - - - - - - - - -

    \\ cos \\ theta = 324\\ cos \\ theta \ n \\]\ n \ nThe i m a g i n a r y part is :\ n \\[\ n \\ text { Im }(468 ie ^{ i \\ theta } + 36 ie ^{ - i \\ theta }) = \\ text { Im }(468 i (\\ cos \\ theta + i \\ sin \\ theta ) + 36 i (\\ cos ( -\\ theta ) + i \\ sin ( -\\ theta ) ) )...

    2025

  40. [49]

    Therefore , Sally Four is 8 years old

    ** Sally Four ’ s age :** Sally Four is the same age as Sally Thirtytwo , who is 8 years old . Therefore , Sally Four is 8 years old

  41. [52]

    \ boxed {51} gemini -1.5 - flash1 : ...( Almost i d e n t i c a l to answer 0) Therefore , the total age of the four birds is 51

    ** Total age :** To find the total age of the four birds , we add the ages of all four birds : 8 ( Sally Four ) + 16 ( Granny Red ) + 19 ( Sally Two ) + 8 ( Sally T h i r t y t w o ) = 51 years old . \ boxed {51} gemini -1.5 - flash1 : ...( Almost i d e n t i c a l to answer 0...

  42. [53]

    Therefore , Sally Four is 8 years old

    ** Sally Four ’ s age :** We know Sally Four is the same age as Sally Thirtytwo , who is 8 years old . Therefore , Sally Four is 8 years old

  43. [54]

    ** Granny Red ’ s age :** Granny Red is twice as old as Sally Four , so Granny Red is 2 * 8 = 16 years old

  44. [55]

    ** Sally Two ’ s age :** Sally Two is three years older than Granny Red , so Sally Two is 16 + 3 = 19 years old

  45. [56]

    Sally Four

    ** Total age :** To find the total age of the four birds , we need to know the age of the fourth bird . The problem only gives us i n f o r m a t i o n about three birds ( Sally Two , Granny Red , and Sally Four / Sally T h i r t y t w o ) . We are missing the age of the fourt...

    2025

  46. [57]

    i n t e r m e d i a t e

    ** Choose D i f f i c u l t y Level **: The game starts at the b eg in ne r level by default . You can change the d i f f i c u l t y level by m o d i f y i n g the ‘ level ‘ var ia bl e in the ‘ main . py ‘ file to " i n t e r m e d i a t e " or " expert ". ... Example 3. Thi...

    2025

  47. [58]

    I m p l e m e n t the f un cti on body for the f unc ti on s p e c i f i e d in the r e q u i r e m e n t s

  48. [60]

    If you want , you could import more modules but still keep the or ig in al ones

    Include the i m p o r t i n g modules , the fu nc tio n d e f i n i t i o n or d o c s t r i n g ** exactly as the user pr ov ide d **. If you want , you could import more modules but still keep the or ig in al ones

  49. [63]

    Write a short code f o l l o w i n g the given format and i n d e n t a t i o n

    I m p l e m e n t the logic that s a t i s f i e s all ex am pl es in the d o c s t r i n g Begin your re sp on se with only the code - do not include any other e x p l a n a t i o n s before the code itself . DS1000 " Write a short code f o l l o w i n g the given format and ...

  50. [67]

    Follow best p r a c t i c e s for Python coding style Begin your re sp on se with only the i m p l e m e n t a t i o n : """ BigCodeBench We use the original prompt in the repository. Please provide a self - c o n t a i n e d Python script that solves the f o l l o w i n g pro...

  51. [70]

    ACTION : the action you call to get i n f o r m a t i o n or submit the task

    Your re sp ons e should be f o r m a t t e d as follows : THOUGHT : the thought process to achieve the goal , i n c l u d i n g the subtask to be handled . ACTION : the action you call to get i n f o r m a t i o n or submit the task . EX TE RN AL R E S O U R C E S :

  52. [72]

    { T A S K _ D E S C R I P T I O N } --- Your Wo rk fl ow ---

    A python no te bo ok to execute python code for n u m e r i c a l o p e r a t i o n s and p la nni ng . { T A S K _ D E S C R I P T I O N } --- Your Wo rk fl ow ---

  53. [74]

    You need to give your ideas until you need to use ex te rna l tools to obtain i n f o r m a t i o n or calculations , and then call an action and waif for the result

    Then you will start to solve it step by step . You need to give your ideas until you need to use ex te rna l tools to obtain i n f o r m a t i o n or calculations , and then call an action and waif for the result

  54. [75]

    < action > over () </ action >

    Finally , call over () to in di ca te task c o m p l e t i o n . After the content in task r e q u i r e m e n t s is completed , call over () i m m e d i a t e l y without making u n n e c e s s a r y pl an ni ng . --- A v a i l a b l e Actions --- { O R I G I N A L _ D O C U...

  55. [76]

    Thin k first , then ** output only the final answer ** , with no explanation , prefixes , markdown , code blocks , or JSON

  56. [77]

    p os it ive

    Output format by task : - s e n t i m e n t _ a n a l y s i s -> exactly one of : " p os it ive " , " n eg at iv e " , " neutral " - h e a d l i n e _ c l a s s i f i c a t i o n -> exactly the dataset ’ s class label ( e . g . , " e co no mi c " , " energy " , " tech " , ...)...

  57. [78]

    If history is empty you may ignore it ; o t h e r w i s e le ve rag e it to improve your answer

  58. [79]

    Final Answer

    Do not apologize , echo the question , or add phrases like " Final Answer " Re me mb er : the user must see only the answer string . RAG-based QA You are an i n f o r m a t i o n s y n t h e s i s expert . You will receive :

  59. [80]

    A * claim * that needs fact - c he ck in g

  60. [81]

    Curie - Solo

    A list of r e t r i e v e d passages , each f o r m a t t e d as : [ < index >] ( score = < score > , id = < pid >) < passage text > Your task : choose ** the 10 pa ssa ge s whose content is most re le va nt for v e r i f y i n g the claim **. Return exactly 10 pids , comma - ...

  61. [82]

    v a l i d a t e d _ q u e s t i o n

    ** Va li da te & Refine the Qu es ti on ** - Parse the user ’ s input to ensure it d e s c r i b e s a clear , f eas ib le e x p e r i m e n t a l problem . - If unclear , ask follow - up q u e s t i o n s to refine scope . - Output a JSON field ‘" v a l i d a t e d _ q u e s ...

  62. [83]

    e x p e r i m e n t _ p l a n

    ** Design E x p e r i m e n t Plan ** - F o r m u l a t e a h y p o t h e s i s . - I de nti fy i n d e p e n d e n t / d e p e n d e n t variables , controls , re qu ire d data , and e v a l u a t i o n metrics . - Break the plan into ordered steps , each with a concise d e s...

  63. [84]

    i m p l e m e n t a t i o n

    ** I m p l e m e n t & Execute ** - G en era te r e p r o d u c i b l e code or c om man ds to carry out each step . - Ensure r e p r o d u c i b i l i t y ( e . g . , specify environment , seed values ) . - Output a JSON field ‘" i m p l e m e n t a t i o n " ‘ c o n t a i n ...

  64. [85]

    v e r i f i c a t i o n

    ** Verify & Debug ** - V al ida te that each i m p l e m e n t a t i o n step runs without errors . - If errors occur , debug and patch code . - Output a JSON field ‘" v e r i f i c a t i o n " ‘ s u m m a r i z i n g any issues e n c o u n t e r e d and fixes applied

  65. [86]

    a na lys is

    ** Analyze Results ** - Process raw outputs to compute metrics , create summaries , or v i s u a l i z a t i o n s . - I n t e r p r e t fi nd in gs in context of the h y p o t h e s i s . - Output a JSON field ‘" a na lys is " ‘ with key results and brief i n t e r p r e t a ...

  66. [87]

    c o n c l u s i o n

    ** Co nc lu de & Report ** - Draw overall conclusions , discuss limitations , and suggest follow - up work . - Compile a concise report . - Output a JSON field ‘" c o n c l u s i o n " ‘ with your summary . ** Output Format ( JSON ) :** ‘‘‘ json 34 { " v a l i d a t e d _ q u ...

  67. [88]

    I m p l e m e n t the f un cti on body for the f unc ti on s p e c i f i e d in the a nal ys is

  68. [90]

    If you want , you could import more modules but still keep the or ig in al ones

    Include the i m p o r t i n g modules , the fu nc tio n d e f i n i t i o n or d o c s t r i n g ** exactly as the user pr ov ide d ** and the an al ys is s p e c i f i e d . If you want , you could import more modules but still keep the or ig in al ones

  69. [91]

    Do not include any e x p l a n a t i o n s or co mm en ts

  70. [92]

    Follow the f un ct io n s i g n a t u r e exactly

  71. [93]

    Handle all edge cases from the an al ys is

  72. [94]

    d a t a _ m a n i p u l a t i o n

    I m p l e m e n t the logic that s a t i s f i e s all ex am pl es in the d o c s t r i n g Begin your re sp on se with i m p o r t i n g modules : DS1000 You are a s p e c i a l i z e d data science code a na ly ze r . Analyze the f o l l o w i n g data science p r o g r a m ...

  73. [95]

    I m p l e m e n t a co mpl et e so lu ti on that a d d r e s s e s the problem

  74. [96]

    Your code must work within the given code context - do not modify e xi st in g v a r i a b l e s

  75. [97]

    Follow the c o n v e n t i o n s of the main library in cl ud ed in the library section of the an al ys is

  76. [98]

    Use e f f i c i e n t and i d i o m a t i c a p p r o a c h e s sp ec ifi c to the library

  77. [99]

    Ensure your so lut io n handles all s p e c i f i e d r e q u i r e m e n t s

  78. [100]

    Comment your code to explain complex o p e r a t i o n s

  79. [101]

    Analyze the f o l l o w i n g Python p r o g r a m m i n g problem and provide a s t r u c t u r e d JSON s p e c i f i c a t i o n

    Focus on r e a d a b i l i t y and m a i n t a i n a b i l i t y Begin your re sp on se with only the i m p l e m e n t a t i o n ( code that should be in ser te d at the po si ti on i n d i c a t e d in the problem ) : MBPPYou are a s p e c i a l i z e d code r e q u i r e m ...

  80. [102]

    I m p l e m e n t a co mpl et e Python fu nc ti on that solves the problem

  81. [103]

    Use proper i n d e n t a t i o n (4 spaces )

  82. [104]

    Make sure your sol ut io n passes all the pr ov id ed test cases

  83. [105]

    Handle all edge cases i d e n t i f i e d in the a na lys is

  84. [106]

    Follow best p r a c t i c e s for Python coding style

  85. [107]

    Use a p p r o p r i a t e fu nct io n s i g n a t u r e as s u g g e s t e d in the a nal ys is Begin your re sp on se with only the i m p l e m e n t a t i o n : BigCodeBench You are ** Analyst ** , an elite algorithm - p la nn ing agent for coding p ro ble ms . Your primary ...

  86. [108]

    plan ": A d i c t i o n a r y c o n t a i n i n g the core p la nn in g details : -

    " plan ": A d i c t i o n a r y c o n t a i n i n g the core p la nn in g details : - " r e s t a t e d _ t a s k ": A concise , single - s en te nce p a r a p h r a s e of the problem . - " s i g n a t u r e ": The exact Python f un cti on s i g n a t u r e r equ ir ed by the...

  87. [109]

    tests ": A list of dictionaries , each r e p r e s e n t i n g a crucial edge - case or tricky test sc en ar io . Each test d i c t i o n a r y must contain : -

    " tests ": A list of dictionaries , each r e p r e s e n t i n g a crucial edge - case or tricky test sc en ar io . Each test d i c t i o n a r y must contain : - " input ": The input a r g u m e n t s for the function , f o r m a t t e d as a Python list or tuple c o r r e s ...

  88. [110]

    U n d e r s t a n d i n g the query , think step - by - step c a r e f u l l y before dec is io n to propose actions to collect n e c e s s a r y i n f o r m a t i o n to resolve the query

  89. [111]

    Only g en er at e 1 block of code at one time

    Do not repeat your actions . Only g en er at e 1 block of code at one time

  90. [112]

    DO NOT GUESS or refuse to re sp on se the planner ’ s request

    Try to use the tools to get the i n f o r m a t i o n you need . DO NOT GUESS or refuse to re sp on se the planner ’ s request . Planner request is always correct . You may only see part of the information , but the planner sees the whole picture

  91. [113]

    If you open a file , but do not find the i n f o r m a t i o n you need , reopen with d i f f e r e n t s t a r t _ l i n e and en d_ li ne or k ey wor ds

    If one tool does not find the i n f o r m a t i o n you need , try another tool . If you open a file , but do not find the i n f o r m a t i o n you need , reopen with d i f f e r e n t s t a r t _ l i n e and en d_ li ne or k ey wor ds

  92. [114]

    ### I m p o r t a n t Notes : 1 Only use the pr ov id ed and p r e d e f i n e d f u n c t i o n s as the f u n c t i o n s

    Your final answer should only include code sn ip pe ts that are re le va nt to query only . ### I m p o r t a n t Notes : 1 Only use the pr ov id ed and p r e d e f i n e d f u n c t i o n s as the f u n c t i o n s . Do not use any other f u n c t i o n s

  93. [115]

    Try to combine d i f f e r e n t tools to seek related i n f o r m a t i o n to the query inside the project

  94. [116]

    This will help you u n d e r s t a n d how the symbol is used in the c od eb as e

    f i n d _ a l l _ r e f s : Use this tool to get all r e f e r e n c e s to a symbol in the c od eb as e . This will help you u n d e r s t a n d how the symbol is used in the c od eb as e . For example , if you want to know where a f un cti on is called , you can use this tool

  95. [117]

    This will help you u n d e r s t a n d the s t r u c t u r e of the file and find the re le va nt symbols before opening the file

    g e t _ a l l _ s y m b o l s : Use this tool to get all symbols in the target file , it should be used with a keyword . This will help you u n d e r s t a n d the s t r u c t u r e of the file and find the re le va nt symbols before opening the file . If you want to look for ...

  96. [118]

    This will help you u n d e r s t a n d the o r g a n i z a t i o n of the codebase , and find the re le van t files to use other tools

    g e t _ f o l d e r _ s t r u c t u r e : Use this tool to get the s t r u c t u r e of the target folder . This will help you u n d e r s t a n d the o r g a n i z a t i o n of the codebase , and find the re le van t files to use other tools

  97. [119]

    c o d e _ s e a r c h : Use this tool to search for symbol name if you know the exact name of the symbol , this is useful to find the d e f i n i t i o n if you ’ re not fa mi li ar with co deb as e yet

  98. [120]

    _print in a class

    g o _ t o _ d e f i n i t i o n : Use this tool to na vi gat e to the d e f i n i t i o n of an identifier , for example self . _print in a class . ( single word only , not a c o m b i n a t i o n like sympy . latex ) , in this case , _print

  99. [121]

    Should be used with a keyword ( single word only , not a c o m b i n a t i o n like sympy

    o p e n _ f i l e : Use this tool to open a file in the codebase , this is useful to read the partial content of the file (40 lines ) . Should be used with a keyword ( single word only , not a c o m b i n a t i o n like sympy . latex just latex ) or limited s t a r t _ l i n e...

  100. [122]

    Action : ‘‘‘ python result = c o d e _ s e a r c h . _run ( names =[

    ** S e a r c h i n g for I d e n t i f i e r s **: A r g u m e n t s : - names : list [ str ] - The names of the i d e n t i f i e r s to search . I d e n t i f i e r should be a single word like ‘ some_function ‘ not ‘ s o m e t h i n g . something ‘" Action : ‘‘‘ python resu...

  101. [124]

    s o m e _ f u n c t i o n

    ** Finding All R e f e r e n c e s of a Symbol **: A r g u m e n t s : 46 - word : str - The alias name of the symbol to find r e f e r e n c e s for . - r e l a t i v e _ f i l e _ p a t h : str - The path to the file where the alias is used . - line : int - The line number w...

  102. [125]

    module / file . py

    ** Getting All Symbols from a File **: A r g u m e n t s : - p a t h _ t o _ f i l e : str - The path to the file to get all symbols from . - keyword : str - The keyword to filter the symbols . Action : ‘‘‘ python result = g e t _ a l l _ s y m b o l s . _run ( p a t h _ t o _...

  103. [127]

    Example : Action : ‘‘‘ python result = o p e n _ f i l e . _run ( r e l a t i v e _ f i l e _ p a t h =

    ** Opening a File and S e a r c h i n g Content **: A r g u m e n t s : - r e l a t i v e _ f i l e _ p a t h : str - The path to the file to open . - k ey wor ds : list [ str ] - The k ey wor ds to search in the file . - s t a r t _ l i n e : int - The start line to read the ...

  104. [128]

    file . py

    ** Finding a File in the R e p o s i t o r y **: A r g u m e n t s : - f i l e _ n a m e : str - The name of the file to find . Action : ‘‘‘ python result = f i n d _ f i l e . _run ( f i l e _ n a m e =" file . py ") print ( result ) ‘‘‘ Always replay with Thought and Action ...

  105. [129]

    Always open the file before editing to see latest code !

  106. [130]

    If you have to edit the code , ensure the code is correct with syntax , dependency , c o n s i s t e n t with the file and the co de bas e

  107. [131]

    You first can open a file to get the context and use editor to apply the patch

    Return final answer if your code is s u c c e s f u l l y applied . You first can open a file to get the context and use editor to apply the patch . You have to ge ne ra te code that is correct with syntax , ensure the dependency , c o n s i s t e n t with the file and the co de ba se

  108. [132]

    If there ’ s no definition , you can use open o p e n _ f i l e tool

    If you have the exact name of the file and symbol to edit , you can use the code search to find the d e f i n i t i o n of the symbol . If there ’ s no definition , you can use open o p e n _ f i l e tool

  109. [133]

    patch ":

    Pay a t t e n t i o n to or ig in al i n d e n t a t i o n ! S o m e t h i n g like this " patch ": " def s o m e t h i n g ( self , s ) : # Check if s o m e t h i n g is s o m e t h i n g return s o m e t h i n g if the or ig in al code is i nd en te d with 4 spaces or " def ...

  110. [134]

    ### F u n c t i o n s :

    The patch should be a block of code that be rep la ce d into the code . ### F u n c t i o n s :

  111. [135]

    module / file . py

    ** Editing A File with r ep la ce d code block **: A r g u m e n t s : - r e l a t i v e _ f i l e _ p a t h : str - The path to the file to edit . - s t a r t _ l i n e : int - The line number where the o rig in al target code block starts . - e nd _li ne : int - The line num...

  112. [136]

    module /

    ** E x p l o r i n g Folder Tree S t r u c t u r e **: A r g u m e n t s : - r e l a t i v e _ p a t h : str - The path to the folder to explore . - depth : int - The depth of the folder s t r u c t u r e to explore . Action : ‘‘‘ python result = g e t _ f o l d e r _ s t r u ...

  113. [137]

    module / file . py

    ** Opening a File and S e a r c h i n g Content **: A r g u m e n t s : - r e l a t i v e _ f i l e _ p a t h : str - The path to the file to open . Action : ‘‘‘ python result = o p e n _ f i l e _ g e n . _run ( r e l a t i v e _ f i l e _ p a t h =" module / file . py " , k ...

  114. [138]

    s o m e _ f u n c t i o n

    ** Finding D e f i n i t i o n of a Symbol **: A r g u m e n t s : - word : str - The alias name of the symbol to find the d e f i n i t i o n for . - r e l a t i v e _ p a t h : str - The path to the file where the alias is used . - line : int - The line number where the alia...

  115. [139]

    s o m e _ f u n c t i o n

    ** Finding All R e f e r e n c e s of a Symbol **: A r g u m e n t s : - word : str - The alias name of the symbol to find r e f e r e n c e s for . - r e l a t i v e _ f i l e _ p a t h : str - The path to the file where the alias is used . - line : int - The line number wher...

  116. [140]

    You should only issue one action at a time

  117. [141]

    You should reason step by step and then issue the next action

  118. [142]

    ACTION : the action you call to assign the subtask or submit the task

    Your re sp ons e should be f o r m a t t e d as follows : THOUGHT : the thought process to achieve the goal , i n c l u d i n g the subtask to be handled . ACTION : the action you call to assign the subtask or submit the task . EX TE RN AL R E S O U R C E S :

  119. [143]

    A d ata ba se c o n t a i n i n g i n f o r m a t i o n about train tickets , attractions , and city t r a n s p o r t a t i o n

  120. [144]

    find the c he ap es t train from Beijing to Sh an gha i in 2023 -7 -1

    A python no te bo ok to execute python code for n u m e r i c a l o p e r a t i o n s and p la nni ng . PLAN AND SUBTASK : If the subtask cannot be easily solved d ire ct ly or r eq ui res the use of ex te rna l resources , please assign it to another agent to co mp le te ( su...

    2023

  121. [145]

    You will first be given a task

  122. [146]

    < action > s u b a g e n t _ h a n d l e ( s u b t a s k _ n a m e ) </ action >

    Then you will start to solve it step by step . M ult ip le s ub ta sk s need to be c o m p l e t e d during the s ol ut io n process . If you cannot solve some su bt as ks independently , for example , it is d i f f i c u l t or r eq ui re s the use of ex te rn al resources , ...

  123. [147]

    After the content in task r e q u i r e m e n t s is completed , call over () i m m e d i a t e l y without making u n n e c e s s a r y pl an ni ng

    Finally , call over () to in di ca te task c o m p l e t i o n . After the content in task r e q u i r e m e n t s is completed , call over () i m m e d i a t e l y without making u n n e c e s s a r y pl an ni ng . --- A v a i l a b l e Actions ---

  124. [148]

    < action > s u b a g e n t _ h a n d l e ( s u b t a s k _ n a m e ) </ action > If you want to assign a subtask to other agents for completion , call < action > s u b a g e n t _ h a n d l e ( s u b t a s k _ n a m e ) </ action > Before you call it , give me the subtask s u ...

  125. [149]

    < action > over () </ action >

    < action > over () </ action > When you think the task is completed , call " < action > over () </ action >". After the content in task r e q u i r e m e n t s is completed , call over () i m m e d i a t e l y without making u n n e c e s s a r y p la nni ng . Now you will be ...

  126. [150]

    After r e c e i v i n g the re spo ns e from the agent , d i v e r s i f y your next subgoal to get more i n f o r m a t i o n

    Do not repeat your actions !. After r e c e i v i n g the re spo ns e from the agent , d i v e r s i f y your next subgoal to get more i n f o r m a t i o n

  127. [151]

    I de nt if y crucial causes of the query , lo cal iz e where the problem is before cho os in g the code editor intern

  128. [152]

    You only need to resolve the issue in the co de ba se

    No need to edit test file or test the code . You only need to resolve the issue in the co de ba se

  129. [153]

    You are only focused on the issue a ss ign ed to you

    Do not care about any Pull Request or Ex is ti ng Issue in the r e p o s i t o r y . You are only focused on the issue a ss ign ed to you

  130. [154]

    Only ge ne rat e the patch for the code , if you already know the issue , know about it causes , and re le va nt k n o w l e d g e to resolve it

  131. [155]

    Give a d et ai le d request to the intern , so that your intern can u n d e r s t a n d the context of the query as well

  132. [156]

    Request exe cu to r intern to run the test cases or run any n e c e s s a r y code snippet or r e p r o d u c e the issue

  133. [157]

    Only give task to one intern at a time

  134. [158]

    ### I m p o r t a n t Notes :

    If you are not c o n f i d e n t about your g e n e r a t e d code , you can ask the ex ec ut or to run the test casses . ### I m p o r t a n t Notes :

  135. [159]

    Make sure to id ent if y the key c o m p o n e n t s of the query and the ex pe ct ed b eh av ior

    Reading the query d e s c r i p t i o n and u n d e r s t a n d i n g the problem is the first step . Make sure to id ent if y the key c o m p o n e n t s of the query and the ex pe ct ed b eh av ior . Pay a t t e n t i o n into error trace

  136. [160]

    Reading the re spo ns e from the interns carefully , think about the i n f o r m a t i o n you have c o l l e c t e d and how it can be used to resolve the task

  137. [161]

    Your thought process is the most i m p o r t a n t part of this task . Make sure to provide a de ta il ed e x p l a n a t i o n of your r e a s o n i n g with program error trace , the issue , code sn ip pe ts and r el ev an t i n f o r m a t i o n c o l l e c t e d from the agents

  138. [162]

    flux ": [99.9 , 99.8 , 99.7]}) ts . _ r e q u i r e d _ c o l u m n s = [

    Stop the task when you have r eso lv ed the query . ( Final Answer ) ### Ex pe ct ed format : Thought : $Your t ho ro ug h th ou gh ts on the task and how you a d a p t i v e l y and p r o g r e s s i v e l y plan to resolve it based on o b s e r v a t i o n s from your intern...

  139. [2025]

    Accessed: 2025-05-15

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.