Pith. sign in

REVIEW 4 major objections 6 minor 82 references

AgentLocate localizes failures in LLM multi-agent systems to the responsible agent and the earliest decisive step by combining an LLM judge, multi-perspective evaluators, and evaluator-guided fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:07 UTC pith:HT66OEO2

load-bearing objection Practical Judge–Evaluator–LoRA pipeline that beats existing multi-agent failure localizers on the usual benchmarks; relative gains look real, absolute step accuracy stays modest, and the causal definition is only approximated. the 4 major comments →

arxiv 2607.07989 v1 pith:HT66OEO2 submitted 2026-07-08 cs.CR cs.AIcs.IRcs.LGcs.MA

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Yufei Xia , Anjun Gao , Yueyang Quan , Zhuqing Liu , Minghong Fang This is my paper
classification cs.CR cs.AIcs.IRcs.LGcs.MA
keywords multi-agent systemsfailure localizationLLM-as-judgeAgentLocatedecisive failure stepconfidence-weighted votingparameter-efficient fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When an LLM multi-agent system fails on a task, the distributed, long-horizon nature of the run makes it hard to say which agent first pushed the trajectory past recovery. This paper introduces AgentLocate, a practical localization pipeline that first has an LLM Judge propose a responsible agent and step, then has several independent Evaluators re-examine the same log under different prompting styles, aggregates their answers with confidence-weighted voting, and finally uses that verified feedback to lightly fine-tune the Judge. On the Who&When and Aegis-Bench suites the method raises both agent-level and step-level accuracy above prior localization techniques and above repurposed poisoning-forensics tools, while keeping token use and wall-clock time modest. The result is a concrete debugging aid: developers can point to the earliest decisive mis-action rather than guessing among many intertwined agents.

Core claim

Treating failure localization as a verifiable cycle—Judge hypothesis, multi-Evaluator critique with confidence-aware aggregation, and LoRA adaptation of the Judge—produces more accurate identification of both the responsible agent and the earliest decisive step than one-shot judges, counterfactual-replay tracers, taxonomy matchers, or poisoning-forensics methods, across short and long multi-agent trajectories.

What carries the argument

The Judge-Evaluator refinement cycle: an LLM Judge emits a candidate pair (agent, step); independent Evaluators, each with a distinct prompt style, re-score the same trajectory and report location, rationale and confidence; a confidence-weighted vote yields a verified label; that label, together with the rationales, becomes a LoRA training instance that adapts the Judge for subsequent localization.

Load-bearing premise

LLM reasoning over the logged trajectory can stand in for the true counterfactual test that defines the earliest decisive step, and the aggregated Evaluator labels are reliable enough to serve as supervision for improving the Judge.

What would settle it

On a held-out set of failed trajectories whose ground-truth decisive steps are known by actual counterfactual re-simulation, measure whether correcting the step predicted by AgentLocate reverses the failure more often than the steps predicted by the strongest baselines; if it does not, the localization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Debugging multi-agent workflows can begin at the earliest decisive mis-action rather than at the final wrong answer.
  • Poisoning-forensics tools that look for adversarial traces are the wrong instrument for ordinary coordination and reasoning failures.
  • A single Judge-Evaluator refinement round already captures most of the accuracy gain, so the method can be used as a lightweight post-mortem step.
  • Visibility bias toward late-stage verifier or retrieval agents can be measured and later mitigated by explicit error-propagation modeling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same verify-and-adapt loop could be applied to non-LLM multi-agent systems whose logs admit a similar decisive-step definition.
  • If Evaluator diversity is the main source of signal, cheaper non-LLM critics might later replace some of the LLM Evaluators without large accuracy loss.
  • Longer trajectories that currently dilute the failure signal may become easier once the Judge is trained to ignore post-decision noise rather than simply fine-tuned on final labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies failure localization in LLM-based multi-agent systems: given a failed trajectory, recover both the responsible agent and the earliest decisive step. It defines the decisive step t★(τ) as the earliest time whose single idealized counterfactual correction R(τ,t) would flip failure to success (Appendix B, Eqs. 7–9), and proposes AgentLocate, a Judge–Evaluator pipeline that (i) has an LLM Judge hypothesize (î,t̂) under all-at-once or step-by-step protocols, (ii) verifies with multiple independent Evaluators under diverse prompts and confidence-weighted aggregation, and (iii) adapts the Judge via LoRA on evaluator-enriched instances. Experiments on Who&When (Algorithm-Generated and Hand-Crafted) and Aegis-Bench report consistent gains over WhichAgent, AgenTracer, ECHO, AEGIS, and two poisoning-forensics baselines, with ablations on variants, evaluator count/model, refinement rounds, trajectory length, and efficiency (tokens, cost, runtime).

Significance. If the results hold under a well-aligned evaluation of the causal object the paper defines, this is a useful systems contribution for multi-agent reliability: a practical, relatively efficient localization pipeline with multi-model evidence, complementary benchmarks (short vs long trajectories; agent+step vs agent+error-mode), and explicit comparison to both attribution and forensics methods. Strengths include a clear problem statement, multi-perspective verification rather than one-shot judging, ablations of the refinement cycle (Variants I–III), and efficiency tables that make the method usable for debugging. The work is timely for multi-agent deployment and debugging, even if absolute step-level accuracy remains modest.

major comments (4)
  1. [Appendix B, Eqs. (7)–(9); §3.1; Tables 1–3] Appendix B, Eqs. (7)–(9) define t★(τ) as the earliest step for which the counterfactual trajectory R(τ,t) succeeds. Section 3.1 states that the Judge does not execute R and only approximates this criterion by reasoning over the logged trajectory. The manuscript never shows that Who&When / Aegis-Bench ground-truth labels were obtained by constructing and re-simulating R under the same policies and scheduler. Tables 1–3 therefore measure agreement with benchmark annotations, not necessarily recovery of the causal object defined in Appendix B. This is load-bearing for the central claim. Please either (a) validate a subset of labels by actual counterfactual replay, (b) restate claims as agreement with human/algorithmic earliest-error annotations rather than verified counterfactual flip points, or (c) provide evidence that the annotations match the R-based definition.
  2. [§3.2–3.3; Table 9; Table 13] The adaptive stage fine-tunes the Judge on evaluator-aggregated labels (Eqs. 5–6), often with Evaluators from the same model family as the Judge (default Qwen-7B). Table 13 shows Variant II (evaluator aggregation alone) is already strong, so the LoRA stage may largely amplify shared model biases rather than inject independent causal signal. The paper should quantify agreement between Judge, individual Evaluators, and ground truth before fine-tuning, report inter-evaluator agreement, and include a cross-family setting (e.g., Judge Qwen, Evaluators Llama/Mistral only) as a primary rather than secondary result (Table 9 is partial). Without this, the reported gains risk partial circular reinforcement of the same approximation used for labels.
  3. [Tables 1–2; §4.1 dataset splits] Who&When test splits are small (~40% of 126 Algorithm-Generated and of 58 Hand-Crafted cases). Point estimates such as 69.05% / 38.10% (Table 1, Qwen-7B all-at-once) are reported without confidence intervals, bootstrap standard errors, or significance tests against the strongest baseline. On Hand-Crafted (Table 2), step-level accuracy is low for all methods and differences can be a few cases. Please add uncertainty estimates and, where possible, paired tests; otherwise the claim of consistent outperformance is overstated relative to sample size.
  4. [Abstract; Table 3; §4.2 / Appendix G] AEGIS is agent-level only and ECHO is omitted on Aegis-Bench; pair-level gains on Aegis-Bench (Table 3) are modest (e.g., 10.50% vs 7.33% for AgenTracer under Qwen all-at-once). The abstract and conclusion claim strong localization of both agent and step across benchmarks. Please qualify claims for Aegis-Bench (agent vs pair) and avoid treating pair-level as equivalent to step-level localization. Also clarify whether AEGIS “–” step cells are excluded fairly from averages when summarizing “consistently outperforms.”
minor comments (6)
  1. [Figure 1] Figure 1 is a useful failure-propagation case; make the decisive step (Step 6) and agent names visually easier to parse (e.g., bold the decisive action and align step numbers with the narrative).
  2. [§2.1; §3] Notation: ρ(t) for the scheduler and i★=ρ(t★) is clear, but the main text sometimes uses (î,t̂) and (ieval,teval) without restating domains; a short notation table would help.
  3. [Appendix C–D; §4.1] Appendix C/D prompts are valuable; state temperature / decoding settings and whether Judge and Evaluators use identical decoding hyperparameters.
  4. [Tables 6–8; §4.2 efficiency] Tables 6–8 include training overhead for AgentLocate and AgenTracer; explicitly state whether baseline inference-only costs exclude any offline training those methods require, for a fair comparison.
  5. [Abstract; §1] Typos / polish: “AgentLocatecombines” spacing issues appear in the abstract and introduction; unify hyphenation of multi-agent and step-by-step throughout.
  6. [§2.2; Appendix A] Related work on poisoning forensics is appropriately distinguished in Appendix A; a one-sentence pointer in the main related-work section would help readers who skip the appendix.

Circularity Check

0 steps flagged

No significant circularity: AgentLocate is an empirical localization pipeline evaluated against external benchmark labels and baselines, not a first-principles derivation that reduces to its own inputs.

full rationale

The paper’s load-bearing claim is empirical outperformance on Who&When and Aegis-Bench (agent/step or agent/pair accuracy) versus WhichAgent, AgenTracer, ECHO, AEGIS, and two poisoning-forensics baselines. Those benchmarks supply independent ground-truth annotations; the method is trained on a disjoint refinement split and scored on held-out test splits. The Judge–Evaluator–LoRA loop is a disclosed training procedure (hypothesis → multi-evaluator aggregation → PEFT), not a definitional identity: ablations show that using only the Judge hypothesis, only evaluator aggregation, or direct fine-tuning on original ground-truth labels all underperform the full pipeline (Table 13), so the reported gains are not forced by construction. The counterfactual definition of t⋆(τ) via R(τ,t) (Appendix B, Eqs. 7–9) is approximated by LLM reasoning rather than executed (Section 3.1); that is a validity/approximation gap relative to the causal object, not circularity of the accuracy claim. Self-citations to related forensics work by overlapping co-authors are used only as comparison baselines, not as uniqueness theorems or load-bearing premises. No fitted parameter is renamed as a prediction of a quantity fixed by that fit; no uniqueness result is imported from the authors to forbid alternatives; no known empirical pattern is merely renamed. Against external benchmarks and baselines the derivation chain is self-contained for circularity purposes.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The paper is empirical systems work. Its claims rest on a counterfactual definition of decisive failure, the LLM-as-judge paradigm, public multi-agent failure benchmarks and their annotations, and standard PEFT training choices—not on new physical entities. Free parameters are mainly training/hyperparameter and aggregation design choices; invented entities are methodological constructs (AgentLocate stages and the decisive-step formalization).

free parameters (4)
  • LoRA rank / alpha / dropout / epochs / batch
    Rank 64, scaling 128, dropout 0.05, 3 epochs, effective batch 16 are chosen design hyperparameters that affect Judge adaptation quality; not derived from theory.
  • Number and prompt styles of Evaluators
    Default H=3 with base/concise/evidence prompts; ablations show 3 is a sweet spot, so the operating point is selected from data rather than fixed a priori.
  • Train/val/test split fractions (Who&When)
    50%/10%/40% split and Aegis 8933/600 split determine which failures supervise the Judge versus evaluate it.
  • Evaluator self-reported confidence weights c_h
    Aggregation A(i,t)=Σ c_h 1{(i_h,t_h)=(i,t)} treats model-reported confidences as calibrated weights without external calibration.
axioms (5)
  • domain assumption A single earliest step exists whose idealized one-step correction would reverse system failure (nonempty K(τ); t⋆=min K(τ)).
    Appendix B defines decisive failure via counterfactual R(τ,t); multi-cause or non-reversible failures may not fit this single-step model.
  • domain assumption LLM Judges/Evaluators can approximate the counterfactual decisive-error criterion by reading logs without executing R.
    Section 3.1 explicitly states the Judge does not execute R and instead reasons over the trajectory.
  • domain assumption Turn-based multi-agent formalization M=⟨N,S,{A_i},F,ρ⟩ with one active agent per step.
    Section 2.1 adopts this protocol following prior work; concurrent or continuous control settings are out of scope.
  • domain assumption Benchmark annotations of responsible agent/step (and Aegis error modes) are valid ground truth for evaluation and for Variant III.
    All accuracy claims depend on Who&When and Aegis-Bench labels being correct and decisive in the paper’s sense.
  • standard math Standard supervised LoRA fine-tuning improves alignment of Judge outputs with desired (agent, step) labels.
    Uses conventional PEFT training; no new learning theory claimed.
invented entities (2)
  • AgentLocate Judge–Evaluator refinement cycle no independent evidence
    purpose: Generate, verify, aggregate, and adapt failure attributions for multi-agent trajectories.
    Methodological system introduced by the paper; evaluated only within this work’s experiments.
  • Decisive failure step t⋆(τ) / responsible agent i⋆(τ) no independent evidence
    purpose: Provide a formal target for localization as earliest single-step counterfactual fix.
    Defined in Appendix B; operationalized via LLM approximation rather than executed counterfactuals in the main pipeline.

pith-pipeline@v1.1.0-grok45 · 29938 in / 3379 out tokens · 38243 ms · 2026-07-10T14:07:43.296897+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Figures

Figures reproduced from arXiv: 2607.07989 by Anjun Gao, Minghong Fang, Yueyang Quan, Yufei Xia, Zhuqing Liu.

Figure 1
Figure 1. Figure 1: Illustrative example of failure propagation in a multi-agent system. [PITH_FULL_IMAGE:figures/full_fig_p021_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of step localization error. [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Localization performance of different methods with different step tolerance [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of failure trajectories by token length. [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Top-4 most frequent failure-causing agents in the Who&When dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

82 extracted references · 82 canonical work pages · 19 internal anchors

  1. [1]

    ICML , year=

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems , author=. ICML , year=

    work page

  2. [2]

    USENIX Security Symposium , year=

    Patcher: Post-Hoc Patching of Backdoored Large Language Models , author=. USENIX Security Symposium , year=

    work page

  3. [3]

    NeurIPS , year=

    Why do multi-agent llm systems fail? , author=. NeurIPS , year=

    work page

  4. [4]

    arXiv preprint arXiv:2506.09443 , year=

    LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge , author=. arXiv preprint arXiv:2506.09443 , year=

    work page arXiv

  5. [5]

    ICLR , year=

    Lora: Low-rank adaptation of large language models , author=. ICLR , year=

    work page

  6. [6]

    arXiv e-prints , year=

    AEGIS: Automated Error Generation and Identification for Multi-Agent Systems , author=. arXiv e-prints , year=

    work page

  7. [9]

    IEEE Symposium on Security and Privacy , year=

    Who taught the lie? responsibility attribution for poisoned knowledge in retrieval-augmented generation , author=. IEEE Symposium on Security and Privacy , year=

    work page

  8. [10]

    The Web Conference 2025 , year=

    Traceback of poisoning attacks to retrieval-augmented generation , author=. The Web Conference 2025 , year=

    work page 2025

  9. [11]

    COLM , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. COLM , year=

    work page

  10. [12]

    NeurIPS , year=

    Camel: Communicative agents for "mind" exploration of large language model society , author=. NeurIPS , year=

    work page

  11. [13]

    ACL , year=

    Chatdev: Communicative agents for software development , author=. ACL , year=

    work page

  12. [15]

    ICML , year=

    Improving factuality and reasoning in language models through multiagent debate , author=. ICML , year=

    work page

  13. [16]

    ICML , year=

    Gptswarm: Language agents as optimizable graphs , author=. ICML , year=

    work page

  14. [19]

    ICML , year=

    Executable code actions elicit better llm agents , author=. ICML , year=

    work page

  15. [20]

    ICML Workshop on LLMs and Cognition , year=

    Sheetagent: A generalist agent for spreadsheet reasoning and manipulation via large language models , author=. ICML Workshop on LLMs and Cognition , year=

    work page

  16. [25]

    LLaMA: Open and Efficient Foundation Language Models

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [27]

    Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal=

    work page

  18. [28]

    Vicinagearth , year=

    A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges , author=. Vicinagearth , year=

    work page

  19. [29]

    ICLR , year=

    MetaGPT: Meta programming for a multi-agent collaborative framework , author=. ICLR , year=

    work page

  20. [32]

    Engineering Applications of Artificial Intelligence , year=

    Generative Agent-Based Modeling with Large Language Models for insider threat detection , author=. Engineering Applications of Artificial Intelligence , year=

    work page

  21. [33]

    NeurIPS , year=

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face , author=. NeurIPS , year=

    work page

  22. [35]

    ICLR , year=

    Gaia: a benchmark for general ai assistants , author=. ICLR , year=

    work page

  23. [36]

    NeurIPS , year=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. NeurIPS , year=

    work page

  24. [38]

    ACL , year=

    Can large language models be an alternative to human evaluations? , author=. ACL , year=

    work page

  25. [39]

    USENIX Security Symposium , year=

    Poison forensics: Traceback of data poisoning attacks in neural networks , author=. USENIX Security Symposium , year=

    work page

  26. [40]

    NDSS , year=

    Beagle: Forensics of deep learning backdoor attack for better defense , author=. NDSS , year=

    work page

  27. [41]

    CCS , year=

    Identifying a training-set attack's target using renormalized influence estimation , author=. CCS , year=

    work page

  28. [42]

    NeurIPS , year=

    Tracing Back the Malicious Clients in Poisoning Attacks to Federated Learning , author=. NeurIPS , year=

    work page

  29. [44]

    EMNLP , year=

    From generation to judgment: Opportunities and challenges of llm-as-a-judge , author=. EMNLP , year=

    work page

  30. [49]

    IEEE Symposium on Security and Privacy , year=

    Promptlocate: Localizing prompt injection attacks , author=. IEEE Symposium on Security and Privacy , year=

    work page

  31. [51]

    ISI , year=

    No-doubt: Attack attribution based on threat intelligence reports , author=. ISI , year=

    work page

  32. [52]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report . arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [53]

    Qwen-7B , 2025

    Alibaba Cloud . Qwen-7B , 2025. URL https://huggingface.co/Qwen/Qwen-7B

    work page 2025

  34. [54]

    Where did it all go wrong? a hierarchical look into multi-agent error attribution

    Adi Banerjee, Anirudh Nair, and Tarik Borogovac. Where did it all go wrong? a hierarchical look into multi-agent error attribution. arXiv preprint arXiv:2510.04886, 2025

    work page arXiv 2025

  35. [55]

    Emergent autonomous scientific research capabilities of large language models

    Daniil A Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research capabilities of large language models. arXiv preprint arXiv:2304.05332, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  36. [56]

    Why do multi-agent llm systems fail? In NeurIPS, 2025

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail? In NeurIPS, 2025

    work page 2025

  37. [57]

    Optimizing Model Selection for Compound AI Systems

    Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Matei Zaharia, James Zou, and Ion Stoica. Optimizing model selection for compound ai systems. arXiv preprint arXiv:2502.14815, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [58]

    Sheetagent: A generalist agent for spreadsheet reasoning and manipulation via large language models

    Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, and Jianye Hao. Sheetagent: A generalist agent for spreadsheet reasoning and manipulation via large language models. In ICML Workshop on LLMs and Cognition, 2024

    work page 2024

  39. [59]

    Beagle: Forensics of deep learning backdoor attack for better defense

    Siyuan Cheng, Guanhong Tao, Yingqi Liu, Shengwei An, Xiangzhe Xu, Shiwei Feng, Guangyu Shen, Kaiyuan Zhang, Qiuling Xu, Shiqing Ma, et al. Beagle: Forensics of deep learning backdoor attack for better defense. In NDSS, 2023

    work page 2023

  40. [60]

    Can large language models be an alternative to human evaluations? In ACL, 2023

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In ACL, 2023

    work page 2023

  41. [61]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In ICML, 2023

    work page 2023

  42. [62]

    Generative agent-based modeling with large language models for insider threat detection

    Antonino Ferraro, Gian Marco Orlando, and Diego Russo. Generative agent-based modeling with large language models for insider threat detection. In Engineering Applications of Artificial Intelligence, 2025

    work page 2025

  43. [63]

    Patcher: Post-hoc patching of backdoored large language models

    Anjun Gao, Yueyang Quan, Yufei Xia, Zhuqing Liu, and Minghong Fang. Patcher: Post-hoc patching of backdoored large language models. In USENIX Security Symposium, 2026

    work page 2026

  44. [64]

    FlowReasoner: Reinforcing Query-Level Meta-Agents

    Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. Flowreasoner: Reinforcing query-level meta-agents. arXiv preprint arXiv:2504.15257, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  45. [65]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [66]

    Identifying a training-set attack's target using renormalized influence estimation

    Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack's target using renormalized influence estimation. In CCS, 2022

    work page 2022

  47. [67]

    LLM Multi-Agent Systems: Challenges and Open Problems

    Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, and Zhaozhuo Xu. Llm multi-agent systems: Challenges and open problems. arXiv preprint arXiv:2402.03578, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [68]

    Metagpt: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In ICLR, 2024

    work page 2024

  49. [69]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    work page 2022

  50. [70]

    Automated Design of Agentic Systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  51. [71]

    Tracing back the malicious clients in poisoning attacks to federated learning

    Yuqi Jia, Minghong Fang, Hongbin Liu, Jinghuai Zhang, and Neil Zhenqiang Gong. Tracing back the malicious clients in poisoning attacks to federated learning. In NeurIPS, 2025

    work page 2025

  52. [72]

    Promptlocate: Localizing prompt injection attacks

    Yuqi Jia, Yupei Liu, Zedian Shao, Jinyuan Jia, and Neil Gong. Promptlocate: Localizing prompt injection attacks. In IEEE Symposium on Security and Privacy, 2026

    work page 2026

  53. [73]

    Large Language Models Are State-of-the-Art Evaluators of Translation Quality

    Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [74]

    Aegis: Automated error generation and identification for multi-agent systems

    Fanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang, Xiaofei Zhang, Ziang Chen, Zhaowei Zhang, Xiaoyuan Zhang, Song-Chun Zhu, and Xue Feng. Aegis: Automated error generation and identification for multi-agent systems. arXiv e-prints, 2025

    work page 2025

  55. [75]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In EMNLP, 2025

    work page 2025

  56. [76]

    Camel: Communicative agents for "mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In NeurIPS, 2023

    work page 2023

  57. [77]

    A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 2024

    work page 2024

  58. [78]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In ICLR, 2023

    work page 2023

  59. [79]

    Mistral-7B , 2023

    Mistral AI . Mistral-7B , 2023. URL https://www.mistral.ai/

    work page 2023

  60. [80]

    Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors

    Fan Nie, Lan Feng, Haotian Ye, Weixin Liang, Pan Lu, Huaxiu Yao, Alexandre Alahi, and James Zou. Weak-for-strong: Training weak meta-agent to harness strong executors. arXiv preprint arXiv:2504.04785, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  61. [81]

    No-doubt: Attack attribution based on threat intelligence reports

    Lior Perry, Bracha Shapira, and Rami Puzis. No-doubt: Attack attribution based on threat intelligence reports. In ISI, 2019

    work page 2019

  62. [82]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In ACL, 2024

    work page 2024

  63. [83]

    Utrace: Poisoning forensics for private collaborative learning

    Evan Rose, Hidde Lycklama, Harsh Chaudhari, Anwar Hithnawi, and Alina Oprea. Utrace: Poisoning forensics for private collaborative learning. arXiv preprint arXiv:2409.15126, 2024

    work page arXiv 2024

  64. [84]

    Poison forensics: Traceback of data poisoning attacks in neural networks

    Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng, and Ben Y Zhao. Poison forensics: Traceback of data poisoning attacks in neural networks. In USENIX Security Symposium, 2022

    work page 2022

  65. [85]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. In NeurIPS, 2023

    work page 2023

  66. [86]

    LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

    Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  67. [87]

    Multi-Agent Coordination across Diverse Applications: A Survey

    Lijun Sun, Yijun Yang, Qiqi Duan, Yuhui Shi, Chao Lyu, Yu-Cheng Chang, Chin-Teng Lin, and Yang Shen. Multi-agent coordination across diverse applications: A survey. arXiv preprint arXiv:2502.14743, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  68. [88]

    Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

    Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  69. [89]

    AutoDev: Automated AI-Driven Development

    Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. Autodev: Automated ai-driven development. arXiv preprint arXiv:2403.08299, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  70. [90]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In ICML, 2024

    work page 2024

  71. [91]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. In COLM, 2024

    work page 2024

  72. [92]

    MasRouter: Learning to Route LLMs for Multi-Agent Systems

    Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route llms for multi-agent systems. arXiv preprint arXiv:2502.11133, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  73. [93]

    TextGrad: Automatic "Differentiation" via Text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  74. [94]

    Traceback of poisoning attacks to retrieval-augmented generation

    Baolei Zhang, Haoran Xin, Minghong Fang, Zhuqing Liu, Biao Yi, Tong Li, and Zheli Liu. Traceback of poisoning attacks to retrieval-augmented generation. In The Web Conference 2025, 2025 a

    work page 2025

  75. [95]

    Who taught the lie? responsibility attribution for poisoned knowledge in retrieval-augmented generation

    Baolei Zhang, Haoran Xin, Yuxi Chen, Zhuqing Liu, Biao Yi, Tong Li, Lihai Nie, Zheli Liu, and Minghong Fang. Who taught the lie? responsibility attribution for poisoned knowledge in retrieval-augmented generation. In IEEE Symposium on Security and Privacy, 2026

    work page 2026

  76. [96]

    EvoFlow: Evolving Diverse Agentic Workflows On The Fly

    Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. Evoflow: Evolving diverse agentic workflows on the fly. arXiv preprint arXiv:2502.07373, 2025 b

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  77. [97]

    AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

    Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. Agentracer: Who is inducing failure in the llm agentic systems? arXiv preprint arXiv:2509.03312, 2025 c

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  78. [98]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. In ICML, 2025 d

    work page 2025

  79. [99]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023

    work page 2023

  80. [100]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

Showing first 80 references.