Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

This paper argues that reversing the direction of data-dependency edges in execution logs yields a performance causal graph that locates the root-cause agent and critical step of a multi-agent system failure, and that the resulting attribut

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A hierarchical causal attribution framework (performance causal inversion, Shapley values, CDC-MAS step discovery) localizes failure agents and steps in LLM multi-agent systems, reporting up to 36.2% step accuracy.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Well-motivated and genuinely new in combination, but Definition 1 as written reverses cause and effect, and the counterfactual machinery is unspecified—send to referees with expectations of major revision. the 5 major comments →

arxiv 2509.08682 v1 pith:YG7BM6DL submitted 2025-09-10 cs.AI

Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference

Guoqing Ma , Jia Zhu , Hanghui Guo , Weijie Shi , Jiawei Shen , Jingjiang Liu , Yidan Liang This is my paper
classification cs.AI
keywords multi-agent systemsfailure attributioncausal inferenceperformance causal inversionShapley valuecausal discoveryLLM agentscounterfactual reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-agent systems built on large language models fail often, and existing diagnostic tools—manual review, statistical correlation, LLM-as-Judge—mostly blame the last agent that produced a bad output. This paper is trying to establish that failure attribution should be posed as a causal-inference problem, and that the key move is to reverse the direction of data dependencies in execution logs: if step i's output feeds step j, the performance causal edge should point from j back to i, because the downstream symptom is caused by the upstream deficiency. On that reversed graph, Shapley values assign each agent a system-importance score, and a counterfactual "would the task succeed if this agent had behaved well?" filter selects the bottleneck agent; a four-stage causal discovery algorithm then ranks the critical step within that agent's trace. If the framework is right, causal diagnosis replaces correlation-based guessing, and the paper reports that the resulting attribution reports boost task success rates by 22.4 percentage points in a closed-loop repair experiment. This matters because current best methods locate the critical error step in under 20% of cases, so a principled path to root-cause localization would make large multi-agent deployments cheaper to debug.

Core claim

The paper's central claim is that in an LLM multi-agent system, the direction of information flow is the opposite of the direction of performance causality. Concretely, from a failed trajectory T={(a_i,s_i,t_i,c_i)}, it builds a data-dependency graph G_data with edges output(v_i)⊆input(v_j), then defines the performance causal graph by the inversion E_causal={(v_j,v_i):(v_i,v_j)∈E_data}. This reversed graph is the structural basis for all attribution. On top of it, the SBSLocator module estimates each agent's Shapley value, integrates it with a counterfactual outcome Y_cf_j in the bottleneck score BS_j=φ_j×(Y_cf_j−Y_original)×I, and identifies the responsible agent; the CPIdentifier module c

What carries the argument

The load-bearing object is the performance causal graph G_causal=(V,E_causal), obtained by reversing every edge of the execution-log data-dependency graph. It is what turns a correlation-based trace into a causal structure: edges now point from the presumed performance cause to its downstream effect, so an upstream agent whose bad output corrupts later steps becomes reachable as a cause rather than being masked by the final symptom. Two algorithms ride on this graph: SBSLocator (agent-level: Shapley values plus a counterfactual bottleneck score) and CPIdentifier (step-level: CDC-MAS, a four-stage discovery pipeline that conditions on a learned context vector C_MAS to absorb non-stationarity

Load-bearing premise

The framework assumes that reversing every data-dependency edge in the execution log produces a graph in which edges point from the true performance cause to its effect, and that the counterfactual simulator can faithfully predict what would have happened if an agent or step had behaved correctly.

What would settle it

Inject a known single-point failure into a controlled MAS trace—for example, corrupt only the final summarizer's prompt while all upstream agents are perfect. If the method's stated inversion E_causal={(v_j,v_i)|(v_i,v_j)∈E_data} attributes blame to an upstream agent, or if the injected step is not ranked first, then the inversion direction is not causal. Running such injected-failure traces across many random single-point faults would settle whether the reversal rule is correct.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any MAS execution log can be turned into a blame graph simply by reversing observed data dependencies, with no extra instrumentation.
  • The two-level design—locate the responsible agent first, then the critical step inside its trace—produces a human-readable diagnostic report that can be fed directly to an LLM repair agent.
  • In closed-loop experiments, targeted repairs based on the causal report outperform random repairs by a wide margin (37.8% vs 18.3% success), indicating that the diagnostic information, not the act of editing, drives the improvement.
  • CDC-MAS's context vector provides a template for causal discovery in non-stationary interaction data, the usual case for live agent deployments rather than curated benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The inversion principle, if it generalizes, would apply to any pipeline where downstream outputs depend on upstream ones—software build traces, data-pipeline failures, even human team post-mortems—by reversing dependency edges to define blame propagation.
  • The counterfactual outcome Y_cf_j is the least specified component of the framework; making explicit how "agent j performs normally" is simulated would be the decisive reproducibility test and the natural next paper.
  • The 22.4% success-rate gain is measured on 50 selected trajectories; a larger randomized sample of failures would test whether the optimization loop's gains persist outside the hand-picked set.
  • If the causal rankings are stable, they could be used as continuous reward signals for self-healing agents, creating a closed loop where attribution is updated after every repair attempt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a two-stage causal-inference framework for automatic failure attribution in LLM-based multi-agent systems. The SBSLocator module constructs a 'performance causal graph' by reversing data-dependency edges, applies Shapley values and a counterfactual bottleneck score to identify the responsible agent, and the CPIdentifier module extracts multi-dimensional features and runs a proposed CDC-MAS algorithm to locate the critical step. The framework is evaluated on the Who&When and TRAIL benchmarks, with reported step-level accuracy up to 36.2% and a 22.4% average improvement in task success rate in a closed-loop repair experiment.

Significance. The problem is important, and the paper's high-level idea of separating correlational from causal attribution is sensible. The paper also makes a useful attempt at actionable validation via closed-loop optimization and compares against several baselines. However, the central formal construction is internally inconsistent, the counterfactual simulator is not specified, the CDC-MAS algorithm is not reproducible as written, and the empirical claims overstate the results. In its current form the contribution is not established; with substantial revision the core idea could be salvageable.

major comments (5)
  1. [SBSLocator, Definition 1 (Performance Causal Inversion)] Definition 1 defines E_causal = {(v_j, v_i) | (v_i, v_j) in E_data}. In the paper's running example, upstream A supplies data to downstream B, so (A,B) in E_data; the transformation yields (B,A) in E_causal, i.e. B's performance is the cause of A's performance. This contradicts the immediately preceding prose, which states that A's performance state is the root cause of B's performance manifestation. Because Eq. (1) uses parent sets PA_j from this graph and Eq. (2) inherits the same orientation, the entire agent-level attribution is built on a graph whose edges point from effect back to cause. This must be corrected and the relationship between data-flow edges and causal edges restated; as written, the method would blame downstream consumers rather than upstream root causes.
  2. [Definition 3 / Eq. (2) and Eq. (6)] The quantities Y_cf_j and E[Y|do(X_k = x_optimal_k)] are used to compute BS_j and Delta_k, yet no counterfactual simulator is described: there is no model of task outcome, no training objective, no identification assumptions, and no validation. The intervention value x_optimal_k is also undefined. Since the bottleneck score and final step ranking depend directly on these quantities, the attribution results are uninterpretable unless this component is specified and calibrated. The heading 'Robustness Guarantees' promises guarantees that the text does not deliver.
  3. [CPIdentifier / CDC-MAS (Eq. (5), Theorem 1)] CDC-MAS is described only at a high level. Stage II's conditional independence tests, Stage III's orientation rule, the construction of C_MAS as a proxy for confounding, LocalEffect(e) as a local linear regression coefficient, and Context Weight(e, C_MAS) are all undefined; Eq. (5) is therefore not computable from the text. Theorem 1 claims asymptotic consistency under causal Markov, faithfulness, and contextual sufficiency, but its proof and the algorithm pseudocode are relegated to Appendix B, which is absent from the preprint. This makes the step-level contribution non-reproducible.
  4. [Table 1 / Main Results] The text states that the framework 'consistently outperforms all baselines across all metrics and subsets.' Table 1 shows the opposite on the Algo-Generated subset: All-at-Once achieves 54.33% agent accuracy while Ours achieves 48.5%. The Hand-Crafted agent-level margin (56.8 vs 55.17) is small. The results should be reported accurately, with confidence intervals and p-values (claimed in the setup but never reported). In addition, the baseline Step-by-Step step accuracy of 25.51% on Algo is higher than the 17.1% cited for best existing methods; the numbers need reconciliation.
  5. [Closed-Loop Optimization / Evaluation] The causal validity of the framework is not independently tested. The step-level ranking in Eq. (5) uses regression coefficients and a context vector fit on the same trajectories being diagnosed, and the counterfactual outcomes in Eq. (6) are produced by the same framework's unspecified model; evaluation on the same data can reflect fit rather than causal effect. The closed-loop repair experiment uses 50 trajectories and a GPT-4o repair agent, so the 22.4% success-rate gain may partly reflect the repair agent's general ability rather than correct attribution. Please provide out-of-sample evaluation, calibration of the counterfactual model, and a matched intervention control (e.g., oracle or shuffled diagnoses).
minor comments (6)
  1. [Appendices] Appendix A (theoretical boundaries of PCI) and Appendix B (CDC-MAS pseudocode and proof of Theorem 1) are referenced but not present in the preprint. Please include them or integrate the content into the main text.
  2. [Figures] Figure 2 contains unreadable LaTeX-like fragments (e.g., \hat{\phi}, \frac expressions). Figure 1 has a typo 'Shaply' instead of 'Shapley', and the axis label in Figure 3 is misspelled as 'Methonds'.
  3. [Eq. (2)] The indicator function I[...] in the bottleneck score is left unspecified; the success threshold theta_success is introduced in prose but does not appear in the displayed formula.
  4. [Statistical reporting] The setup claims t-tests with p < 0.05, but no p-values, confidence intervals, or test details are reported anywhere. The closed-loop experiment appears to be a single run on 50 trajectories.
  5. [Novelty claims] The abstract and contributions describe this as the 'first' causal-inference failure attribution framework for MAS. Given the cited counterfactual-importance work (Chen et al. 2025) and existing attribution benchmarks, the first/claim needs qualification or evidence of a systematic literature gap.
  6. [Abstract/Introduction consistency] The abstract says state-of-the-art methods achieve less than 15% accuracy, the introduction says 17.1%, and Table 1's Step-by-Step baseline reaches 25.51% on Algo-Generated. These numbers should be reconciled.

Circularity Check

2 steps flagged

The central 'performance causal graph' is defined by a data-flow reversal that points edges opposite to the paper's own running example, and the counterfactual 'validation' scores are produced by regressions fit on the same trajectory; the attribution loop is partly closed by construction.

specific steps
  1. self definitional [Methodology, SBSLocator, 'Theoretical Foundation of Performance Causal Inversion', Definition 1]
    "To construct an accurate performance causal graph, the data dependencies observed from the execution logs must be reversed in direction. ... E_causal = {(v_j, v_i) | (v_i, v_j) ∈ E_data}. This transformation ensures that each edge in the causal graph points from the performance cause to the performance effect."

    The running example says upstream agent A's performance deficiency causes downstream B's failure, so the causal edge should be A→B. But E_data contains (A,B) when output(A)⊆input(B), and the definition sets E_causal to contain (B,A), making B the cause of A. The 'correct' causal direction is therefore not inferred; it is fixed by a definitional reversal that contradicts the prose. Eq. 1's parent sets, the Shapley values, and BS_j in Eq. 2 all inherit this direction, so the root-cause attribution is an artifact of the definition.

  2. fitted input called prediction [Methodology, CPIdentifier, 'Counterfactual Validation and Robustness Guarantees', Eq. (6) and CDC-MAS Stage IV]
    "For each candidate step s_k, we simulate an 'optimal' intervention and calculate its expected improvement on the final outcome ∆k: ∆k = E[Y|do(X_k = x_optimal_k)] − E[Y|X_k = x_observed_k]. ... LocalEffect(e) is estimated by the coefficient in a local linear regression model."

    The counterfactual improvement ∆k is not measured on independent data; it is the difference of two expectations from a model whose LocalEffect is a local linear regression coefficient fit on the same failed trajectory being diagnosed. FinalScore_k then combines this fitted ∆k with ACE_k and a bootstrap confidence. Presenting this as 'counterfactual validation' makes the model's own fitted outputs confirm the model's ranking. The closed-loop experiment does re-execute real repairs, so the loop is not fully closed, but the internal validation step is circular by construction.

full rationale

The paper's external evaluations (Who&When/TRAIL with human-annotated ground truth, and the real re-execution in the closed-loop experiment) prevent a fully circular verdict. However, the central PCI principle is declared rather than derived: Definition 1 defines G_causal as the reverse of G_data, and the accompanying example shows that this reversal points edges from the downstream effect to the upstream cause, contradicting the intended 'cause-to-effect' claim. All downstream Shapley and bottleneck computations inherit this definitional direction. Separately, the counterfactual scores that the paper calls validation are computed from local linear regressions and a learned context vector fit on the same trajectory, so the internal validity check reduces to the fitted model's own outputs. The missing Appendices A and B (theoretical boundaries and proof of Theorem 1) leave these steps unverified, but missing proofs and notational inconsistency are correctness risks rather than additional circularity. Overall, the central claim is partially forced by definition and by the fitted counterfactual model, giving a score of 6.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The framework introduces no new physical or ontological entities. It introduces a learned context vector C_MAS and a 'performance causal graph,' but these are model constructs, not entities with external falsifiable handles. The central claim rests on the listed free parameters, the causal Markov/faithfulness/acylcicity assumptions, and the unverified counterfactual simulation model.

free parameters (5)
  • alpha (Eq. 1)
    Amplification factor controlling how much the Shapley value phi_hat_j affects agent performance X_j. Value not specified in text.
  • omega_1, omega_2, omega_3 (Eq. 7)
    Weights fusing ACE, counterfactual improvement, and confidence into the final step score. Values not reported.
  • theta_success (Eq. 2)
    Predefined success threshold used in the indicator function for bottleneck score. Not defined.
  • x_optimal_k (Eq. 6)
    Definition of the 'optimal' intervention for counterfactual step simulation. Not specified; it is determined by the framework's own model.
  • Transformer and regression hyperparameters
    Hyperparameters for the context encoder C_MAS and the local linear regressions for LocalEffect are not listed.
axioms (4)
  • domain assumption Causal Markov property, faithfulness, and contextual sufficiency
    Assumed for Theorem 1's consistency guarantee of CDC-MAS, stated in the CPIdentifier section without proof.
  • ad hoc to paper Acyclicity of the performance causal graph
    Admitted in the SBSLocator section: 'its application relies on key assumptions such as the acyclicity of the performance graph, and has limitations in scenarios with complex feedback loops or collaborative failures.'
  • ad hoc to paper Performance Causal Inversion principle
    The claim that reversing data-flow edges yields the performance causal graph is a modeling assumption introduced by the authors, not a derived theorem.
  • domain assumption Shapley values quantify agent importance
    The framework treats agents as players in a cooperative game; the characteristic function defining the game is not specified, so the Shapley computation is an assumption of applicability.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference." pith.science (2026). https://pith.science/paper/YG7BM6DL

@misc{pith2026250908682,
  author       = {Pith},
  title        = {Pith review of: Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YG7BM6DL}},
  note         = {Machine review of arXiv:2509.08682}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multi-agent systems (MAS) are critical for automating complex tasks, yet their practical deployment is severely hampered by the challenge of failure attribution. Current diagnostic tools, which rely on statistical correlations, are fundamentally inadequate; on challenging benchmarks like Who\&When, state-of-the-art methods achieve less than 15\% accuracy in locating the root-cause step of a failure. To address this critical gap, we introduce the first failure attribution framework for MAS grounded in multi-granularity causal inference. Our approach makes two key technical contributions: (1) a performance causal inversion principle, which correctly models performance dependencies by reversing the data flow in execution logs, combined with Shapley values to accurately assign agent-level blame; (2) a novel causal discovery algorithm, CDC-MAS, that robustly identifies critical failure steps by tackling the non-stationary nature of MAS interaction data. The framework's attribution results directly fuel an automated optimization loop, generating targeted suggestions whose efficacy is validated via counterfactual simulations. Evaluations on the Who\&When and TRAIL benchmarks demonstrate a significant leap in performance. Our method achieves up to 36.2\% step-level accuracy. Crucially, the generated optimizations boost overall task success rates by an average of 22.4\%. This work provides a principled and effective solution for debugging complex agent interactions, paving the way for more reliable and interpretable multi-agent systems.

Figures

Figures reproduced from arXiv: 2509.08682 by Guoqing Ma, Hanghui Guo, Jiawei Shen, Jia Zhu, Jingjiang Liu, Weijie Shi, YiDan Liang.

Figure 1
Figure 1. Figure 1: Challenges in Fault Attribution for Multi-Agent [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Failure attribution process based on performance causality inversion and hierarchical causal discovery. This figure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Step-level attribution accuracy on the TRAIL [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems

    cs.AI 2026-06 unverdicted novelty 6.0

    StepFinder turns execution logs into temporal semantic sequences via LLMs then uses temporal modeling plus attention to attribute failures to specific steps more accurately and 79% faster than direct LLM methods on th...

  2. Agent System Operations: Categorization, Challenges, and Future Directions

    cs.MA 2026-06 unverdicted novelty 3.0

    This survey categorizes anomalies in agent systems into intra-agent and inter-agent types and introduces the AgentOps framework with four operational stages.

Reference graph

Works this paper leans on

19 extracted references · 15 linked inside Pith · cited by 2 Pith papers

  1. [2]

    Chen, J.; Wang, Y .; Wang, J.; Xie, X.; Hu, J.; Wang, Q.; and Xu, F

    Chateval: Towards better llm- based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201. Chen, J.; Wang, Y .; Wang, J.; Xie, X.; Hu, J.; Wang, Q.; and Xu, F

    Pith/arXiv arXiv

  2. [4]

    InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–15

    Interactive debugging and steering of multi-agent ai systems. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–15. Fu, J.; Ng, S.-K.; Jiang, Z.; and Liu, P

    2025

  3. [5]

    Gao, M.; Li, Y .; Liu, B.; Yu, Y .; Wang, P.; Lin, C.-Y .; and Lai, F

    Gptscore: Evaluate as you desire.arXiv preprint arXiv:2302.04166. Gao, M.; Li, Y .; Liu, B.; Yu, Y .; Wang, P.; Lin, C.-Y .; and Lai, F

    Pith/arXiv arXiv

  4. [6]

    Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y .; Ma, S.; Liu, H.; et al

    Single-agent or Multi-agent Systems? Why Not Both?arXiv preprint arXiv:2505.18286. Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y .; Ma, S.; Liu, H.; et al

    Pith/arXiv arXiv

  5. [8]

    Jiang, H.; Chen, Y .; Cao, Y .; Lee, H.-y.; and Tan, R

    Language model preference evaluation with multiple weak evaluators.arXiv preprint arXiv:2410.12869. Jiang, H.; Chen, Y .; Cao, Y .; Lee, H.-y.; and Tan, R. T

    arXiv

  6. [9]

    Jiang, S.; Huang, Z.; Luo, X.; and Sun, Y

    CodeJudgeBench: Benchmarking LLM-as-a-Judge for Cod- ing Tasks.arXiv preprint arXiv:2507.10535. Jiang, S.; Huang, Z.; Luo, X.; and Sun, Y

    Pith/arXiv arXiv

  7. [11]

    arXiv preprint arXiv:2308.00436

    Selfcheck: Us- ing llms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00436. Pan, M. Z.; Cemri, M.; Agrawal, L. A.; Yang, S.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Ramchan- dran, K.; Klein, D.; et al

    Pith/arXiv arXiv

  8. [12]

    Why do multiagent systems fail? InICLR 2025 Workshop on Building Trust in Language Models and Applications. Park, J. S.; O’Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S

    2025

  9. [13]

    InProceedings of the 36th annual acm symposium on user interface software and tech- nology, 1–22

    Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface software and tech- nology, 1–22. Pearl, J. 2009.Causality. Cambridge university press. Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y .; Li, J.; Yang, C.; Chen, W.; Su, Y .; Cong, X.; et al

    2009

  10. [14]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S

    Chatdev: Com- municative agents for software development.arXiv preprint arXiv:2307.07924. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S

    Pith/arXiv arXiv

  11. [15]

    arXiv preprint arXiv:2406.12624

    Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624. Van Schaik, T. A.; and Pugh, B

    Pith/arXiv arXiv

  12. [17]

    Witmer, J

    A Survey on Large Language Model based Autonomous Agents.arXiv preprint arXiv:2308.11432. Witmer, J

    Pith/arXiv arXiv

  13. [18]

    Zhang, S.; Dong, Y .; Zhang, J.; Kautz, J.; Catanzaro, B.; Tao, A.; Wu, Q.; Yu, Z.; and Liu, G

    Ecoassistant: Using llm assistant more affordably and accurately.arXiv preprint arXiv:2310.03046. Zhang, S.; Dong, Y .; Zhang, J.; Kautz, J.; Catanzaro, B.; Tao, A.; Wu, Q.; Yu, Z.; and Liu, G. 2025a. Nemotron-Research- Tool-N1: Exploring Tool-Using Language Models with Re- inforced Reasoning.arXiv preprint arXiv:2505.00024. Zhang, S.; Yin, M.; Zhang, J.;...

    Pith/arXiv arXiv

  14. [19]

    arXiv preprint arXiv:2410.10934

    Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934

    Pith/arXiv arXiv

  15. [2022]

    InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, 12334– 12341

    Model-based diagnosis of multi-agent systems: A survey. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, 12334– 12341. Li, G.; Hammoud, H.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023a. Camel: Communicative agents for mind exploration of large language model society.Advances in Neural Information Processing Systems, 36: 51991...

    Pith/arXiv arXiv

  16. [2023]

    Chan, C.-M.; Chen, W.; Su, Y .; Yu, J.; Xue, W.; Zhang, S.; Fu, J.; and Liu, Z

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774. Chan, C.-M.; Chen, W.; Su, Y .; Yu, J.; Xue, W.; Zhang, S.; Fu, J.; and Liu, Z

    Pith/arXiv arXiv

  17. [2024]

    Herrera, M.; P´erez-Hern´andez, M.; Kumar Parlikad, A.; and Izquierdo, J

    A survey on llm-as-a- judge.arXiv preprint arXiv:2411.15594. Herrera, M.; P´erez-Hern´andez, M.; Kumar Parlikad, A.; and Izquierdo, J

    Pith/arXiv arXiv

  18. [2025]

    Epperson, W.; Bansal, G.; Dibia, V

    TRAIL: Trace Rea- soning and Agentic Issue Localization.arXiv preprint arXiv:2505.08638. Epperson, W.; Bansal, G.; Dibia, V . C.; Fourney, A.; Gerrits, J.; Zhu, E.; and Amershi, S

    Pith/arXiv arXiv

  19. [2836]

    Wang, F.; Zhang, Z.; Zhang, X.; Wu, Z.; Mo, T.; Lu, Q.; Wang, W.; Li, R.; Xu, J.; Tang, X.; et al. 2024a. A compre- hensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness.arXiv preprint arXiv:2411.03350. Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H...

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.