Pith. sign in

REVIEW 4 major objections 4 minor 72 references

CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For embodied multi-agent failures, dependency distance—not transcript position—should drive blame attribution.

desk verdict Solid new benchmark and credible external results, but the dependency-graph ablation is confounded by a global-LLM prompt that contradicts the claimed deterministic pipeline. read the letter →

arxiv 2608.01805 v1 pith:CHD5N5XK submitted 2026-08-03 cs.AI

Wei Wang , Shuanghe Liu , Zhu Zhuo , Jiaqi Zhong , Xiaozhao Zhao , Xiaojie Zuo , Jie Su This is my paper · ORCID
classification cs.AI
keywords LLMmulti-agentsystemserrorattributiondependencygraphhierarchicalcontextembodiedsafety-criticalagentsISO26262ASILCockpitBenchCorrectnessCollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM multi-agent systems can look accurate while failing badly at the process level, a pattern the paper calls Correctness Collapse, and in a car cockpit a lexically correct utterance can trigger a dangerous physical action. The paper's central claim is that finding which agent and which step caused a failure should be driven by dependency structure rather than by how close steps appear in the transcript. It introduces CockpitHAT, which lifts each trace into an interaction DAG using reference, plan, tool, and control signals, slices per-step context by dependency distance, attaches embodied-channel evidence, and combines analyst votes with a safety-weighted consensus. On the public Who&When benchmark, the paper reports 77.9 and 86.5 percent agent-level accuracy and 37.8 and 46.0 percent step-exact accuracy, gains over the text-only ECHO baseline of up to 17.6 and 16.7 points. It also releases CockpitBench, 212 failure traces labeled with ISO 26262 ASIL severity and three-channel embodied evidence.

What carries the argument

The load-bearing object is the interaction DAG with its dependency-distance slicing. Each trace step becomes a node, and an edge vi→vj is added when step j depends on step i through one of four signals: an explicit reference to an entity or tool result, execution or refinement of a proposed plan, use of a tool output field, or dispatch/control invocation. Dependency distance is the shortest undirected path between nodes in that graph, and each focal step is presented to the analysts as four layers (immediate, local, distant, global) determined by that distance rather than by transcript position. The second mechanism is the Embodied Channel Adapter, a lightweight per-trace scan that tags steps as interaction-risk, environmental misgrounding, safety-violation, or cross-domain and promotes them one layer (safety violations into the immediate layer) with matching channel fields attached. The third is the safety-uplifted consensus: a mandatory Safety Analyst plus three persona-sampled analysts, a maximum-severity prior over ASIL levels, and an exponential uplift on the safety analyst's vote at high severity, with a disagreement-resolution fallback when margins are thin. These three mechanisms form the argument that error propagation travels along dependency edges, so context, evidence, and voting all need to be dependency- and risk-aware.

What would settle it

Take a set of traces whose true error depends on an implicit shared fact (for example, two agents both act on an unstated map constraint). Run CockpitHAT on those traces and check step-exact accuracy. The four-signal graph cannot draw an edge through shared background knowledge, so if the method still localizes those failures at chance level, the dependency-graph premise is falsified; if it localizes them anyway, the LLM entity linker or another component is carrying the weight.

Watch

Extended reading notes

Core claim

CockpitHAT's central discovery is that dependency-aware context outperforms position-aware context for locating the responsible agent and step. Instead of taking the k nearest transcript positions around a focal step, the method builds a directed acyclic graph whose edges mark reference, plan, tool, and control dependencies, then defines dependency distance as the shortest undirected path in that graph and uses it to build four context layers: verbatim immediate neighbors, compressed local, summarized distant, and milestone global. An embodied adapter promotes flagged steps upward, forcing safety violations into the immediate layer regardless of distance. A four-analyst panel then reaches a two-stage consensus in which severity is chosen by the maximum ASIL level among confident analysts and the safety analyst's vote is exponentially uplifted at high severity. On CockpitBench the method reaches 78.3 percent agent-level and 38.2 percent step-exact accuracy, and it attributes high-severity failures (ASIL C/D) more accurately than low-severity ones. The paper's case is that these results establish dependency-aware, multi-channel, risk-calibrated attribution as the right paradigm for diagnosing failures in embodied multi-agent systems.

Load-bearing premise

The load-bearing premise is that the interaction DAG built from the four explicit signals—reference, plan, tool, control—plus entity linking captures every dependency that matters; any causal link that is implicit or based on shared background knowledge will be missing from the graph and can push the true error step into a compressed layer.

Editorial extensions

If this is right

  • Attribution pipelines for multi-agent systems should replace positional context windows with dependency-distance layers, because error propagation follows dependency edges rather than transcript adjacency.
  • Embodied failure diagnosis needs multi-channel evidence: dialogue alone cannot see a lexically correct command that triggers a hazardous vehicle-state change.
  • Safety-severity labels should be part of attribution benchmarks, since methods can be accurate overall while degrading on the rare high-severity cases that matter most.
  • A safety-uplifted consensus prevents high-risk failures from being outvoted by generic analyst signals, at a modest token cost (68,287 tokens per trace on the Hand-Crafted split).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension, replacing the four hand-specified edge signals with learned or provenance-based dependency labels would directly test whether the dependency graph, rather than the voting stage, is the performance ceiling; better edge recall should raise step-exact accuracy.
  • As an extension, the same dependency-distance slicing could transfer to code-level or hardware-in-the-loop multi-agent systems where a static call graph already exists, turning attribution into a graph-metrics problem.
  • As an extension, the fixed automotive safety-uplift parameters could be made adaptive to a domain's ASIL distribution; the paper's own limitation note indicates this is an open knob.
  • As an extension, because CockpitBench has only 26 ASIL-D traces, the reported high-severity step accuracy would need a larger severity-stratified sample before the risk-calibrated advantage is treated as settled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CockpitHAT, a hierarchical attribution framework for failure diagnosis in embodied multi-agent LLM systems. It constructs a dependency DAG from traces via four operational signals (reference, plan, tool, control), slices each focal step's context by dependency distance rather than positional offset, adds an embodied channel adapter for cockpit signals, and combines a safety-uplifted, confidence-weighted analyst consensus that predicts ISO 26262 ASIL severity. The authors also introduce CockpitBench, 212 annotated traces with ASIL labels and three-expert consensus. On the public Who&When benchmark, CockpitHAT reports agent-level/step-exact accuracies of 77.9%/37.8% on Hand-Crafted and 86.5%/46.0% on Algorithm-Generated, outperforming the ECHO baseline by up to 17.6/16.7 points. On CockpitBench it reports 78.3% agent-level and 38.2% step-exact accuracy. The paper claims this establishes dependency-aware, multi-channel, risk-calibrated attribution as an effective paradigm for reliable failure diagnosis in embodied multi-agent systems.

Significance. The problem is real and timely: Correctness Collapse in safety-critical embodied LLM agents is an important failure mode, and the paper's design of dependency-distance slicing, multi-channel evidence, and ASIL-aware consensus is a plausible direction. CockpitBench, with its ISO 26262 ASIL labels, three-expert consensus, and embodied channel annotations, could be a valuable community resource. The Who&When comparison against ECHO is externally grounded, and the paper reports statistical significance and token costs. However, the central ablation I2→I3 is confounded by the mismatch between the described deterministic graph construction and the appendix's global LLM prompt, and CockpitBench lacks external baseline numbers. The significance of the empirical claims cannot be fully assessed until these issues are resolved; the framework's potential is clear, but the current evidence does not yet establish the mechanism claimed.

major comments (4)
  1. [§3.1, §4.1, Appendix A.3 Prompt 1, Table 2] The paper states in §3.1 that edge induction 'reduces to local pattern matching plus entity linking' and in §4.1 that graph construction 'uses deterministic pattern matching; only the entity-linking sub-step invokes a lightweight LLM call.' Appendix A.3, Prompt 1, however, instructs the model to 'Analyze the full trace globally before making local edge decisions,' to 'Consider both linguistic and embodied evidence,' and to output the entire edge set with justifications. These descriptions are irreconcilable. If Prompt 1 is the implementation behind Table 2, then the I2→I3 ablation is confounded: I3 adds a global LLM graph-reasoning pass that I2 lacks, so the reported jumps (+13.7 HC, +8.6 AG, +25.2 CockpitBench agent-level) could stem from this extra reasoning rather than from dependency-distance slicing. If Prompt 1 is not the experimental code path, the paper must state which one is and explain why Appendix A.3 presents Prompt 1 as the method. Either way, the central claim that dependency-aware slicing is the operative mechanism is not established. Please provide a controlled ablation in which I3 uses only the deterministic local pattern matcher, or in which I2 is augmented with an equivalent global LLM pass, and clearly state the code path that produced Table 2.
  2. [§4.3, Table 3, Appendix A.1.3] CockpitBench results are reported only for CockpitHAT in Table 3; no external baseline numbers appear in the main text. Appendix A.1.3 states that 'the state-of-the-art ECHO model cannot achieve saturated performance' but gives no figures, and Table 2's CockpitBench column compares only ablations of CockpitHAT itself. Because CockpitBench is newly introduced by the authors, the absence of independently comparable baselines leaves the embodied-channel contribution unvalidated: the reader cannot determine whether 78.3% agent-level / 38.2% step-exact represents an advance over existing attribution methods on embodied traces. Please report at least ECHO, and ideally the other Who&When baselines, on CockpitBench under the same evaluation protocol.
  3. [§4.1, Table 2, §3.4] The additive ablation chain I1→I6 is central to the paper, but the operational definitions of the intermediate variants are not given. In particular, the paper never states what 'Objective Analysis (I4)' adds over I3, nor what 'Decoupled Attribution (I5)' changes relative to I4, beyond the prose labels in Section 3.4. Without a precise mapping from each I_k to a concrete pipeline configuration, the ablation is not reproducible, and the attribution of gains to specific components (e.g., 'Decoupled Attribution ... contributes the table's largest single AG agent-level jump, +17.6 points') cannot be verified. Please specify the exact configuration of each ablation variant, including which modules are enabled or disabled.
  4. [§4.1, Algorithm 1, Eqs. (3)-(4)] Several hyperparameters required to reproduce the method are not reported with values. The disagreement threshold δ_cons(η*) in §3.4 is described only as 'monotone in severity'; no values or calibration procedure are given. The look-back cap L used in Algorithm 1 is not specified anywhere in the paper. Also, the safety-uplift parameters λ=0.5, β=0.4, κ=1.0 are acknowledged as hand-calibrated for the cockpit domain in the Limitations, which is acceptable if clearly flagged, but the paper applies this uplift on Who&When, where no ASIL ground truth exists, without discussing how the predicted severities are grounded. Please provide δ_cons and L, state the calibration source for λ/β/κ, and comment on the validity of applying the cockpit-calibrated uplift to non-cockpit traces.
minor comments (4)
  1. [§3.3] There is a stray 'onG, , causing' typo in the sentence about the slicing rule compressing seeded steps.
  2. [Abstract and §1] The phrase 'surpassing the text-only SOTA ECHO by up to 17.6 / 16.7 points' is ambiguous; please specify that the first number is agent-level and the second is step-level accuracy.
  3. [Table 6] The P-values in the backbone ablation table lack a footnote describing the statistical test; please state whether these are the same chi-squared tests as in Table 1 or a different procedure.
  4. [Ethics Statement] The statement 'permissive open-source license' is vague; specifying the exact license (e.g., Apache 2.0) would help readers assess reuse conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Who&When comparison is external, the dependency graph is derived from trace content rather than from attribution labels, and no equation reduces to a fitted parameter by construction.

full rationale

CockpitHAT's headline claims are evaluated on the public Who&When benchmark against ECHO, an external baseline, so the reported gains cannot be an artifact of redefining the method's own inputs. The dependency graph (Section 3.1) is constructed from trace content via four operational signals (reference, plan, tool, control) plus entity linking; this is an input transformation, not a label-informed fit. The slicing rule and the safety-uplift formula (Eq. 4) are deterministic functions of the graph and of analyst-reported severity, and neither is fitted to the target labels in a way that forces the reported accuracies. The paper's own Limitations disclose that the safety-uplift hyperparameters are domain-calibrated and that the dependency graph misses implicit dependencies; these are honest caveats, not hidden circularities. One inconsistency is worth noting: Section 3.1 and Section 4.1 describe graph construction as deterministic pattern matching, while Appendix A.3 Prompt 1 instructs an LLM to analyze the full trace globally before making edge decisions. This threatens the clean interpretation of the I2-to-I3 ablation as isolating dependency-distance slicing, but it is an experimental confound rather than a circular reduction: the graph still does not contain the ground-truth agent/step labels, and the external Who&When result does not depend on this internal ablation. No equation in the paper is equivalent to another by construction, and no prediction is statistically forced by a fitted input, so the paper is not circular in the sense defined here.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The framework rests on hand-set weighting parameters (lambda, beta, kappa, delta_conf, plus unspecified delta_cons and look-back cap), on the sufficiency of four edge signals for reconstructing dependencies, and on the reliability of LLM-based entity linking and salience scanning. No new physical entities are introduced; the only new artifacts are the benchmark and label taxonomy. The central external benchmark (Who&When) is independent, but the domain-specific calibration of the uplift parameters is in-sample.

free parameters (7)
  • Safety-uplift lambda = 0.5
    Eq. (4) scales the safety analyst's weight; hand-set, and the Limitations say the uplift hyperparameters are calibrated specifically for automotive cockpits.
  • Safety-uplift beta = 0.4
    Exponential growth rate in Eq. (4), hand-set; no derivation is given for the super-linear ASIL severity model.
  • Safety-uplift kappa = 1.0
    Cap on the uplift in Eq. (4), hand-set to prevent single-analyst tyranny.
  • Confidence threshold delta_conf = 0.3
    Minimum confidence for including an analyst's ASIL vote in the maximum-severity prior (Eq. 3), hand-set.
  • Disagreement threshold delta_cons(eta*) = not specified
    ASIL-calibrated threshold that triggers Disagreement Resolution (Section 3.4); the paper never gives the function or numeric values, so it is an unquantified tuning parameter.
  • Edge look-back cap L = not specified
    Algorithm 1 line 6 restricts edge search to the previous L steps for O(n) practical complexity, but L is never stated in the paper.
  • ASIL severity value mapping v(eta) = implicitly D=4, C=3, B=2, A=1
    Used in Eq. (4) for the exponential uplift; the mapping is never explicitly defined in the text.
assumptions (4)
  • domain assumption The four operational signals (reference, plan, tool, control) plus entity linking are sufficient to reconstruct the true dependency structure of a trace.
    Section 3.1 edge induction. The Limitations explicitly admit implicit dependencies from shared background knowledge are missed. If the DAG is incomplete, dependency-distance slicing compresses the wrong steps.
  • domain assumption The LLM entity-linking and ECA salience scanner outputs are reliable.
    Sections 3.1 and 3.3 invoke DOUBAO-SEED-1.6-FLASH to build edges and flag salient steps; errors propagate into the context layers and final attribution.
  • domain assumption The CockpitBench annotations (agent, step, category, ASIL) are correct and the three-expert consensus resolves ambiguity adequately.
    Appendix A.1.3. Fleiss kappas are only 0.68 for step and 0.71 for agent, so a nontrivial fraction of labels are contested; the evaluation treats them as ground truth.
  • domain assumption ASIL severity is a monotone ordinal scale suitable for the max and uplift operations in Eqs. (3)-(4).
    Section 3.4 uses the ASIL partial order D>C>B>A as a scalar priority; no evidence is given that severity differences are exponential as modeled by the beta parameter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits." pith.science (2026). https://pith.science/paper/CHD5N5XK

@misc{pith2026260801805,
  author       = {Pith},
  title        = {Pith review of: CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHD5N5XK}},
  note         = {Machine review of arXiv:2608.01805}
}
read the original abstract

LLM multi-agent systems suffer from Correctness Collapse, where high task-level accuracy conceals severe process-level failures. This is especially hazardous in safety-critical embodied settings such as automotive cockpits, where lexically correct utterances may trigger dangerous physical operations. Existing attribution methods rely on text traces alone, missing dependency structure, multi-channel evidence, and safety-aware evaluation. We introduce CockpitHAT, a hierarchical attribution framework that replaces positional windows with dependency-distance thresholds from interaction DAGs, integrates multi-channel evidence via an embodied adapter, and applies a safety-uplift to high-risk failures during confidence-weighted analyst consensus. We further release CockpitBench, a benchmark of 212 annotated failure traces spanning dialogue, vehicle-state, environmental, and memory channels, each labeled with ISO 26262 ASIL severity via three-expert consensus. On the public Who&When benchmark, CockpitHAT achieves agent-level / step-exact accuracies of 77.9% / 37.8% on the Hand-Crafted split and 86.5% / 46.0% on the Algorithm-Generated split, surpassing the text-only SOTA ECHO by up to 17.6 / 16.7 points. On CockpitBench, it attains 78.3% agent-level and 38.2% step-exact accuracy. These results establish dependency-aware, multi-channel, risk-calibrated attribution as an effective paradigm for reliable failure diagnosis in real-world embodied LLM multi-agent systems.

Figures

Figures reproduced from arXiv: 2608.01805 by the authors.

Figure 1
Figure 1. Dependency graph illustration of a multi-agent failure trace. The pipeline ingests role-tagged traces from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 54 canonical work pages

  1. [1]

    arXiv preprint arXiv:2509.13782 , year=

    Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis , author=. arXiv preprint arXiv:2509.13782 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2509.10401 , year=

    Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems , author=. arXiv preprint arXiv:2509.10401 , year=

    arXiv

  3. [3]

    2024 , howpublished =

    Vehicle Signal Specification (. 2024 , howpublished =

    work page 2024

  4. [4]

    arXiv preprint arXiv:2603.25001 , year=

    Rethinking failure attribution in multi-agent systems: A multi-perspective benchmark and evaluation , author=. arXiv preprint arXiv:2603.25001 , year=

    arXiv

  5. [5]

    Which Agent Causes Task Failures and When? On Automated Failure Attribution of

    Shaokun Zhang and Ming Yin and Jieyu Zhang and Jiale Liu and Zhiguang Han and Jingyang Zhang and Beibin Li and Chi Wang and Huazheng Wang and Yiran Chen and Qingyun Wu , booktitle=. Which Agent Causes Task Failures and When? On Automated Failure Attribution of. 2025 , url=

    2025

  6. [6]

    and Chai, S

    Lin, L. and Chai, S. and Wu, J. and Hu, H. and Gu, X. and Hu, H. and Zhang, F. and Wang, W. and Zhang, D. , title =. Proc. ACL Findings , year =

    work page

  7. [7]

    and others , title =

    Budzianowski, M. and others , title =. Proc. EMNLP , year =

    work page

  8. [8]

    DynaEval: Unifying turn and dialogue level evaluation , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

    work page

Show all 72 references
  1. [9]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    How to train long-context language models (effectively) , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  2. [10]

    2025 , eprint=

    Benchmarking Contextual Understanding for In-Car Conversational Systems , author=. 2025 , eprint=

    2025

  3. [11]

    Hange Zhou and Zhonglin Jiang and yingjie cui and Mingzhe Zhang and Xiaotang Wang and Hengwei Dai and Qiyao Yu and Yong Chen and Yongqi Zhang , booktitle=. Clarify. 2026 , url=

    2026

  4. [12]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Correctable-DST: mitigating historical context mismatch between training and inference for improved dialogue state tracking , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  5. [13]

    Proceedings of the IEEE , volume=

    Toward causal representation learning , author=. Proceedings of the IEEE , volume=. 2021 , publisher=

    2021

  6. [14]

    Landis, J. R. and Koch, G. G. , title =. Biometrics , volume =

  7. [15]

    Proceedings of the SIGDIAL 2013 Conference , pages=

    The dialog state tracking challenge , author=. Proceedings of the SIGDIAL 2013 Conference , pages=

    2013

  8. [16]

    Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue , pages=

    TD-EVAL: Revisiting task-oriented dialogue evaluation by combining turn-level precision with dialogue-level comparisons , author=. Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue , pages=

  9. [17]

    and others , title =

    Driess, D. and others , title =. Proc. ICML , year =

  10. [18]

    arXiv preprint arXiv:2509.15273 , year=

    Embodied arena: A comprehensive, unified, and evolving evaluation platform for embodied ai , author=. arXiv preprint arXiv:2509.15273 , year=

  11. [19]

    2025 , eprint=

    ChatGPT on the Road: Leveraging Large Language Model-Powered In-vehicle Conversational Agents for Safer and More Enjoyable Driving Experience , author=. 2025 , eprint=

    2025

  12. [20]

    Proceedings of the AAAI Conference on Artificial Intelligence , pages=

    Is-bench: Evaluating interactive safety of vlm-driven embodied agents in daily household tasks , author=. Proceedings of the AAAI Conference on Artificial Intelligence , pages=

  13. [21]

    2024 , note =

    Robin Bloomfield and John Rushby , title =. 2024 , note =

    2024

  14. [22]

    Voice-Controlled Headlight Shutdown in a Lynk & Co Vehicle Caused an Accident: Automakers Should Not Excessively Pursue Intelligent Features , year =

  15. [23]

    2026 , eprint=

    AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems , author=. 2026 , eprint=

    2026

  16. [24]

    MLAgentBench : Evaluating Language Agents on Machine Learning Experimentation

    Qian Huang and Jian Vora and Percy Liang and Jure Leskovec , booktitle =. MLAgentBench : Evaluating Language Agents on Machine Learning Experimentation

  17. [25]

    AssistantBench : Can Web Agents Solve Realistic and Time-Consuming Tasks?

    Ori Yoran and Samuel Joseph Amouyal and Chaitanya Malaviya and Ben Bogin and Ofir Press and Jonathan Berant , booktitle =. AssistantBench : Can Web Agents Solve Realistic and Time-Consuming Tasks?

  18. [26]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Agent-as-a-Judge: Evaluate Agents with Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  19. [27]

    Toolformer : Language Models Can Teach Themselves to Use Tools

    Timo Schick and Jane Dwivedi-Yu and Roberto Dess. Toolformer : Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems (NeurIPS) , year =

  20. [28]

    AnyTool : Self-Reflective, Hierarchical Agents for Large-Scale API Calls

    Yujia Qin and Shengding Hu and Yankai Lin and Weize Chen and Ning Ding and Ganqu Cui and Zheni Zeng and Yufei Huang and Chaojun Xiao and Chi Han and Yi Ren Fung and Yusheng Su and Huadong Wang and Cheng Qian and Ran Tian and Kunlun Zhu and Shihao Liang and Xingyu Shen and Boka...

  21. [29]

    ToolLLM : Facilitating Large Language Models to Master 16000+ Real-world APIs

    <tl-view><tl-origin>Yujia Qin</tl-origin><tl-mask>[ph_real_name_8_ph]</tl-mask></tl-view> and <tl-view><tl-origin>Shihao Liang</tl-origin><tl-mask>[ph_real_name_9_ph]</tl-mask></tl-view> and Yining Ye and Kunlun Zhu and Lan Yan and Yaxi Lu and Yankai Lin and Xin Cong and Xiang...

    2024

  22. [30]

    2025 , url =

    Zhu, Kunlun and Du, Hongyi and Hong, Zhaochen and Yang, Xiaocheng and Guo, Shuyi and Wang, Zhe and Wang, Zhenhailong and Qian, Cheng and Tang, Xiangru and Ji, Heng and You, Jiaxuan , booktitle =. 2025 , url =

    2025

  23. [31]

    arXiv preprint arXiv:2505.04364 , year=

    Benchmarking LLMs' Swarm intelligence , author=. arXiv preprint arXiv:2505.04364 , year=

  24. [32]

    2025 , eprint =

    Yang, Rui and Chen, Hanyang and Zhang, Junyu and Zhao, Mark and Qian, Cheng and Wang, Kangrui and Wang, Qineng and Koripella, Teja Venkat and Movahedi, Marziyeh and Li, Manling and others , journal =. 2025 , eprint =

    2025

  25. [33]

    Evaluating

    Ryo Kamoi and Sarkar Snigdha Sarathi Das and Renze Lou and Jihyun Janice Ahn and Yilun Zhao and Xiaoxin Lu and Nan Zhang and Yusen Zhang and Ranran Haoran Zhang and Sujeeth Reddy Vummanthala and Salika Dave and Shaobo Qin and Arman Cohan and Wenpeng Yin and Rui Zhang , booktit...

  26. [34]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Synchronous faithfulness monitoring for trustworthy retrieval-augmented generation , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  27. [35]

    arXiv preprint arXiv:2305.20050 , year =

    Let's Verify Step by Step , author =. arXiv preprint arXiv:2305.20050 , year =

    arXiv

  28. [36]

    ProcessBench : Identifying Process Errors in Mathematical Reasoning with Process Reward Models

    Chujie Zheng and Zhenrui Zhang and Beichen Zhang and Runji Lin and Keming Lu and Bowei Yu and Dayiheng Liu and Jingren Zhou and Junyang Lin , journal =. ProcessBench : Identifying Process Errors in Mathematical Reasoning with Process Reward Models

  29. [37]

    Where Did It All Go Wrong?

    Adi Banerjee and Anirudh Nair and Tarik Borogovac , journal =. Where Did It All Go Wrong?

  30. [38]

    AgentRx : Diagnosing AI Agent Failures from Execution Trajectories

    Shraddha Barke and Arnav Goyal and Alind Khare and Avaljot Singh and Suman Nath and Chetan Bansal , journal =. AgentRx : Diagnosing AI Agent Failures from Execution Trajectories

  31. [39]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    Mismatch between multi-turn dialogue and its evaluation metric in dialogue state tracking , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

  32. [40]

    Xing and Hao Zhang and Joseph E

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , title =. Advances in Neural Information Processing Systems (Neur...

  33. [41]

    Gonzalez and Ion Stoica , title =

    Tianle Li and Wei-Lin Chiang and Evan Frick and Lisa Dunlap and Tianhao Wu and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  34. [42]

    Advances in Neural Information Processing Systems , volume=

    Gorilla: Large language model connected with massive apis , author=. Advances in Neural Information Processing Systems , volume=

  35. [43]

    Proceedings of EMNLP , year =

    Minghao Li and Yingxiu Zhao and Bowen Yu and Feifan Song and Hangyu Li and Haiyang Yu and Zhoujun Li and Fei Huang and Yongbin Li , title =. Proceedings of EMNLP , year =

  36. [44]

    Proceedings of ICLR , year =

    Xiao Liu and Hao Yu and Hanchen Zhang and Yifan Xu and Xuanyu Lei and Hanyu Lai and Yu Gu and Hangliang Ding and Kaiwen Men and Kejuan Yang and Shudan Zhang and Xiang Deng and Aohan Zeng and Zhengxiao Du and Chenhui Zhang and Sheng Shen and Tianjun Zhang and Yu Su and Huan Sun...

  37. [45]

    Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan , title =

    Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan , title =. Proceedings of ICLR , year =

  38. [46]

    Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , title =

    Shuyan Zhou and Frank F. Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , title =. Proceedings of ICLR , year =

  39. [47]

    Proceedings of ICLR , year =

    Gr. Proceedings of ICLR , year =

  40. [48]

    arXiv preprint arXiv:2406.12045 , year =

    Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan , title =. arXiv preprint arXiv:2406.12045 , year =

    arXiv

  41. [49]

    White and Doug Burger and Chi Wang , title =

    Qingyun Wu and Gagan Bansal and Jieyu Zhang and Yiran Wu and Beibin Li and Erkang Zhu and Li Jiang and Xiaoyun Zhang and Shaokun Zhang and Jiale Liu and Ahmed Hassan Awadallah and Ryen W. White and Doug Burger and Chi Wang , title =. arXiv preprint arXiv:2308.08155 , year =

    arXiv

  42. [50]

    Proceedings of ICLR , year =

    Sirui Hong and Mingchen Zhuge and Jonathan Chen and Xiawu Zheng and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu and J. Proceedings of ICLR , year =

  43. [51]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Guohao Li and Hasan Hammoud and Hani Itani and Dmitrii Khizbullin and Bernard Ghanem , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  44. [52]

    Proceedings of ACL , year =

    Chen Qian and Wei Liu and Hongzhang Liu and Nuo Chen and Yufan Dang and Jiahao Li and Cheng Yang and Weize Chen and Yusheng Su and Xin Cong and Juyuan Xu and Dahai Li and Zhiyuan Liu and Maosong Sun , title =. Proceedings of ACL , year =

  45. [53]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Guanzhi Wang and Yuqi Xie and Yunfan Jiang and Ajay Mandlekar and Chaowei Xiao and Yuke Zhu and Linxi Fan and Anima Anandkumar , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  46. [54]

    Potsawee Manakul and Adian Liusie and Mark J. F. Gales , title =. Proceedings of EMNLP , year =

  47. [55]

    Zhao and Ni Lao and Hongrae Lee and Da-Cheng Juan and Kelvin Guu , title =

    Luyu Gao and Zhuyun Dai and Panupong Pasupat and Anthony Chen and Arun Tejasvi Chaganty and Yicheng Fan and Vincent Y. Zhao and Ni Lao and Hongrae Lee and Da-Cheng Juan and Kelvin Guu , title =. Proceedings of ACL , year =

  48. [56]

    Proceedings of EMNLP , year =

    Sewon Min and Kalpesh Krishna and Xinxi Lyu and Mike Lewis and Wen-tau Yih and Pang Wei Koh and Mohit Iyyer and Luke Zettlemoyer and Hannaneh Hajishirzi , title =. Proceedings of EMNLP , year =

  49. [57]

    arXiv preprint arXiv:2211.14275 , year =

    Jonathan Uesato and Nate Kushman and Ramana Kumar and Francis Song and Noah Siegel and Lisa Wang and Antonia Creswell and Geoffrey Irving and Irina Higgins , title =. arXiv preprint arXiv:2211.14275 , year =

    arXiv

  50. [58]

    and Lin, M

    Chen, X. and Lin, M. and Sch\". Teaching Large Language Models to Self-Debug , booktitle =

  51. [59]

    and Tandon, N

    Madaan, A. and Tandon, N. and Gupta, P. and Hallinan, S. and Gao, L. and Wiegreffe, S. and Alon, U. and Dziri, N. and Prabhumoye, S. and Yang, Y. and Gupta, S. and Majumder, B. P. and Hermann, K. and Welleck, S. and Yazdanbakhsh, A. and Clark, P. , title =. Proc. NeurIPS , year =

  52. [60]

    and Cassano, F

    Shinn, N. and Cassano, F. and Berman, E. and Gopinath, A. and Narasimhan, K. and Yao, S. , title =. Proc. NeurIPS , year =

  53. [61]

    arXiv preprint arXiv:2309.07045 , year =

    Zhexin Zhang and Leqi Lei and Lindong Wu and Rui Sun and Yongkang Huang and Chong Long and Xiao Liu and Xuanyu Lei and Jie Tang and Minlie Huang , title =. arXiv preprint arXiv:2309.07045 , year =

    arXiv

  54. [62]

    Proceedings of ICML , year =

    Mantas Mazeika and Long Phan and Xuwang Yin and Andy Zou and Zifan Wang and Norman Mu and Elham Sakhaee and Nathaniel Li and Steven Basart and Bo Li and David Forsyth and Dan Hendrycks , title =. Proceedings of ICML , year =

  55. [63]

    arXiv preprint arXiv:2402.05044 , year =

    Lijun Li and Bowen Dong and Ruohui Wang and Xuhao Hu and Wangmeng Zuo and Dahua Lin and Yu Qiao and Jing Shao , title =. arXiv preprint arXiv:2402.05044 , year =

    arXiv

  56. [64]

    Proceedings of EMNLP-IJCNLP , year =

    Mor Geva and Yoav Goldberg and Jonathan Berant , title =. Proceedings of EMNLP-IJCNLP , year =

  57. [65]

    2001 , publisher=

    Causality: models, reasoning, and inference , author=. 2001 , publisher=

    2001

  58. [66]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Atticus Geiger and Hanson Lu and Thomas Icard and Christopher Potts , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  59. [67]

    Keith and Emaad Manzoor and Reid Pryzant and Dhanya Sridhar and Zach Wood-Doughty and Jacob Eisenstein and Justin Grimmer and Roi Reichart and Margaret E

    Amir Feder and Katherine A. Keith and Emaad Manzoor and Reid Pryzant and Dhanya Sridhar and Zach Wood-Doughty and Jacob Eisenstein and Justin Grimmer and Roi Reichart and Margaret E. Roberts and Brandon M. Stewart and Victor Veitch and Diyi Yang , title =. Transactions of the ...

  60. [68]

    Causal Reasoning and Large Language Models: Opening a New Frontier for Causality , journal =

    Emre K. Causal Reasoning and Large Language Models: Opening a New Frontier for Causality , journal =

  61. [69]

    arXiv preprint arXiv:2204.01691 , year=

    Do as i can, not as i say: Grounding language in robotic affordances , author=. arXiv preprint arXiv:2204.01691 , year=

    arXiv

  62. [70]

    Conference on Robot Learning , pages=

    Rt-2: Vision-language-action models transfer web knowledge to robotic control , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  63. [71]

    Proceedings of the European Conference on Computer Vision (ECCV) , year =

    Chonghao Sima and Katrin Renz and Kashyap Chitta and Li Chen and Hanxue Zhang and Chengen Xie and Jens Beiwenger and Ping Luo and Andreas Geiger and Hongyang Li , title =. Proceedings of the European Conference on Computer Vision (ECCV) , year =

  64. [72]

    Proceedings of EMNLP-IJCNLP , year =

    Thierry Deruyttere and Simon Vandenhende and Dusan Grujicic and Luc Van Gool and Marie-Francine Moens , title =. Proceedings of EMNLP-IJCNLP , year =

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.