Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

The paper claims that a multi-turn, structure-guided diagnosis agent, DeepDebug, localizes root causes in LLM agent traces more accurately than single-pass readers, and that using its evidence-backed fix as the retry directive repairs more

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:26 UTC pith:FBBZW2WN

load-bearing objection Open-source agent debugging toolkit with honest limitations; the headline GAIA recovery gain is a full-recipe result, not an isolation of attribution, and the paper says so itself. the 2 major comments →

arxiv 2607.18754 v1 pith:FBBZW2WN submitted 2026-07-21 cs.AI cs.CL

AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents

Kunlun Zhu , Xuyan Ye , Zhiguang Han , Yuchen Zhao , Bingxuan Li , Weijia Zhang , Muxin Tian , Xiangru Tang
show 4 more authors
Pan Lu James Zou Jiaxuan You Heng Ji
This is my paper
classification cs.AI cs.CL
keywords LLM agentsroot-cause attributionfailure debuggingobservabilityself-correctionmulti-turn diagnosistrace analysisagentic debugging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agents fail in ways that are hard to debug because the step where the error appears is rarely the step that caused it. This paper presents AgentDebugX, a toolkit that organizes debugging as a closed loop — detect, attribute, recover, rerun — and centers on DeepDebug, a multi-turn diagnosis agent that reads the whole trace, investigates the structure of the run, and cross-examines two candidate root-cause steps before emitting an auditable report with one concrete fix. On the Who&When benchmark, DeepDebug reaches 28.8% strict agent-and-step accuracy on an open-weight model, above 21.7% for the strongest single-pass baseline. On GAIA, its fix repairs 13 of 73 failed tasks in one rerun, lifting overall accuracy from 55.8% to 63.6%, compared with 4–6 repairs for three decoupled self-correction baselines. If correct, this shows that better attribution can translate directly into better recovery for agent systems.

Core claim

The central claim is that root-cause attribution in LLM agents is best done by a multi-turn agent that combines a global trajectory read with a structure-guided investigation and a final cross-examination between two candidate steps, reducing the search over the whole trace to an adjudication between two hypotheses. This DeepDebug procedure achieves the best strict attribution accuracy among evaluated methods on open-weight backbones (28.8% agent-and-exact-step on qwen3.5-9b vs 21.7% for the strongest single-pass localizer), with the margin concentrated on traces longer than 40 events. When its localized, evidence-backed fix is fed back as the retry directive, one rerun repairs 13 of 73 fail

What carries the argument

DeepDebug: a read-only, multi-turn diagnosis agent whose load-bearing moves are a global full-trace read; a structure-guided second pass that either walks the handoff cascade upstream (multi-agent) or bisects the step range (single-agent); and a cross-examination step that, when the two passes disagree, adjudicates between the two candidate steps using surrounding context. The agent then emits a structured report naming the responsible agent and step, with quoted evidence and one concrete fix. The paper argues this design shrinks root-cause selection from a search over the entire trace to a binary adjudication, which is why it outperforms single-pass and search baselines on long traces. The

Load-bearing premise

The recovery result comes from comparing complete recipes — DeepDebug's diagnosis with an authored fix versus generic self-correction summaries — so the paper cannot separate how much of the repair gain is due to finding the right step versus writing a better fix; the attribution advantage itself also fails to appear on the hosted backbones tested.

What would settle it

Compare GAIA recovery under three conditions: DeepDebug's report with the true root-cause step, the same report with a randomly replaced step index, and the same report with the fix removed. If repairs stay near 13 when the step index is wrong, or drop when the fix is removed, the claim that localization drives recovery is falsified. A second test: run the full Who&When benchmark on hosted backbones such as gpt-5.4-mini; the paper already reports single-pass reading is stronger there, so DeepDebug should show no strict-accuracy gain on that backbone.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deployments that escalate ambiguous failures to a multi-turn diagnosis agent can expect better root-cause localization on long, multi-agent traces, where single-pass readers anchor on the loudest downstream symptom.
  • Closing the loop from attribution to recovery roughly doubles the one-rerun repair rate over generic self-correction on GAIA (13 vs 4–6 of 73), so evidence-backed, step-specific retry directives appear to make recovery more effective.
  • Attribution gains are model-dependent: on hosted backbones a single global read is already strong, so per-model routing of debugging depth preserves accuracy while keeping token cost near 1.6x a single pass.
  • A portable, framework-independent trace format plus a scrubbed, opt-in error hub can turn failures into regression fixtures and shared debugging memory across teams, though this effect is not yet evaluated.
  • The taxonomy inducer can propose new failure modes (e.g., multi-agent deadlock) without overwriting the curated seed, letting the debugger track long-tail errors over time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment would hold the fix text constant and vary only the step index — comparing DeepDebug's report at the true root-cause step against the same report at a randomly replaced step — to isolate how much of the 13/73 GAIA gain is due to localization rather than the authored fix itself.
  • Because the attribution advantage appears only on open-weight backbones, the multi-turn approach may be most valuable where the underlying policy is weaker; stronger hosted models might need a lighter adjudication rule or a different structural probe.
  • The error-hub memory effect is implemented but untested; if retrieval of similar past diagnoses seeds future hypotheses, repeated debugging within a team could compound over time — a testable extension that would require running the loop over successive tasks with the hub enabled.
  • The cost-aware routing among rules, single-pass readers, and multi-turn adjudication suggests a debugging-as-a-service pattern where easy cases are nearly free to diagnose, making continuous regression debugging in CI a practical possibility.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents AgentDebugX, an open-source toolkit that organizes LLM-agent debugging as a closed Detect–Attribute–Recover–Rerun loop. The core contribution is DeepDebug, a multi-turn diagnosis agent that performs a global trajectory read, a structure-guided investigation, and a cross-examination stage, producing an auditable root-cause report with a concrete fix. The paper evaluates two capabilities: failure attribution on the Who&When benchmark (n=184), where DeepDebug reports 28.8% strict agent-and-step accuracy on qwen3.5-9b versus 21.7% for the strongest single-pass baseline, and end-to-end recovery on GAIA validation, where the native DeepDebug path repairs 13 of 73 failed tasks in one rerun versus 4–6 for three decoupled self-correction baselines, raising overall accuracy from 55.8% to 63.6%. The paper also documents the toolkit's surfaces: Python library, CLI, web console, agentic skill, and an opt-in Error Hub.

Significance. If the results hold, AgentDebugX addresses a real and timely gap: most observability tools replay traces but do not localize root causes or connect diagnosis to recovery. The paper's strengths include a clean portable trace abstraction, a policy-gated rerun loop, a reproducible evaluation artifact (a script regenerating Table 3 and per-task outputs), and unusually honest limitation statements in Appendix E. The Who&When attribution result is internally consistent and uses the benchmark's reference-answer protocol; the reported gains on open-weight backbones are meaningful, though they are only against in-house baselines. The GAIA recovery result is promising as a full-recipe demonstration, but the paper's stronger mechanistic claim—that better attribution drives recovery—is not supported by the experiment as designed, because the compared retry directives differ in both attribution detail and repair content. With an added ablation or carefully reworded conclusions, the toolkit contribution would be solid.

major comments (2)
  1. [§4, Table 3, and Appendix E] The GAIA recovery experiment compares complete retry recipes, not attribution quality in isolation. In the native path, DeepDebug supplies its localized diagnosis, evidence, and an authored fix as the retry directive; the decoupled baselines (Reflexion, CRITIC, AutoManual) receive only a generic judge-produced failure summary. The paper itself acknowledges this in §4 ('the experiment evaluates the full recipe rather than isolating attribution alone') and in Appendix E ('it neither isolates attribution's effect alone'). Because the retry prompt differs in both attribution detail and repair content, the 13/73 versus 4–6 repair gap cannot be attributed to better attribution. Consequently, the Conclusion's statement that 'improving fault attribution can translate into measurable gains in downstream recovery' is stronger than the evidence supports. I request either (a) an ablation that gives
  2. [§4, Table 2, and Takeaway] The attribution experiment only compares DeepDebug against rule heuristics and four simple single-pass localizers; it does not compare against published attribution systems such as AgentDebug, TRAIL, or AgentDiagnose, despite these being discussed in Related Work. The abstract carefully says 'among the evaluated methods,' but the Takeaway in §4 states that DeepDebug 'delivers state-of-the-art attribution on Who&When,' which is not established without external baselines. I recommend either adding comparisons to at least one published attribution system on the same benchmark or softening the 'state-of-the-art' language to 'best among the evaluated baselines.'
minor comments (4)
  1. [Figure 3] The numeric labels in the right panel are visually hard to parse, especially the two-column layout. Since the >40-event group contains only 26 traces, consider adding error bars or explicit confidence intervals, or presenting the numbers in a table, to avoid over-reading the descriptive regime.
  2. [§3.3 / Appendix A] The DeepDebug cross-examination prompt includes the 'Correct answer' from the reference answer. This is consistent with the Who&When protocol and is disclosed, but mentioning it in the main text near Stage 3 would help readers who do not consult Appendix C understand that the model receives the reference answer.
  3. [Appendix A] The example diagnostic report contains the metadata field "backend": "aao_moe", which is not defined anywhere in the paper. Either define this backend or remove the field from the example.
  4. [Table 1] The column header 'Port. schema' is abbreviated without explanation in the table caption. Since this is a key differentiator, spell out 'Portable schema' or add a footnote.

Circularity Check

0 steps flagged

No significant circularity: headline results are externally benchmarked and the only close concern is an explicitly acknowledged experimental confound, not a circular derivation.

full rationale

The paper's central claims are empirical measurements against external benchmarks, not derivations from fitted parameters or definitions. Who&When attribution results (Table 2) use the benchmark's reference-answer protocol with gold labels withheld, and DeepDebug's outputs are compared against independent single-pass baselines; the strict agent-and-step metric is not constructed from DeepDebug's own predictions. GAIA recovery results (Table 3) are scored by the official GAIA scorer on a fixed 73-task failed subset, with DeepDebug's recovery compared against three decoupled self-correction baselines. The one legitimate concern—that the GAIA experiment evaluates the full retry recipe rather than isolating attribution's causal contribution—is explicitly acknowledged by the authors in Section 4 ('the experiment evaluates the full recipe rather than isolating attribution alone') and Appendix E ('it neither isolates attribution's effect alone nor is a blind single-shot score'). That is an experimental confound, not circularity: DeepDebug's diagnosis is not force-fed into the metric, and the result would be the same even if it were a weaker method that happened to include a good fix. Self-citations (e.g., AgentDebug, Zhu et al. 2025) appear only as related-work context and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work to justify the method. The paper is self-contained against external benchmarks, so no circular step is exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is an empirical systems paper with no numerical free parameters or fitted constants. The central claims rest on benchmark protocol assumptions, the fidelity of the trace schema, and the representativeness of a single temperature-0 run.

axioms (4)
  • domain assumption Who&When gold attribution labels and the reference-answer protocol are correct and exhaustive for measuring root-cause attribution.
    Section 4 and Appendix C treat the benchmark's gold agent/step as ground truth for all methods; if labels are noisy, all attribution metrics shift.
  • domain assumption A single GAIA rerun at temperature 0 with the same tool stack is a valid measure of recovery improvement.
    Section 4 and Appendix C run one pass per method with no repeated seeds or significance bounds, so this is an unverified assumption.
  • domain assumption The portable AgentEvent/AgentTrajectory schema faithfully preserves the information needed for root-cause attribution across frameworks.
    Section 3.1 assumes events converted from LangGraph, CrewAI, OpenAI Agents SDK, and OpenTelemetry retain the inputs, outputs, and context needed for diagnosis.
  • domain assumption LLM judge and DeepDebug decisions at temperature 0 are deterministic and representative of deployment behavior.
    Appendix C sets decoding at temperature 0 with thinking disabled and no repeated runs, leaving variance and determinism unmeasured.

pith-pipeline@v1.3.0-alltime-deepseek · 11103 in / 12097 out tokens · 193416 ms · 2026-08-01T14:26:16.999170+00:00 · methodology

0 comments
read the original abstract

LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it. Existing observability tools replay execution traces but provide little support for identifying the root cause or translating diagnosis into recovery. We present AgentDebugX, an open-source debugging framework that organizes debugging as a closed loop of Detect, Attribute, Recover, and Rerun. At its core, DeepDebug performs multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who and When benchmark, DeepDebug achieves the best strict attribution accuracy among the evaluated methods on both tested open-weight backbones, reaching 28.8 percent exact agent-and-step accuracy on qwen3.5-9b versus 21.7 percent for the strongest single-pass baseline. On GAIA, DeepDebug repairs 13 of 73 failed tasks in a single rerun, compared with 4 to 6 for three decoupled self-correction baselines, improving overall accuracy from 55.8 percent to 63.6 percent. AgentDebugX exposes this workflow through a Python library, CLI, web console, and installable agentic skill, and provides an opt-in Error Hub for sharing scrubbed failure-diagnosis-repair bundles and reusing them as debugging memory.

Figures

Figures reproduced from arXiv: 2607.18754 by Bingxuan Li, Heng Ji, James Zou, Jiaxuan You, Kunlun Zhu, Muxin Tian, Pan Lu, Weijia Zhang, Xiangru Tang, Xuyan Ye, Yuchen Zhao, Zhiguang Han.

Figure 1
Figure 1. Figure 1: Overview of AgentDebugX. AgentDebugX forms a closed debugging loop: Detect identifies step-level errors, Attribute locates their root causes, Recover proposes ranked fixes, and Rerun regenerates the trajectory around the failure point. Hard cases are escalated to DeepDebug, a multi-round diagnosis agent that produces an auditable root-cause report with evidence and a recommended fix. AgentDebugX supports l… view at source ↗
Figure 2
Figure 2. Figure 2: The AgentDebugX console on a failed run, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Who&When accuracy by trace length (n=184; qwen3.5-9b). Left: responsible-agent ac￾curacy (who). Right: strict joint accuracy requiring both the responsible agent and the exact mistake step (who+when) to be correct. Method Agent Step Step A+S A+S (exact) (±1) (exact) (±1) Rule heuristic 14.1 1.6 6.5 1.6 2.7 All-at-Once 47.8 22.3 35.3 21.7 23.9 Step-by-Step 41.8 18.5 38.0 17.9 19.6 Binary-Search 41.8 17.4 38… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Cognitive Architectures to Language Agents: A Mechanism-Level Review of Lineage, Convergence, and Migration Gaps

    cs.AI 2026-07 conditional novelty 6.0

    Coding each mechanism for evidence of lineage and implementation depth, the review closes one candidate gap (GraSP) and isolates five residual control bundles for language agents.

  2. Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

    cs.SE 2026-07 conditional novelty 6.0

    On the tqdm benchmark tasks, the best output format depends on the model: Doubao thrives with JSON Patch (94%), DeepSeek with unified diff (66%), and Qwen with full file (50%).

Reference graph

Works this paper leans on

12 extracted references · 4 linked inside Pith · cited by 2 Pith papers

  1. [3]

    trace_id

    DeepDebug report (GAIA task 04a04a9b, abridged) { "trace_id": "04a04a9b-...", "root_cause_agent": "print", "root_cause_step_index": 9, "findings": [{ "failure_mode": "attribution.root_cause", "step_index": 9, "confidence": 0.6, "evidence": [ "# Assuming uniform distribution [0, 2*average_p_value]", "proportion_incorrect = (2*average_p_value - threshold) /...

    2026

  2. [4]

    Liming Dong, Qinghua Lu, and Liming Zhu

    Trail: Trace reasoning and agentic issue local- ization.Preprint, arXiv:2505.08638. Liming Dong, Qinghua Lu, and Liming Zhu

    Pith/arXiv arXiv

  3. [5]

    Preprint, arXiv:2411.05285

    Agentops: Enabling observability of llm agents. Preprint, arXiv:2411.05285. Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen

    Pith/arXiv arXiv

  4. [6]

    A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark

    arXiv

  5. [7]

    https://github.com/ open-telemetry/semantic-conventions-genai

    Opentelemetry semantic con- ventions for generative AI. https://github.com/ open-telemetry/semantic-conventions-genai . Accessed 2026-07-09. Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue

    2026

  6. [8]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 207–215, Suzhou, China

    AgentDiagnose: An open toolkit for diagnosing LLM agent trajectories. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 207–215, Suzhou, China. As- sociation for Computational Linguistics. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao

    2025

  7. [9]

    InFindings of the Association for Compu- tational Linguistics: ACL 2024, pages 13894–13908, Bangkok, Thailand

    LLMs cannot find rea- soning errors, but can correct them given the error location. InFindings of the Association for Compu- tational Linguistics: ACL 2024, pages 13894–13908, Bangkok, Thailand. Association for Computational Linguistics. Jian Yang, Xianglong Liu, Weifeng Lv, Ken Deng, Shawn Guo, Lin Jing, Yizhi Li, Shark Liu, Xianzhen Luo, Yuyu Luo, and 1 others

    2024

  8. [10]

    Guibin Zhang, Junhao Wang, Junjie Chen, Wangchun- shu Zhou, Kun Wang, and Shuicheng YAN

    From code foundation models to agents and applications: A comprehensive survey and practical guide to code intelligence.arXiv preprint arXiv:2511.18538. Guibin Zhang, Junhao Wang, Junjie Chen, Wangchun- shu Zhou, Kun Wang, and Shuicheng YAN

    arXiv

  9. [11]

    findings

    Where llm agents fail and how they can learn from failures. Preprint, arXiv:2509.25370. A System and Prompt Details Trace schema.Each run is an AgentTrajectory of AgentEvents (type, agent, module, step in- dex, parent, timestamp, inputs/outputs, error, du- ration, metadata, artifacts). Artifacts can hold text, images, audio, UI state, files, or environmen...

    arXiv

  10. [2024]

    Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian

    Automan- ual: Constructing instruction manuals by llm agents via interactive environmental learning.Preprint, arXiv:2405.16247. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian

    Pith/arXiv arXiv

  11. [2025]

    Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal

    A survey on large language models for automated planning.arXiv preprint arXiv:2502.12435. Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal

    Pith/arXiv arXiv

  12. [2026]

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E

    Agen- trx: Diagnosing ai agent failures from execution tra- jectories.Preprint, arXiv:2602.02475. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica

    arXiv