REVIEW 3 major objections 8 minor 32 references
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
T0 review · 3 major / 8 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that a multi-perspective auditor can find the exact step where a long-horizon search agent fails, explain why, and repair it, reaching a 32.3% end-to-end pass rate and recovering 17.4% of failed runs on a held-out…
desk verdict Solid benchmark and a genuinely useful recovery result; the headline gap over baselines is plausible but rests on author-produced labels with no inter-annotator agreement, so treat the exact numbers as provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the decoupling of diagnosis proposal from adjudication. Three parallel audit branches—a holistic read, a backward constraint audit that checks each query constraint against trajectory evidence, and a forward timeline audit that walks decisions chronologically—each propose a critical step and a root cause. A single adjudicator then re-decides the diagnosis using deterministic disagreement summaries, a global outline of all decision steps, and verbatim evidence windows around each proposed step, following rules that treat agreement as evidence rather than a vote and prefer the earliest supported step. Only after the diagnosis is fixed does a repair synthesizer write two to four atomic, process-level directives that are executable from the critical step onward. This three-stage pipeline is what the ablations identify as the source of the gains: replacing the specialized branches with extra holistic copies drops the end-to-end fully-passed score by 4.3 points, replacing the LLM adjudicator with voting drops it by 6.3 points, and removing the outline and evidence windows drops it by 4.3 points.
What would settle it
Re-annotate a random sample of about 200 SearchAuditBench trajectories with at least two independent annotators who are blind to the original labels, and measure agreement on the critical step (within the tolerance span) and on the root cause; if agreement falls well below the levels the benchmark implicitly assumes (for example, Cohen's kappa below 0.5 on root causes, or fewer than 60% of step choices inside the original tolerance spans), then the gold labels are unstable and the reported 32.3% end-to-end pass rate would not survive a change of annotator.
Extended reading notes
Core claim
The central claim is that failures of deep-search agents can be audited automatically: an LLM can identify the earliest assistant decision that fixes the failure path, assign it one of six root causes (Candidate Mismanagement, Search Coverage Gap, Constraint Neglect, Unverified Source Reliance, Entity–Relation Misbinding, Unsupported Answer), and write a case-specific process repair. The paper also claims that doing this well requires more than a single pass over the trace: three complementary audit perspectives (holistic, backward constraint tracing, forward timeline) generate candidate diagnoses, and an adjudicator re-decides the diagnosis over bounded evidence windows plus a global outline, which beats both majority voting and any single perspective. On 1,243 expert-annotated failed trajectories, this reaches 44.9% exact critical-step localization and 32.3% end-to-end fully passed audits with a GPT-5.5 backbone, and the improvement holds across Gemini and Claude backbones. The paper further claims the repairs are practically useful: resuming a failed run from the predicted critical step with the repair directive appended fixes 17.4% of Kimi-K2.6 failures and 10.3% of Quest-35B failures on LiveBrowseComp, roughly doubling unguided retry, and outperforming a generic hint that shares the same localization. Finally, the corpus shows that most failures (77.2%) are not retrieval failures but errors in how the agent used evidence it already had, and that nearly half of all tool calls and generated tokens occur after the critical error has already been committed.
Load-bearing premise
The gold labels that define a correct audit—the critical step, its tolerance span, the root cause, and the repair rubrics—are trustworthy, even though each trajectory was annotated by a single person with no reported agreement check, so noisy or biased labels would make every pass rate uninterpretable.
Editorial extensions
If this is right
- If the benchmark's pass rates are accepted, automated auditors are now competitive enough to serve as a first-pass failure diagnosis tool for deep-search agents, cutting the human inspection burden on 73-message, 65K-token trajectories.
- Repair-guided resumption, where a failed run is restarted from the predicted critical step with the repair appended, approximately doubles the recovery rate of unguided retry on both tested models.
- Because 77.2% of failures are attributed to how evidence was processed rather than whether it was retrieved, improving candidate management and constraint checking may matter more for search agents than expanding their search budgets.
- Since 47.1% of generated tokens and 45.4% of tool calls occur after the critical error is established, reliable online auditing could reclaim nearly half the compute spent by failing search agents.
- The failure-mode shift with exploration budget (Pearson r = −0.88 between Search Coverage Gap share and median tool calls) implies that simply scaling test-time search shifts, rather than removes, the binding failure mode.
Reading between the lines
- A practical near-term use may be human-in-the-loop auditing: the auditor proposes a diagnosis and a repair, a human confirms before the run is resumed, and that would sidestep the roughly 68% of audits that currently fail end-to-end.
- The benchmark's design could transfer to other tool-using agents (code, CLI, memory) by replacing the six-way search taxonomy with a domain-specific one and keeping the three-branch proposal-plus-adjudication architecture.
- The resumption-asymmetry observation suggests a testable extension: truncating the trajectory prefix more aggressively (earlier than the predicted critical step) might let agents escape flawed reasoning better than resuming exactly at the critical step.
- The 32.3% ceiling may understate practical value, because repair rubrics require all atomic directives to pass; a graded or partial-credit repair score could show larger recoverable fractions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents two contributions. First, SearchAuditBench, a benchmark of 1,243 expert-annotated failed deep-search trajectories (averaging 73.1 messages and 65.1K tokens) collected from eight open-weight models running five deep-search benchmarks under a unified scaffolding; each trajectory is labeled with a critical error step and tolerance span, a root cause from a six-way taxonomy, and a reference repair with atomic rubrics. Second, SearchAuditor, a prompt-based auditing pipeline consisting of three parallel audit branches (holistic, backward constraint, forward timeline), an evidence-grounded adjudicator, and a diagnosis-conditioned repair synthesizer. The paper reports that SearchAuditor outperforms all baselines (All-at-Once, Step-by-Step, Binary Search, AgentRx) on every metric across three frontier backbones (GPT-5.5, Gemini-3.1-Pro, Claude-Opus-4.8), raising the end-to-end fully-passed score from 26.55% to 32.26% with GPT-5.5. It also reports that resuming failed LiveBrowseComp runs with SearchAuditor's repairs lifts Kimi-K2.6 accuracy from 34.03% to 45.07%, that compute-matched ablations support the multi-perspective design, and that the corpus reveals systematic failure patterns such as a negative correlation between exploration budget and Search Coverage Gap share.
Significance. If the claims hold, the work is a solid contribution to agent failure auditing. Strengths include: a large reusable annotated corpus for an under-studied task; a parameter-free prompted method with full prompts in the appendix; a frozen-trajectory protocol that makes auditor evaluation reproducible and independent of live-web drift; well-designed compute-matched ablations (Table 3); and, most importantly, a held-out LiveBrowseComp recovery experiment whose generic-hint control cleanly isolates the value of the repair directive. The corpus-level findings (25% answer-in-hand rate, r=-0.88 between exploration budget and Search Coverage Gap share) are falsifiable claims other groups can test. The principal weakness is that all benchmark-relative numbers in Table 2 are anchored to gold labels authored by the same team, single-annotator, without inter-annotator agreement, and scored by an LLM rubric grader with a 21% case-level false-failure rate (Appendix D). The LiveBrowseComp result, being annotation-free, survives these concerns, but it validates the repair/utility claim rather than the benchmark-relative superiority claim.
major comments (3)
- [Section 3.3; Tables 2 and 6] The benchmark's gold labels are produced by a single annotator per case, with the 1,243 cases split disjointly across four annotators, and no inter-annotator agreement is reported. This is load-bearing because all Table 2 metrics are computed against these labels, and the labels themselves are highly judgment-dependent: the critical step is defined counterfactually ('correcting k* would remove this mechanism and make a correct final answer attainable under competent subsequent search', Section 3.2), and the tolerance span is constructed by the same annotator who chooses k*. Without an agreement measurement, the benchmark's reliability is unestablished and the 5.7-point FPS gap in Table 2 cannot be separated from annotator idiosyncrasy. Please re-annotate a random subset (at least 100 instances) with all four annotators and report per-field agreement (exact k*, within-span match, and root-cause kappa), and report Table 2 metrics by annotator partition to confirm the gains are not driven by a single annotator's cases.
- [Sections 3.3 and 4; Appendices B and G; Table 2] There is a genuine alignment risk between the annotation conventions and SearchAuditor's prompts. The gold rule labels 'the earliest decision step at which the annotated failure mechanism becomes established' (Section 3.2), and the same 'earliest decisive error' rule and the same six-way taxonomy definitions are embedded in the stage-1 and stage-2 prompts of Appendix G; the repair synthesizer is additionally instructed to emit 2-4 rubric-shaped atomic directives, mirroring the format of the gold rubrics it is graded against. SearchAuditor is therefore likely to reproduce the authors' labeling conventions, and part of its Table 2 advantage may reflect that alignment rather than general auditing skill. This is not strict circularity (the method never sees the gold labels) and it does not affect the annotation-free LiveBrowseComp experiment, but the benchmark-relative claim needs a robustness check: (i) the IAA measurement requested above; (ii) evaluation of SearchAuditor and the baselines against independently produced labels on a subset; and (iii) a per-annotator breakdown of the Table 2 metrics.
- [Appendix D; Table 2] The rubric-grading procedure is load-bearing for the FPS headline. The blind validation reports case-level kappa of 0.650 and a 21% false-failure rate (43 of 159 grader failures accepted by the human reference, Table 8), and the statement that the shared grader 'does not affect the relative comparisons' is asserted, not tested: the FP/FF rates are not broken down by method, backbone, or repair format. Because SearchAuditor's repairs are deliberately format-matched to the rubric style while the baselines' repairs are not, an LLM grader could systematically favor one family of outputs. Please report per-method agreement on the 200-case blind set, or recompute back-to-back FPS on that set with the human case outcomes, and ideally confirm the ranking with a second independent grader model.
minor comments (8)
- [Section 5.2; Table 2 and 4] No significance tests or confidence intervals accompany the Table 2 comparisons or the Table 4 fix-rate differences; with N=1,243 and paired failed-run cohorts, McNemar tests would be appropriate.
- [Table 3] Table 3's 300-instance subset has no reported sampling seed, and the Full-variant scores (CS-Strict 49.00, Rep@Diag 79.18) deviate from the full-benchmark SearchAuditor scores (44.89 and 84.78, Table 2) in opposite directions; the paper should report the seed and briefly reconcile the subset with the full-benchmark results.
- [Figure 4] Figure 4 reports Pearson r=-0.88 on eight points; a p-value or bootstrap interval should be reported, and the analysis should state whether the correlation uses model-level aggregates (n=8).
- [Appendix B.2] The pre-screening model is Claude Opus 4.6, from the same family as the evaluated Claude-Opus-4.8 backbone, so the claimed generator separation from the evaluated auditors is partial; the paper should report how often annotators accepted, corrected, or discarded the LLM proposals.
- [Table 4] The headline recovery gain rests on small counts (37 vs 20 fixed runs out of 213 for Kimi-K2.6); a test and confidence intervals for these proportions should be reported.
- [Appendix B.1] Appendix B.1 reports that 1,674 failed runs were reduced to 1,243 by the D1-D5 discard criteria, but the per-criterion discard counts are not given; reporting them would help readers assess selection bias in the root-cause distribution of Table 6.
- [Table 1] Table 1's header has a garbled column layout ('Scale' and 'Avg. trace length' appear in sequence); the table should state explicitly what each column measures.
- [Section 4, Stage 3] The sizes of the 'bounded window' and 'evidence window' used by the adjudicator and repair synthesizer are not specified; state them, since they determine how much context each stage sees.
Circularity Check
No construction-level circularity: SearchAuditor's benchmark results are empirical evaluations, and the repair-recovery claim is independently grounded on LiveBrowseComp; the remaining concerns are label reliability and prompt/label alignment, not circular derivation.
full rationale
SearchAuditBench is an expert-annotated evaluation corpus, not an input to a fitted model: SearchAuditor has no fitted parameters, and no passage in the paper derives its measured FPS or repair-recovery rates from the annotation labels by construction. The strongest practical claim (repairs lift Kimi-K2.6 accuracy on LiveBrowseComp from 34.03% to 45.07%) is checked against gold answers on a disjoint benchmark, where the only signal is whether the resumed run reaches the correct answer, so it is externally grounded and cannot reduce to the authors' rubric. The benchmark-based comparisons do embed the same taxonomy and the same 'earliest decisive error' rule in the auditor prompts (Appendix G) that the annotation guideline uses to create gold labels (Section 3.2 and Appendix B), and the gold labels are produced by a single annotator per case with no inter-annotator agreement reported (Section 3.3); these are validity and alignment risks that limit the generalizability of the headline FPS gap, but they do not make any predicted quantity equal to its input by construction. The Appendix D rubric grader additionally reports a 21% case-level false-failure rate without a per-method breakdown, another evaluation-robustness concern, yet the same grader is applied to all methods and the external LiveBrowseComp repair experiment provides an independent check. No load-bearing self-citation, uniqueness theorem, or imported ansatz is used. Therefore, no circular step is exhibited.
Assumptions & free parameters
assumptions (5)
- domain assumption Every retained failed trajectory contains enough frozen evidence to identify the critical error without web access.
- domain assumption A single earliest critical step exists for each trajectory and correcting it would make a correct answer attainable.
- ad hoc to paper The six-way root-cause taxonomy is complete and mutually exclusive for search-agent failures.
- domain assumption The LLM-based answer evaluator and the DeepSeek-V4-Flash rubric grader judge correctness reliably.
- domain assumption The unified Tongyi DeepResearch scaffold and the eight chosen models are representative of long-horizon search agents.
Cite this review
Pith. "Pith review of SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents." pith.science (2026). https://pith.science/paper/BV4IVKGN
@misc{pith2026260805212,
author = {Pith},
title = {Pith review of: SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/BV4IVKGN}},
note = {Machine review of arXiv:2608.05212}
}
read the original abstract
Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
exactly one`root_cause_primary`
-
[2]
Mind2web 2: Evaluating agentic search with agent- as-a-judge.AdvancesinNeuralInformationProcessingSys- tems, 38. Gupta, N.; Chatterjee, R.; Haas, L.; Tao, C.; Wang, A.; Liu, C.; Oiwa, H.; Gribovskaya, E.; Ackermann, J.; Blitzer, J.; Goldshtein, S.; and Das, D. 2026. DeepSearchQA: Bridg- ing the Comprehensiveness Gap for Deep Research Agents. arXiv:2601.20...
arXiv 2026
-
[3]
the tight tolerance span`[ks, ke]`
-
[4]
a trace-grounded`failure_rationale`
-
[5]
QUEST:TrainingFrontierDeepResearchAgentswith Fully Synthetic Tasks. arXiv:2605.24218. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu,B.;Gao,C.;Huang,C.;Lv,C.;Zheng,C.;Liu,D.;Zhou, F.;Huang,F.;Hu,F.;Ge,H.;Wei,H.;Lin,H.;Tang,J.;Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Zhou, J.; Lin, J.; Dang, K.; Bao, K.; Yang, K.; Yu, L.; D...
arXiv 2025
-
[6]
three to five required, atomic`repair_rubrics`. ## Root Cause Choose the earliest and most direct mechanism that establishes the failure path, not a downstream symptom and not merely the stage at which the error becomes visible. Use exactly one label; full definitions and boundary rules are given in Appendix A. -`Search Coverage Gap`: the decisive candida...
-
[7]
one critical assistant step`k *`
-
[10]
one executable`repair_directive`; and
Show all 32 references
-
[12]
, "predicted_answer
As of 2016, the individual stated that they dreamt about living in America had a degree in economics. 2. In a 2022 news article, the individual was trolled on social media as being ugly, even trolled by an old school senior. It was also stated in the article that they began da...
2016
-
[15]
dreamt about living in America had a degree in economics
As of 2022, it was stated that the individual's spouse was born in the United States and the spouse's father owned an investment company. We need to find who this is. Likely a public figure from Asia (maybe Korea, Japan, China?) who married an American spouse. "dreamt about li...
2022
-
[16]
As of 2016, the individual stated that they dreamt about living in America and had a degree in economics
2016
-
[17]
It was also stated in the article that they began dating their spouse at the age of 22
In a 2022 news article, the individual was trolled on social media as being ugly, even trolled by an old school senior. It was also stated in the article that they began dating their spouse at the age of 22
2022
-
[18]
"" tool_calls: [ {
As of 2022, it was stated that the individual's spouse was born in the United States and the spouse's father owned an investment company. I've confirmed that Peter Biar Ajak is a South Sudanese economist. He married Nyathon Hoth Mai. I need to verify more details about: - The ...
2022
-
[19]
only", "first
Parse the`query`into (a) the answer slot: what kind of entity/value must be produced, and (b) every hard constraint the answer must satisfy: exact dates, ages, counts, places, roles, relations, orderings, qualifiers such as "only", "first", " most". List each constraint separa...
-
[20]
Test the`predicted_answer`against each constraint using only trajectory evidence. Mark each constraint:`supported`(verified against opened, trustworthy content),` violated`(trajectory evidence contradicts it), or`unverified`(never checked against reliable content, or checked o...
-
[21]
Trace each`violated`or`unverified`constraint backward through the trajectory: find where the agent saw, should have seen, or skipped the relevant evidence, and locate the earliest assistant message where the mishandling happened—the constraint was ignored, relaxed, rationalize...
-
[22]
root_cause_primary
Decide the earliest decisive assistant step and the root cause per the taxonomy below . If several constraints were mishandled, pick the mishandling that established the failure path earliest and explains the final wrong answer most directly. ## Required Output Return exactly ...
-
[23]
From the`query`, sketch what a competent search plan would have to do: which entities or candidate sets to enumerate, which facts to verify against which kind of source, and in what order
-
[24]
Walk the assistant decision turns in chronological order (use`assistant_outline`to enumerate them, and read the full messages in`trajectory`). At each turn ask: after this turn, is the investigation still on a path that could reach a correct, fully verified answer? Watch for: ...
-
[25]
Note also the immediately preceding healthy turn as a sanity check
Identify the EARLIEST assistant turn at which the failure path becomes established— after it, subsequent turns mostly execute or rationalize the doomed direction. Note also the immediately preceding healthy turn as a sanity check
-
[26]
root_cause_primary
Decide the root cause per the taxonomy below, based on the mechanism of that earliest decisive turn, not on later symptoms. Guard against one timing mistake above all: do not drift late. The decisive turn is where the wrong direction is SET—the wrong framing, wrong exclusion, ...
-
[27]
A minority replica can be correct.** Never pick a label or step merely because two replicas said so; verify against the evidence windows and trajectory
**Majority vote is evidence only. A minority replica can be correct.** Never pick a label or step merely because two replicas said so; verify against the evidence windows and trajectory
-
[28]
message_index`
Re-decide **field by field**: first`root_cause_primary`, then`critical_step. message_index`. The two must be consistent—the chosen step must be where the chosen failure mechanism is actually committed
-
[29]
If all proposals are wrong and the true decisive step is elsewhere, you may override with a different assistant`message_index`, but only after locating it in the trajectory
For every proposed critical step, read its evidence window before judging. If all proposals are wrong and the true decisive step is elsewhere, you may override with a different assistant`message_index`, but only after locating it in the trajectory
-
[30]
root_cause_primary
Record provenance: for each decided field, note which replica you sided with, or` adjudicator_override`. ## Critical-Step Calibration `critical_step.message_index`marks the EARLIEST decisive error step—the point where the failure path becomes established and later errors are m...
-
[31]
Name its concrete target in THIS case—the exact search terms or query reformulation to run, the exact source/site/table/page type to open, the exact constraint or fact to re-verify (quote the query's own wording), the exact candidate operation (reject candidate X for failing c...
-
[32]
Be executable: an operator rerunning the search from the decisive step could follow it without further interpretation
-
[33]
For count/ranking/date/numeric questions, require recomputation or re-extraction from an authoritative source instead of giving the value
Stay process-level: never state, name, or numerically give the hidden target answer. For count/ranking/date/numeric questions, require recomputation or re-extraction from an authoritative source instead of giving the value. Naming a WRONG candidate that must be rejected (e.g.,...
-
[34]
repair_directive
Match the diagnosed root cause: repair the mechanism (coverage, source verification, candidate management, constraint checking, entity binding, or evidence grounding), not just the symptom. Recommended directive shapes (adapt the concrete items from this case): - "Reject <pred...
-
[2021]
Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332. OpenAI. 2026. OpenAI GPT-5 System Card. arXiv:2601.03267. Pham, T.; Nguyen, N.; Zunjare, P.; Chen, W.; Tseng, Y.-M.; and Vu, T. 2026. SealQA: Raising the Bar for Reasoning in Searc...
2026 arXiv
-
[2025]
arXiv:2504.12516
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516. Xie,J.;Lin,T.;Wang,Z.;Ning,Y.;Yao,Y.;Xue,T.;Zhang, Z.; Li, Z.; Zhang, K.; Wu, Y.; Chen, S.; Gou, B.; Han, M.; Wang, Y.; Lee, V.; Wei, X.; Wang, X.; Su, Y.; and Sun, H
-
[2026]
InLiakata,M.; Moreira, V
Beyond Single-shot Writing: Deep Research Agents areUnreliableatMulti-turnReportRevision. InLiakata,M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds.,Proceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 13325–133...
2025 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.