DR-Arena: an Automated Evaluation Framework for Deep Research Agents

Ruochen Zhao; Wenxuan Zhang; Yang Deng; Yiwen Gao

REVIEW 4 major objections 6 minor 1 cited by

This paper claims that a fully automated arena, generating live-verified tasks and adaptively escalating them, reproduces human-preference rankings of deep-research agents (Spearman 0.94).

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:16 UTC pith:SDVJB5LA

load-bearing objection Read this one for the framework, not for the 0.94; the live-tree adaptive-evaluation design is a real step forward, but the headline alignment with human preferences is a fragile six-model point estimate. the 4 major comments →

arxiv 2601.10504 v3 pith:SDVJB5LA submitted 2026-01-15 cs.CL

DR-Arena: an Automated Evaluation Framework for Deep Research Agents

Yiwen Gao , Ruochen Zhao , Yang Deng , Wenxuan Zhang This is my paper

classification cs.CL

keywords deep research agentsautomated evaluationdynamic benchmarksinformation treesLLM-as-a-judgeadaptive evaluationdepth and widthweb search evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper attempts to establish that evaluating deep-research agents can be fully automated without sacrificing alignment with human judgment. It does so with an automated Examiner that scrapes fresh web pages into verifiable Information Trees, generates questions and grading rubrics from those trees, and then runs adaptive follow-up rounds that push agents to their capability boundaries. The paper reports that this automated arena matches a human-preference leaderboard with a Spearman correlation of 0.94, the highest among the compared benchmarks and achieved with zero manual annotation. A sympathetic reader would care because static benchmarks decay and become contaminated, while DR-Arena offers a live, self-updating alternative that also diagnoses whether an agent fails at reasoning or at information coverage.

Core claim

The central claim is that a closed-loop, fully automated evaluation can recover the human ranking of top deep-research agents. The authors construct real-time Information Trees by scraping live web pages; an automated Examiner turns a branch of a tree into a question that requires both multi-hop reasoning (depth) and cross-entity aggregation (width), and simultaneously writes a rubric grounded in the tree's content. Candidate responses are judged against that rubric, the loser's failure is tagged as a DEEP, WIDE, BOTH, or NONE failure, and the Examiner chooses the next move from a state-machine transition table—descending deeper, widening the sibling set, backtracking, or pressure-testing—un

What carries the argument

The central mechanism is the dynamically constructed Information Tree: a directed graph of live web pages whose edges carry semantic relationship types. The tree serves simultaneously as the question source, the ground-truth rubric, and the map that the Adaptive Evolvement Loop uses to escalate task depth or width. The automated Examiner—an LLM agent—is the operator: it generates each question, grades responses against the checklist, tags the losing side's failure type, and uses that tag to select the next action from a transition matrix. The load-bearing loop is the cycle of tree expansion, task generation, rubric-based adjudication, and targeted escalation that continues until a decisive v

Load-bearing premise

The load-bearing premise is that the automatically scraped Information Trees and the Examiner-generated rubrics are factually correct often enough to serve as ground truth—yet the paper's own audit measured rubric factuality at 89.1%, and its Limitations section concedes hallucinations can originate at either generation or judging, including a documented case in which the judge overrode a correct rubric with a false memory and penalized the agent that was actually right.

What would settle it

Recompute the tournament leaderboard with judge overrides disabled (strict adherence to tree-derived rubrics only) and again with a different Examiner model; if either variant drops the Spearman correlation materially below 0.94, the reported alignment comes from the judge's parametric knowledge rather than from the automated dynamic trees.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Automated evaluation can replace human adjudication for ranking deep-research agents, at least when the goal is agreement with aggregate human preference.
Because questions and ground-truth rubrics are generated fresh from the live web, the benchmark does not decay as facts change and is less vulnerable to pretraining contamination.
The adaptive loop spends computation where it matters: closely matched pairs receive more probing rounds, while lopsided pairs resolve quickly, making ranking cheaper than round-robin evaluation.
Per-agent failure tags (Deep vs Wide) give diagnostic profiles of whether a model is weaker at multi-hop reasoning or at broad information gathering.
Rubric-based judgment and adaptive escalation are both load-bearing: removing the rubrics lowers Spearman from 0.94 to 0.83, and stopping after round one lowers it to 0.77.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Inference: The 0.94 correlation is likely a ceiling for the fully automated pipeline as described; a strict no-override variant would probably rank differently on documented judge-hallucination cases.
Inference: At 89.1% rubric factuality, roughly one in ten checklists contains an inaccuracy; a larger audit could measure how much that noise compresses true capability gaps.
Inference: The framework's explicit preference for factual exactness over fluency predicts that automated rankings will diverge most from crowd-based human rankings on stylistically polished but slightly stale answers—a testable prediction for future leaderboards.
Inference: The same closed loop could be extended to contradictory-source or adversarial tasks, where the tree would need to encode disagreement rather than a single ground truth, to test whether dynamic escalation still converges.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Read this one for the framework, not for the 0.94; the live-tree adaptive-evaluation design is a real step forward, but the headline alignment with human preferences is a fragile six-model point estimate.

read the letter

Read this one for the framework, not for the 0.94. The core idea—building a live Information Tree from fresh web content, generating depth/width tasks, then running an adaptive escalation loop that pushes agents until a decisive verdict emerges—is a real step forward for evaluating deep-research agents. It is not in the prior work they cite, and it directly targets contamination and temporal decay. The ablation is genuinely informative: cutting the evolvement loop drops Spearman from 0.94 to 0.77, and keeping only round 2 gives 0.94. That suggests the dynamic loop is doing real work, not just adding noise.

Credit where due: the paper is unusually candid. The Limitations section and Appendix G document a false judge override where the Examiner penalized the correct agent, a beneficial correction where the judge ignored a hallucinated rubric, and the factuality-fluency trade-off. That level of transparency makes me trust the pipeline more, not less. The human audit (kappa 0.91, rubric factuality 89.1%, transition accuracy 96.9%) is decent evidence that the components work on their own.

The soft spots are mostly around the headline. Spearman 0.94 is computed on six models, no confidence interval, no significance test. One adjacent swap drives it; with N=6, a second swap would put it around 0.83, which is the same as their w/o-rubric ablation. So the "state-of-the-art alignment" claim is not statistically grounded. Table 4 compares against baselines with different Ns (4–6), which makes those comparisons unreliable. There is also a single Examiner model (Gemini-3-Pro) and no robustness ablation across judges; given Appendix G.2, judge identity could matter a lot. Rubric factuality at 89.1% is good but not perfect, and in a small tournament a few wrong rubric items or judge overrides could flip the ordering.

The circularity worry—the same Examiner diagnoses failure, escalates, and terminates the match—is legitimate. The loop could theoretically amplify judge bias. But the external correlation with human LMSYS scores is the sanity check, and it is positive, which is evidence against the worst version of that risk. The fragility of N=6 is what should bother you, not the framework's internal logic.

These are fixable: bootstrap intervals, report all pairwise ranks, ablate the judge model, and ship pinned trees and logs. As written, the central claim is plausible but provisional. This deserves a serious referee—send it to review, with the expectation of heavy revision on the statistical front, and it could become a reference framework for deep-research evaluation.

Referee Report

4 major / 6 minor

Summary. DR-Arena proposes a fully automated, dynamic benchmark for deep-research (DR) agents. The system builds real-time Information Trees from live web sources, uses an LLM-based Examiner to generate Depth/Width tasks and rubric-grounded judgments, and then runs an Adaptive Evolvement Loop that escalates task difficulty based on the losing agent's diagnosed failure type. The authors evaluate six DR agents in a Swiss-style tournament and report a Spearman correlation of 0.94 with the human-annotated LMSYS Search Arena leaderboard, which they describe as state-of-the-art alignment without manual annotation. They also provide ablations, a human audit of components, and qualitative analyses showing strengths and failure modes of the automated pipeline.

Significance. If the headline result were robust, DR-Arena would be an important enabling contribution: it would provide a contamination-resistant, temporally aligned, fully automated evaluation protocol for open-ended research agents, and it would offer a quantitative alternative to costly human adjudication. The paper's core ideas are timely and well-motivated: dynamic task generation from live information trees, separation of depth vs. width capabilities, and an adaptive loop that concentrates computation at capability boundaries are genuinely useful design innovations. The authors also deserve credit for shipping machine-verifiable artifacts—code release, a full match walkthrough, prompt templates, and a documented human audit—and for candidly reporting failure cases and limitations. However, the central quantitative claim (0.94 Spearman) is based on only six models, and the paper's own evidence shows that the automated ground truth can be wrong; this makes the headline result statistically fragile and the 'state-of-the-art alignment' claim stronger than the current evidence supports.

major comments (4)

[§4.2, Table 3, Figure 3] The headline Spearman correlation of 0.94 is computed on only six models and corresponds to exactly one adjacent rank swap (Perplexity-Sonar-Pro and Claude-Opus-4.1). No confidence interval, bootstrap, or significance test is reported. With N=6, one additional rank perturbation can move the correlation below 0.90, so the gap between 0.94 and the ablation values of 0.83 and 0.77 (Table 6) is within the sampling noise of the estimator. Please report a bootstrap/permutation interval for the six-model correlation and state the exact p-value; also show the raw pairwise verdict matrix so readers can assess the sensitivity of the Elo ranks to individual judgments.
[§4.2, Table 4] The 'superiority over static benchmarks' comparison is not statistically meaningful as presented because the baselines use different numbers of models (N=4–6). Spearman values from different n are not directly comparable, and the variance of each estimate is large at these sample sizes. Please recompute all baseline correlations on a common subset of the same six models, and report uncertainty intervals for each benchmark.
[§3.2, Table 7, Appendix G.2, Limitations] The paper asserts in §3.2 that generated rubrics are 'factually correct and traceable to source URLs', but Table 7's own human audit measures rubric factuality at 89.1%, and Appendix G.2 documents a false judge override in which the Examiner's parametric knowledge reversed a correct, grounded checklist and penalized the agent that was historically accurate. A 10.9% rubric error rate, combined with an undocumented frequency of judge overrides, directly contaminates the ground truth on which verdicts, Elo scores, and the final ranking are built. With only six models, even a handful of incorrect verdicts can flip the reported order. The paper needs to quantify the judge-override rate and assess the sensitivity of the final leaderboard to rubric errors and overrides (e.g., by re-auditing all matches or by flagging and excluding conflicting judgments).
[§3.3, Table 2, Table 6] The Adaptive Evolvement Loop uses the same Examiner's failure diagnosis both to select the next harder question and to terminate at a decisive verdict. This creates a risk of bias amplification: if the judge misdiagnoses a failure in an early round, all subsequent pressure tests and the stopping criterion will be optimized toward that incorrect error model. The ablation shows that Round-1-only judgments yield Spearman 0.77 and that the loop increases this to 0.94, but this does not by itself prove that later rounds improve correctness—it could also reflect systematic amplification of an early bias. Please provide evidence that the loop improves verdict accuracy against human-adjudicated gold labels (e.g., per-round agreement with the human audit), rather than only improving agreement with the final ranking that the loop itself produced.

minor comments (6)

[§5.1, Figure 5] The caption and text state 'r = 0.61' while the text says 'r = −0.61, p = 0.045'. The sign is inconsistent; the negative sign is presumably correct for a negative correlation.
[Table 4] Two different benchmarks are both called 'Deep Research Bench' (Du et al. and FutureSearch et al.). Please disambiguate the names to avoid confusion.
[Abstract/§4.4] The abstract says 'without any manual efforts' but the paper relies on human annotators for the component audit and the task-generation ablation. Please rephrase to 'without manual annotation for the evaluation itself' or otherwise clarify.
[Appendix D] The 'Mercy Rule' is referenced in the walkthrough but is not defined in Algorithm 1 or in the main text. Please define it formally.
[§4.4, Table 7] The human audit uses only 30 full-match logs (64 turns) and two annotators. The reported Cohen's kappa of 0.91 is encouraging but the sample is small; a short statement on the confidence interval or a larger audit would be helpful.
[References] The reference 'Elo and Sloan., 1978' is malformed; the relevant citation should be Elo (1978) with a standard publisher entry.

Circularity Check

0 steps flagged

No significant circularity: the 0.94 alignment is measured against an external human leaderboard and is not defined in terms of DR-Arena's own outputs.

full rationale

The paper's central claim is that DR-Arena's automated ranking correlates with the human-built LMSYS Search Arena leaderboard (Spearman 0.94). This is an external comparison: the LMSYS scores are not inputs to DR-Arena, and no parameter is fitted to them. The evaluation pipeline (information-tree construction, rubric generation, judge verdicts, adaptive loop) produces an independent ranking that is then compared with human scores. The rubrics, although generated by the same Examiner LLM, are grounded in scraped live web content and are not defined in terms of the target LMSYS ranking. The self-citations (Auto-Arena, FACT-AUDIT, Ravaut et al.) appear only in related work and are not load-bearing for the headline result. Reported limitations such as 89.1% rubric factuality, the false judge override in Appendix G.2, and the small N=6 with no confidence interval are validity and statistical-concerns, not circular derivation steps. No quoted equation or procedure reduces the 0.94 correlation to an input by construction, so no circularity is established.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

The central claim depends on: a single LLM judge being reliable, automatically generated rubrics being factually correct, and the LMSYS leaderboard being a trustworthy human target. The paper's own audit quantifies rubric factuality at 89.1% and documents a false judge override. The free parameters are mainly thresholds and experimental design choices that are not swept, plus the fixed judge model.

free parameters (3)

Evolvement-loop termination thresholds = unspecified (e.g., mercy-rule diff ≥2.0 in Appendix D)
Hand-set constants that determine when a match ends and how much escalation occurs; they directly shape the Elo scores and the reported correlations.
Tournament configuration (4 Swiss rounds, 30 trees, dynamic word-limit formula) = 4 rounds; 30 trees; word limit formula not disclosed
Experimental design choices affecting ranking precision; no sensitivity analysis is provided.
Examiner model choice = Gemini-3-Pro
A single fixed judge model is used; no ablation across judge models, so the 0.94 correlation may depend on this model choice.

axioms (5)

domain assumption LMSYS Search Arena Elo scores are a valid, unbiased human ground truth.
Used as the target for all correlation claims, but its own variance and potential biases are not quantified (Section 4.2).
domain assumption The live web scrape plus LLM extraction produces factually correct Information Trees and rubrics.
Human audit found 89.1% rubric factuality (Table 7); Appendix G.2/G.3 demonstrate generation and judge errors.
domain assumption Depth and Width (from DeepWideSearch, Lan et al. 2025) are the two orthogonal dimensions sufficient to characterize deep-research-agent capability.
All tasks are generated along these two axes; the paper does not validate that this two-dimensional characterization covers the full space of DR-agent skills.
domain assumption Gemini-3-Pro as Examiner produces unbiased verdicts across all candidate-model families.
No judge-robustness ablation; the paper's Limitations section admits the Judge occasionally overrides correct rubrics using parametric knowledge.
standard math Bradley-Terry/Elo model is an appropriate aggregation of pairwise verdicts.
Standard ranking model, but its assumptions (stable strength, transitivity) are not tested on this small tournament.

invented entities (3)

Automated Examiner independent evidence
purpose: Generates tasks and rubrics from Information Trees, judges answers, diagnoses failures, and decides escalation.
The Examiner's verdicts are externally audited against human labels (Table 7) and the final ranking against LMSYS, providing a falsifiable check.
Dynamic Information Tree no independent evidence
purpose: Live-web-derived knowledge structure that provides the ground-truth content and task topology.
Factuality is checked only through the paper's human audit; no independent prediction or outside benchmark evaluates the tree itself.
Adaptive Evolvement Loop no independent evidence
purpose: State-machine controller that escalates depth/width based on the Examiner's verdicts until a decisive capability boundary emerges.
The efficiency claim (r=-0.61) is measured on the same data the loop produced; there is no external validation of the escalation mechanism.

pith-pipeline@v1.3.0-alltime-deepseek · 22244 in / 13722 out tokens · 151159 ms · 2026-08-03T10:16:08.164785+00:00 · methodology

0 comments

read the original abstract

As Large Language Models (LLMs) increasingly operate as Deep Research (DR) Agents capable of autonomous investigation and information synthesis, reliable evaluation of their task performance has become a critical bottleneck. Current benchmarks predominantly rely on static datasets, which suffer from several limitations: limited task generality, temporal misalignment, and data contamination. To address these, we introduce DR-Arena, a fully automated evaluation framework that pushes DR agents to their capability limits through dynamic investigation. DR-Arena constructs real-time Information Trees from fresh web trends to ensure the evaluation rubric is synchronized with the live world state, and employs an automated Examiner to generate structured tasks testing two orthogonal capabilities: Deep reasoning and Wide coverage. DR-Arena further adopts Adaptive Evolvement Loop, a state-machine controller that dynamically escalates task complexity based on real-time performance, demanding deeper deduction or wider aggregation until a decisive capability boundary emerges. Experiments with six advanced DR agents demonstrate that DR-Arena achieves a Spearman correlation of 0.94 with the LMSYS Search Arena leaderboard. This represents the state-of-the-art alignment with human preferences without any manual efforts, validating DR-Arena as a reliable alternative for costly human adjudication.

Figures

Figures reproduced from arXiv: 2601.10504 by Ruochen Zhao, Wenxuan Zhang, Yang Deng, Yiwen Gao.

**Figure 1.** Figure 1: Overview of the DR-Arena Framework. didate answers, and conducts follow-up rounds. To draft dynamic questions, the Examiner first constructs information trees by scraping informative websites in real-time. Based on the trees, the Examiner then devises a challenging question for the candidate DR agents. Since the Examiner has access to the ground-truth information source, it also keeps a mental set of gr… view at source ↗

**Figure 2.** Figure 2: Automated Task Generation via Dynamic Information Trees. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Leaderboard Correlation Analysis. While the overall alignment is high, a minor rank swap occurs between Perplexity-Sonar-Pro (#5 in DR-Arena) and Claude-Opus-4.1 (#6). As detailed in Appendix G, this divergence could be an example of the Factuality-Fluency Trade-off : Our qualitative analysis reveals that the automated Examiner strictly penalizes hallucinated numbers in fluent paragraphs, which human ann… view at source ↗

**Figure 4.** Figure 4: Evolution of Correlation across Rounds. intuition Judge can capture stylistic quality, it struggles to detect subtle hallucinations in synthesis. This demonstrates that Evidence-Based Adjudication is critical for reducing judge hallucination and ensuring rigorous alignment with human standards. The Evolvement Loop. We assess the impact of the multi-round dynamic investigation by cutting off investigation… view at source ↗

**Figure 6.** Figure 6: Per-Model Performance Profile. Models are [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Macro-level Evaluation Diagnostics. The left panel displays the distribution of verdicts across 789 unique rounds. The right panel shows the distribution of identified failure types for the losing agents. 1 2 3 4 5 6 8 Search Tree Depth 10 1 10 2 Count (Log Scale) 2 3 4 5 6 7 8 11 Width Constraint 10 0 10 1 10 2 Count (Log Scale) [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

**Figure 8.** Figure 8: Information Trees Topology Distribution. Histograms showing the Search Tree Depth and Width Constraints of the active nodes at the conclusion of all evaluation matches. Note the Log Scale on the Y-axis, indicating that while most tasks conclude at moderate complexity, the system is capable of scaling to high-complexity configurations. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent
cs.AI 2026-03 conditional novelty 5.0

A synthetic benchmark of 9,000 multi-hop web-research questions with difficulty tiers and teacher-generated search trajectories, plus an open-source RL training framework that reportedly lets 3B-parameter agents beat ...

Reference graph

Works this paper leans on

12 extracted references · cited by 1 Pith paper

[1]

Data Hallucination/Drift: Agent A cited the median pay for Managers as$169,510(likely 2023 data), differing from the strict 2024 con- text of$171,200retrieved by Agent B

2023
[2]

Database Architects,

Violating Negative Constraints: Agent A in- cluded “Database Architects,” which, while factually a high-paying role, violated the spe- cific grouping logic of the source text pro- vided in the context window. This case highlights that DR-Arena acts as a rig- orous “Factuality Auditor.” While humans may prioritize utility and style (favoring Claude), the E...

2005
[3]

RetroSearch

assesses multi-step research abilities with 100 PhD-level research tasks.LiveNewsBench (Anonymous, 2025) andLiveSearchBench(Zhou et al., 2025) focus on real-time information re- trieval from news and search results respectively. LiveResearchBench(Wang et al., 2025) evalu- ates citation-grounded long-form reports with 100 expert-curated tasks . Finally,Dee...

2025
[4]

Transition Logic:Did the Evolvement Loop correctly identify whether to deepen or widen the search based on the previous turn?
[5]

Deep & Wide

Stop Condition:Did the match conclude at an efficient point without redundant rounds? H.1 Annotation Instruments Below, we provide the exact questionnaires and criteria presented to the annotators for both the Task Generation Preference Study (Experiment I) and the Pipeline Validation Audit (Experiment II). Instrument I: Task Generation Preference.In this...

2013
[6]

Question Quality:Which generated ques- tion best necessitates multi-hop reasoning and multi-source synthesis?
[7]

• Dataset:We randomly sampled 30 full-match logs, comprising a total of 64 interaction turns

Rubric Accuracy:Which verification checklist best captures the ground truth con- straints without hallucination? Experiment II: Pipeline Validation Audit (Re- liability Study).This study corresponds to the results in Table 7. • Dataset:We randomly sampled 30 full-match logs, comprising a total of 64 interaction turns. • Task:Annotators were provided with ...
[8]

Deep & Wide

Question Validity:Does the question struc- turally adhere to the "Deep & Wide" defini- tion?
[9]

Rubric Factuality:Are the evaluation checkpoints supported by the source URLs?
[10]

Verdict Alignment:Do the annotators agree with the Examiner’s Win/Loss/Tie de- cision? (Used to calculate Cohen’s Kappa)
[2023]

Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, and Kaifu Zhang

Realtime QA: what’s the answer right now? InNeurIPS. Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2025. Deepwidesearch: Benchmarking depth and width in agentic information seeking.Preprint, arXiv:2510.20168. Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, and Tat-Seng Chua

arXiv 2025
[2025]

Deep & Wide

FACT-AUDIT: an adaptive multi-agent frame- work for dynamic fact-checking evaluation of large language models. InACL (1), pages 360–381. Asso- ciation for Computational Linguistics. 9 Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. InICLR. Open- Review.net. Mihran Miroya...

arXiv 2024

[1] [1]

Data Hallucination/Drift: Agent A cited the median pay for Managers as$169,510(likely 2023 data), differing from the strict 2024 con- text of$171,200retrieved by Agent B

2023

[2] [2]

Database Architects,

Violating Negative Constraints: Agent A in- cluded “Database Architects,” which, while factually a high-paying role, violated the spe- cific grouping logic of the source text pro- vided in the context window. This case highlights that DR-Arena acts as a rig- orous “Factuality Auditor.” While humans may prioritize utility and style (favoring Claude), the E...

2005

[3] [3]

RetroSearch

assesses multi-step research abilities with 100 PhD-level research tasks.LiveNewsBench (Anonymous, 2025) andLiveSearchBench(Zhou et al., 2025) focus on real-time information re- trieval from news and search results respectively. LiveResearchBench(Wang et al., 2025) evalu- ates citation-grounded long-form reports with 100 expert-curated tasks . Finally,Dee...

2025

[4] [4]

Transition Logic:Did the Evolvement Loop correctly identify whether to deepen or widen the search based on the previous turn?

[5] [5]

Deep & Wide

Stop Condition:Did the match conclude at an efficient point without redundant rounds? H.1 Annotation Instruments Below, we provide the exact questionnaires and criteria presented to the annotators for both the Task Generation Preference Study (Experiment I) and the Pipeline Validation Audit (Experiment II). Instrument I: Task Generation Preference.In this...

2013

[6] [6]

Question Quality:Which generated ques- tion best necessitates multi-hop reasoning and multi-source synthesis?

[7] [7]

• Dataset:We randomly sampled 30 full-match logs, comprising a total of 64 interaction turns

Rubric Accuracy:Which verification checklist best captures the ground truth con- straints without hallucination? Experiment II: Pipeline Validation Audit (Re- liability Study).This study corresponds to the results in Table 7. • Dataset:We randomly sampled 30 full-match logs, comprising a total of 64 interaction turns. • Task:Annotators were provided with ...

[8] [8]

Deep & Wide

Question Validity:Does the question struc- turally adhere to the "Deep & Wide" defini- tion?

[9] [9]

Rubric Factuality:Are the evaluation checkpoints supported by the source URLs?

[10] [10]

Verdict Alignment:Do the annotators agree with the Examiner’s Win/Loss/Tie de- cision? (Used to calculate Cohen’s Kappa)

[11] [2023]

Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, and Kaifu Zhang

Realtime QA: what’s the answer right now? InNeurIPS. Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2025. Deepwidesearch: Benchmarking depth and width in agentic information seeking.Preprint, arXiv:2510.20168. Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, and Tat-Seng Chua

arXiv 2025

[12] [2025]

Deep & Wide

FACT-AUDIT: an adaptive multi-agent frame- work for dynamic fact-checking evaluation of large language models. InACL (1), pages 360–381. Asso- ciation for Computational Linguistics. 9 Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. InICLR. Open- Review.net. Mihran Miroya...

arXiv 2024