{"id":"80a7e4a4-6a68-410e-8bbc-8ba6deb818b5","arxiv_id":"2601.06487","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ArenaRL trains LLM agents by ranking each batch of candidate trajectories through a seeded single-elimination tournament instead of assigning individual scalar scores, claiming better open-ended task performance at O(N) comparison cost.","lead":"ArenaRL replaces per-answer scalar rewards with pairwise tournament comparisons during RL training of LLM agents, and reports large win-rate gains over standard RL baselines on new travel-planning and deep-research benchmarks. The catch: the main results are evaluated with the same family of LLM judges used to train the model, and the headline O(N)-vs-O(N^2) ranking-accuracy claim is never directly measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Seeded single-elimination's O(N) accuracy claim rests on the untested assumption that greedy decoding provides a low-bias quality anchor; the paper never measures ranking fidelity directly, so the close Table 2 win rates don't establish the claimed equivalence.","rationale":"The reader's weakest assumption identifies the greedy anchor as the critical dependency, and I agree. The paper's Table 2 provides only indirect downstream evidence; the anchor is never validated directly. If the anchor assumption fails, the O(N) ranking no longer approximates round-robin, which would invalidate the abstract's main complexity/accuracy claim. The proposed test directly measures ranking fidelity and anchor quality, and would settle the issue. Therefore the CONDITIONAL verdict remains appropriate.","tokens_in":30150,"tokens_out":6680,"duration_ms":61462,"concrete_test":"On 100 groups from Open-Travel (N=16) and 100 from Open-DeepResearch (N=8), compute full round-robin scores with the same Arena Judge. Run SSE (Algorithm 1) and random-seeded single-elimination on the same groups. Report Kendall's tau and top-2 overlap between each tournament ranking and the round-robin ranking, with bootstrap CIs. Separately, compute the correlation between the greedy anchor's average pairwise score and each trajectory's round-robin score; compare against using a randomly sampled anchor. If SSE's tau is not close to 1 (e.g., <0.9) or not clearly better than random seeding, or if the greedy anchor's score correlation is low (<0.5), the central equivalence claim fails; otherwise the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that seeded single-elimination (SSE) achieves 'nearly equivalent advantage estimation accuracy' to round-robin at O(N) cost rests on an untested premise: in Sec 4.3, the greedy-decoding trajectory is assumed to be a 'quality anchor' providing a 'low-biased initial estimate' for seeding. If the greedy trajectory is degenerate, stylistically atypical, or systematically weaker than exploratory samples, the seed order is wrong, top trajectories can eliminate each other early, and the SSE ranking diverges from the round-robin ranking. The paper never measures anchor quality: no distribution of anchor scores, no correlation between anchor-based seed scores and round-robin scores, no ablation with alternative anchors (random, median, or a second greedy run). Moreover, the only evidence for the equivalence is Table 2, which compares downstream RL win rates (SSE 32.5 vs round-robin 32.9) on a single benchmark. That conflates ranking fidelity with judge noise, exploration, group size, and the RL update; it does not directly establish that SSE recovers the round-robin advantage estimates. Thus the O(N)/O(N^2) equivalence — a headline contribution — is currently unsupported, and the anchor assumption is the most fragile link in that argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ArenaRL, a reinforcement-learning framework for open-ended LLM agents that replaces pointwise scalar rewards with an intra-group relative ranking derived from pairwise comparisons. The authors argue that pointwise LLM-judge scores suffer from a 'discrimination collapse' (low signal-to-noise ratio) and introduce a process-aware pairwise judge with bidirectional scoring, plus several tournament topologies to aggregate comparisons. The main algorithmic contribution is a 'seeded single-elimination' (SSE) scheme that uses a greedy-decoding trajectory as an anchor for seeding, claiming O(N) cost with advantage-estimation accuracy close to the O(N^2) round-robin benchmark. The paper also introduces two benchmarks, Open-Travel and Open-DeepResearch, with SFT/RL pipelines, and reports experiments showing ArenaRL outperforming GRPO, GSPO, and several closed-source models on travel planning, deep research, and open-ended writing.","tokens_in":30449,"tokens_out":6139,"duration_ms":58772,"significance":"If the SSE claim were established, ArenaRL would provide a practical answer to a real bottleneck in preference-based RL for open-ended tasks: extracting stable intra-group rankings at linear comparison cost. The paper also contributes two full-cycle open-ended agent benchmarks, a detailed rubric-based pairwise evaluation protocol, and a human-consistency check, all of which are useful community assets. These strengths are real and should be credited. However, the central claim of 'nearly equivalent advantage estimation accuracy' is currently supported only by end-to-end win rates under a partly circular judge protocol; direct ranking-fidelity evidence is missing. The idea is promising, but the manuscript does not yet demonstrate the headline result as rigorously as claimed.","major_comments":[{"comment":"The headline claim that seeded single-elimination (SSE) achieves 'nearly equivalent advantage estimation accuracy' to round-robin is never tested directly. Table 2 reports end-to-end RL win rates after full training; this conflates ranking fidelity with exploration, group size, judge noise, and the policy update. There is no correlation measure between SSE rankings and round-robin rankings on the same trajectory groups, no ranking-error metric, and no ablation isolating the tournament topology. As written, the O(N) equivalence is a plausible conjecture, not an established result.","section":"Abstract; §6.2, Table 2"},{"comment":"The 'quality anchor' assumption is load-bearing. The paper asserts that the greedy-decoding trajectory provides a 'low-biased initial estimate' for seeding, but offers no empirical support: no distribution of anchor scores, no correlation between anchor-based seed scores and round-robin win rates, and no ablation with random or simple-alternative seeding (e.g., median score, second greedy run). If greedy decoding produces a degenerate or stylistically atypical trajectory, seeds are misordered and the SSE ranking can diverge from the round-robin ranking. This needs to be measured or, at minimum, ablated.","section":"§4.3, Phase 1"},{"comment":"The evaluation protocol is partly circular. Qwen3-Max is used as the arena judge during RL training (Appendix A) and is also one of the two final evaluation judges (§5.3). Reported win rates may therefore reward alignment to Qwen3-Max's preferences rather than general output quality. The 73.9% human-agreement check is encouraging, but the manuscript gives no sample size, rater details, or per-method agreement, and it does not rule out a judge-specific advantage. Please report results with a judge family not seen during training, or provide a human evaluation on a representative, documented sample.","section":"Appendix A; §5.3, §6.4"},{"comment":"Tournament topology selection is performed at group size N=8 (§6.2), but the final Open-Travel and open-ended-writing runs use N=16, and Open-DeepResearch uses N=8 with K=4 (Appendix A). Table 2's conclusion that SSE 'achieves' round-robin performance is therefore established only at N=8; no evidence shows that the equivalence transfers to the deployed N=16. Moreover, no results carry error bars or significance tests, so the 32.5 vs. 32.9 difference may be within noise. At minimum, the paper should state the actual training configuration used for each table and justify extrapolating the topology choice from N=8.","section":"§6.2 vs. Appendix A"}],"minor_comments":[{"comment":"'Swiss Round' should be 'Swiss-System' to match §4.5 and Figure 2.","section":"§6.2"},{"comment":"The text refers to 'Figure 1(c)', but the displayed figure has only panels (a) and (b). Please correct the citation or add the missing panel.","section":"Introduction"},{"comment":"The phrase 'denoted as (defined as the 1-Day subtask)' contains a duplicated parenthetical; please fix.","section":"§5.1"},{"comment":"The evaluation judges are described as 'Qwen3-Max and Claude-4-Sonnet Anthropic (2023)', but the reference list only contains a generic Claude (2023) entry. A specific citation for Claude-4-Sonnet is needed.","section":"Appendix A"},{"comment":"The bracket-array construction (lines 11–21) is not explained in the main text. A sentence explaining why seeds are assigned alternately from front and back would improve readability and reproducibility.","section":"Algorithm 1"},{"comment":"The repository link is given as https://github.com/Alibaba-NLP/qqr, which appears unrelated to ArenaRL. Please verify that the URL points to the correct code release.","section":"Code availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-written technical report from an industry lab, with useful benchmark contributions and an interesting algorithmic idea. My main concern is evidential: the strongest claim (SSE ≈ round-robin at O(N) cost) is not directly measured, and the anchor assumption is untested. These can likely be addressed with additional experiments within the manuscript's scope, so I recommend major revision rather than rejection. I would also suggest the editor ask the authors to clarify the judge-circularity issue, as it affects how the main results should be interpreted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the seeded single-elimination scheme is a genuinely nice idea, and the paper is honest about the trade-offs between tournament topologies. But the claim that it achieves \"nearly equivalent advantage estimation accuracy\" to round-robin is not actually measured. Table 2 shows final policy win rates (32.5 vs 32.9) on one benchmark, with no error bars or significance tests. That is suggestive, not conclusive.\n\nWhat's new: the tournament topology comparison is the real contribution. Five topologies, including round-robin as a gold standard, are compared under a unified RL setup. Anchor-based seeding to prevent premature elimination of strong candidates is a sensible fix, and the downstream results point in the right direction. The two benchmarks, Open-Travel and Open-DeepResearch, with SFT→RL→evaluation pipelines, fill a genuine gap even if the test sets are small (250 and 100).\n\nWhere it's soft:\n\n1. The anchor assumption. The seeding argument rests on greedy decoding giving a low-bias reference. The paper never checks this. If the greedy trajectory is not a reliable anchor, seeds are misordered and the O(N) ranking can diverge from round-robin. This needs an ablation: random seeding, median anchor, correlation of seed scores with round-robin scores.\n\n2. Judge contamination. Qwen3-Max is the arena judge during RL and one of the two evaluation judges. Some of the gains may reflect overfitting to that judge's preferences. The 73.9% human agreement is partial relief, but a held-out judge family that never appears in training would be far more convincing.\n\n3. Reproducibility. The code link points to a repo called 'qqr', which doesn't match the paper. No commit hash, no data release. For a paper whose main claims are empirical, that matters.\n\nThe stress-test note's concern about the anchor is fair. It doesn't kill the paper—the downstream win rates are close to round-robin—but it does mean the headline complexity/accuracy equivalence is currently an assertion, not a demonstrated result.\n\nWho this is for: people working on reward modeling and RL for open-ended LLM agents. It deserves a serious referee: the idea is worth testing, and the topology comparison is worth having in the literature. I'd send it out, with the request that the authors add direct ranking-fidelity experiments, a held-out judge, and error bars.\n\nFor peer review: accept with major revision.","headline":"Seeded single-elimination is a clever, well-motivated tournament design, but the headline O(N) accuracy equivalence is asserted rather than directly measured; worth refereeing, not accepting on these experiments alone.","tokens_in":31001,"tokens_out":2400,"would_cite":true,"duration_ms":23644,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-ended agent RL fails because pointwise judge scores collapse within high-quality trajectory groups; ArenaRL replaces them with seeded single-elimination rankings, reporting near-round-robin accuracy at linear cost and large task gains.","keywords":["reinforcement learning","LLM agents","open-ended tasks","relative ranking","tournament ranking","reward collapse","pairwise evaluation","LLM-as-judge"],"falsifier":"Take held-out trajectory groups, compute the seeded single-elimination ranking and the exhaustive round-robin ranking, and measure their agreement (e.g., Kendall tau). Then corrupt or replace the greedy anchor—random seeds, worst-quality seeds, or seeds from a different decoding temperature—and repeat. The central claim is falsified if anchor-based seeding does not robustly beat random seeding in agreement with round-robin across trajectory distributions, or if there is any natural distribution where the O(N) ranking drops sharply below round-robin accuracy.","tokens_in":30017,"feed_emoji":"🏆","tokens_out":7423,"duration_ms":78815,"temperature":0.7,"pith_summary":"Open-ended tasks such as travel planning and deep research have no verifiable ground truth, so RL for LLM agents must get reward from an LLM judge. ArenaRL's central contention is that pointwise scalar scoring suffers from discrimination collapse: once a policy improves, trajectories in the same group look near-identical to the judge, and the differences that remain are smaller than the judge's own noise. The paper replaces scalar scores with relative rankings over each trajectory group, computed by a process-aware pairwise judge and organised as a seeded single-elimination tournament. It reports that this tournament needs only O(N) comparisons while matching the advantage-estimation accuracy of an O(N^2) round-robin, and that agents trained this way substantially outperform standard RL baselines on travel planning, deep research, and open-ended writing. The paper also introduces two full-pipeline benchmarks, Open-Travel and Open-DeepResearch, so the SFT-to-RL-to-evaluation loop can be reproduced.","feed_headline":"O(N) tournament ranking rivals O(N^2) pairwise comparisons","feed_subtitle":"Swapping scalar LLM-judge scores for seeded tournament rankings lifts win rates across travel, research, and writing.","key_machinery":"The central object is the seeded single-elimination tournament over a group of N trajectories sampled from the current policy. A greedy-decoding trajectory acts as a 'quality anchor': each exploratory trajectory is first compared with it, producing a seed ordering at low bias. The bracket then pairs the strongest seed against the weakest seed in a binary elimination tree, and each match is decided by the process-aware pairwise judge, which scores both trajectories under a rubric covering chain-of-thought coherence, tool-call effectiveness, and answer reliability in both presentation orders to cancel positional bias. Final rank comes from survival depth, with ties within a round broken by acc","core_discovery":"The core claim is that the reward format, not the absence of a reward model, is what stalls RL in open-ended domains. ArenaRL formalises discrimination collapse—within-group pointwise scores compress toward a narrow band whose width is comparable to judge noise—and argues that normalisation tricks inherit that noise. The fix is intra-group relative ranking: a bidirectional, rubric-guided judge compares trajectories on reasoning, tool use, and final answer, and a tournament converts the comparisons into a stable ordering. The paper's key empirical assertion is that a seeded single-elimination design—using a greedy-decoding trajectory as an anchor to seed the bracket—gives advantage estimates","pith_inferences":["Because the greedy anchor is never refreshed, a natural extension is to make the anchor adaptive—re-seeding from the current policy's best trajectory each training round—so the bracket stays calibrated as the policy outgrows greedy decoding.","The tournament idea transfers beyond agent RL: any expensive pairwise-preference setting (reward-model training, human evaluation, model selection) could use a cheap prior-seeded tournament to approximate full round-robin rankings.","The reported 73.9% human-judge agreement suggests a calibration protocol: if tournament rankings from two independent judge families disagree, the ranking signal itself is suspect; this could be monitored as a training-time diagnostic.","The paper's group-size ablation shows monotonic improvement up to N=16 but does not identify a saturation point; testing larger N would reveal whether the exploration benefit continues or whether judge and compute costs eventually dominate."],"forward_implications":["If discrimination collapse is the binding constraint, pointwise-reward RL will plateau once judge noise covers within-group quality differences; the reported gains are direct evidence for that failure mode.","Pairwise comparison becomes affordable in online training: O(N) advantage estimation removes the quadratic bottleneck that made full preference ranking impractical for long-context agents.","The mechanism transfers across domains—tool-using travel and research agents and ordinary open-ended writing—so relative ranking is a general fix for open-ended reward acquisition, not a benchmark-specific patch.","ArenaRL's reported ability to train directly from a generic base model without SFT cold-start implies tournament ranking can bootstrap tool-use behavior where annotated trajectories are scarce.","Open-Travel and Open-DeepResearch provide a full SFT-Training-Evaluation pipeline, letting later work compare complete agent training loops instead of static test sets."],"fun_headline_variants":["Seeded tournaments give O(N) RL advantage estimates","From pointwise scores to relative ranks: ArenaRL stabilizes RL","Open-ended RL fixed by tournament-based ranking","O(N) tournament ranking matches O(N^2) pairwise in RL","Relative ranking, not scalar scores, unlocks open-ended RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire O(N)-with-round-robin-accuracy claim rests on the greedy-decoding trajectory being a reliable, low-bias quality anchor; if it is degenerate or unrepresentative, the seed ordering is wrong, strong trajectories can eliminate each other early, and the tournament ranking stops approximating the full pairwise ranking.","fun_headline_variants_meta":{"raw":{"variants":["Seeded tournaments give O(N) RL advantage estimates","From pointwise scores to relative ranks: ArenaRL stabilizes RL","Open-ended RL fixed by tournament-based ranking","O(N) tournament ranking matches O(N^2) pairwise in RL","Relative ranking, not scalar scores, unlocks open-ended RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2175,"prompt_tokens":819,"completion_tokens":1356,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1288}},"tokens_in":563,"tokens_out":1356,"duration_ms":11046,"temperature":1.0,"reasoning_tokens":1288,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:22:33.080157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take held-out trajectory groups, compute the seeded single-elimination ranking and the exhaustive round-robin ranking, and measure their agreement (e.g., Kendall tau). Then corrupt or replace the greedy anchor—random seeds, worst-quality seeds, or seeds from a different decoding temperature—and repeat. The central claim is falsified if anchor-based seeding does not robustly beat random seeding in agreement with round-robin across trajectory distributions, or if there is any natural distribution where the O(N) ranking drops sharply below round-robin accuracy.","supporting_citations":[],"review_version":1}