{"id":"de04cb2d-ca92-473d-aa83-d49b11ce96e5","arxiv_id":"2505.12808","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fully automatic LLM evaluation framework where all evaluated models serve as judges for one another reaches 97% Spearman correlation with human preference rankings while keeping cost sub-quadratic.","lead":"This paper introduces Decentralized Arena, a system in which every participating language model judges every other model on open-ended questions, replacing a single 'expert judge' with a crowd of models. The authors report that the resulting rankings match human preference rankings from Chatbot Arena up to 97% of the time, at lower cost than human annotation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The advertised sub-quadratic complexity is not established: Section 3.1's binary search uses all existing models as judges for each comparison, yielding Θ(k n^2 log n) total judge calls rather than O(k n log n).","rationale":"The reader identified the Elo-weighted judge loop as the weakest assumption. That is a real concern, but it is not the most load-bearing issue for the central claim: Table 7 shows that removing Elo weights lowers correlation only from 0.957 to 0.949 on MT-Bench and from 0.959 to 0.953 on Math, so the headline empirical ranking does not collapse on that issue. The stronger problem is the unsubstantiated scalability claim, which is a direct mathematical consequence of the algorithm as written in Section 3.1. Since every binary-search comparison uses all existing models as judges, the total number of judge calls is Θ(k n^2 log n), not sub-quadratic. This is checkable without any benchmark data and does not depend on tuning or benchmark selection. The empirical result could still be valid, but the paper must either correct the complexity claim to 'total judge calls are O(k n^2 log n), a large reduction from the O(n^3 k) naive method', or modify the algorithm to use a fixed-size judge committee to achieve genuine sub-quadratic cost. The reader's verdict of CONDITIONAL remains appropriate; the concern adds a specific technical condition that should be addressed during revision.","tokens_in":20995,"tokens_out":12628,"duration_ms":139946,"concrete_test":"Instrument the released implementation to count every judge call (judge model × question × pairwise comparison) for n = 32, 64, 128, 256 models at fixed k, and fit the growth. If total judge calls scale as c n^2 log n rather than c n log n or c n^2, the sub-quadratic claim in the abstract is false. Independently, write the recurrence T(n) = T(n-1) + c k n log n, whose closed form is Θ(k n^2 log n), and check it against the algorithm in Section 3.1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and introduction claim that De-Arena ranks LLMs with sub-quadratic complexity, but the algorithm described in Section 3.1 does not support that claim. In the coarse-grained binary search, the new model is compared against the midpoint model using 'all other models in the list' as judges over k questions. Thus inserting the t-th model costs O(t k log t) judge calls for the binary search plus O(t k) for the fine-grained window reranking. Summing over t = 6..n gives Θ(k n^2 log n) total judge calls, which is not sub-quadratic in n. The O(k n log n) statement in the paper is at best the cost of a single insertion if one counts every judge individually; it is not the total cost of the full ranking algorithm. If 'sub-quadratic' refers instead to the number of model-pair comparisons, that is O(n log n), but the paper's own cost metric in Figure 2b is 'average judge counts', and an evaluation pipeline's cost must count judge calls. Therefore a headline contribution, scalable sub-quadratic automatic evaluation, rests on an incorrect or at least unverified complexity analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes De-Arena, a fully automatic LLM evaluation framework in which every model under evaluation also serves as a judge. The method combines coarse-to-fine incremental ranking (binary search followed by local in-window reranking), Elo-based score aggregation with adaptively weighted judges, and automatic representative question selection. The authors evaluate De-Arena on 66 LLMs across nine fine-grained dimensions and report Spearman correlations with the human-judged Chatbot Arena leaderboard up to 0.974 overall, outperforming 14 compared benchmarks including MixEval, WildBench, and Auto Arena. Additional experiments examine single-judge bias, group bias, insertion-order stability, question selection, style control, and cost.","tokens_in":21216,"tokens_out":6122,"duration_ms":70484,"significance":"If the reported correlations are robust, De-Arena is a significant contribution to automatic LLM evaluation: it replaces one or a few authority judges with collective voting by all evaluated models, thereby addressing a recognized source of judge bias, and it includes an automatic question-selection mechanism for building new evaluation dimensions. The paper's strengths include a broad comparison against 14 benchmarks, experiments with 66 models, nine fine-grained dimensions, stability studies under shuffled insertion order, and the promise of public code and data. The empirical comparisons and ablations are useful regardless of the theoretical framing. However, the central scalability claim and the independence of the reported headline correlations both need additional scrutiny before the contribution can be accepted at face value.","major_comments":[{"comment":"The claimed sub-quadratic complexity is not supported by the described algorithm. Section 3.1 states that the coarse-grained binary search has time complexity O(kn log n), and the abstract advertises \"sub-quadratic complexity\" and efficiency at scale. In the algorithm, each binary-search comparison uses all other models in the list as judges over k questions, so one insertion costs Θ(k t log t) judge calls for the binary search plus additional Θ(k t) judge calls for the fine-grained window reranking. Summing over t = 6..n insertions gives Θ(k n^2 log n) total judge calls. Since Figure 2b measures cost in \"average judge counts,\" judge calls are the natural cost metric, and the total is not sub-quadratic. If the intended claim is only about the number of model-pair comparisons, that count is O(n log n), but the paper's own cost metric counts judge calls. The manuscript should state the per-insertion complexity explicitly and either correct or qualify the total-complexity claim; as written, the scalability headline is unsupported.","section":"Abstract; §3.1; Figure 2b"},{"comment":"The key hyperparameters (window size and base model number) are selected by maximizing the same Chatbot Arena Spearman correlation that is reported as the main result in Table 2. For example, Table 10 chooses base model number 6 because it yields the highest MT-Bench and Math correlations, and Table 9 chooses window size 1 for the same reason. This is a form of selection on the evaluation target, so the reported correlations (including 0.957, 0.962, and the 0.974 in Table 2) are optimistic estimates of the method's agreement with human judgments. The authors should provide a validation split, report performance across a range of hyperparameter choices instead of only the selected maximum, or otherwise show that the headline numbers are not artifacts of this tuning.","section":"Appendix B, Tables 9 and 10"},{"comment":"The judge-weighting scheme creates an unexamined feedback loop. The paper assigns each judge a weight proportional to its normalized Elo score and updates both the Elo scores and the weights iteratively. Because the Elo scores are themselves estimated from judge-weighted comparisons, this is a fixed-point iteration, but the paper offers no convergence, uniqueness, or bias analysis. A plausible failure mode is that a model with inflated Elo (for example, due to self-voting or family bias) receives higher weight and thereby further inflates its own Elo. Table 7 only compares final correlations with and without weights; it does not test whether the iterative process converges or whether the resulting fixed point is stable. The authors should report convergence diagnostics (e.g., trajectories of weights and Elo scores across iterations) or otherwise address the stability and fairness of this loop.","section":"§3.1, Score Generation and Style Control; Table 7"}],"minor_comments":[{"comment":"The phrase \"across extensive experiments across 66 LLMs\" contains a duplicated preposition; rephrase for clarity.","section":"Abstract"},{"comment":"The sentence \"The time complexity of this binary search is O(kn log n)\" is ambiguous: it should specify whether this is the cost per insertion or the total cost over all insertions, since the surrounding text compares it with the total O(n^3 k) cost of full pairwise evaluation.","section":"§3.1"},{"comment":"The text says the single-judge variants include GPT-4o-2024-08-06, but Table 3 lists only LLaMA-3-70B, Gemma-2-27B, Qwen2-70B, and De-Arena; the table should either include GPT-4o or the text should be corrected.","section":"§4, Table 3"},{"comment":"The cost metric \"number of judges\" is not precisely defined. Please clarify whether it counts pairwise votes, judge calls, or generated tokens, since this affects how the cost comparison in Figure 2b should be interpreted.","section":"§4.2, Table 4 and Figure 2b"},{"comment":"The hyperparameter choices differ by very small correlation margins (e.g., 0.957 vs. 0.955 for window size 3 in MT-Bench), yet no confidence intervals or significance tests are reported; the authors should indicate whether these differences are meaningful.","section":"Appendix B, Tables 9 and 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is a preprint under review. The complexity issue is a factual error in a headline claim and must be fixed, but it is fixable by clarifying that the per-insertion complexity is O(kn log n) and the total judge-call complexity is O(k n^2 log n) (still a large improvement over O(n^3 k)). The hyperparameter-selection issue is common in leaderboard-driven work, but should be addressed explicitly. The judge-weighting convergence concern is a legitimate correctness risk that the experiments do not currently rule out. I would not reject the paper on these grounds, but I would require a revised version that addresses all three points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea—letting every evaluated LLM serve as a judge in pairwise comparisons, with incremental insertion via a coarse-to-fine ranking algorithm—is genuinely interesting, and the experiments are the most extensive I've seen for this kind of approach: 66 models, 9 dimensions, and a 0.974 Spearman correlation with Chatbot Arena overall. The question-selection strategy based on ranking consistency is a nice addition and appears to beat random selection and other baselines. The paper is also unusually honest in reporting ablations and stability studies across insertion orders.\n\nThe soft spots are in the efficiency analysis, not the experimental protocol. The abstract and introduction claim sub-quadratic complexity, but the algorithm as described in Section 3.1 counts judge calls: when inserting the t-th model, binary search uses all t−1 existing models as judges on k questions, so one insertion costs Θ(tk log t) judge calls, and summing to n gives Θ(k n² log n). That is not sub-quadratic in n unless you count only model-pair comparisons, which is not the paper's own cost metric in Figure 2b. This is a load-bearing error in a headline claim; it needs to be fixed or reframed. The empirical cost at 66 models may still be reasonable, but the asymptotic story is wrong.\n\nTwo more concerns, related to the reader's report. The hyperparameters (window size=1, base models=6) were selected by maximizing the same Chatbot Arena correlation that is the evaluation target, so the reported numbers are optimistic; a held-out human-judgment set or a robustness sweep would help. And the Elo-weighted judge loop, where judge weights are derived from the very scores those judges produce, needs at least an empirical convergence analysis; right now it's an unexamined fixed point.\n\nNone of this is fatal. The correlation result is external and the method is well-specified. I would send this to peer review, with the clear expectation that the authors fix the complexity analysis, discuss the tuning leakage, and add a convergence check. It is the kind of paper the evaluation community will want to engage with, and the all-models-as-judges setting is worth taking seriously.","headline":"All-models-as-judges evaluation is a promising idea with a strong headline correlation, but the sub-quadratic complexity claim is wrong as stated and the hyperparameter tuning leaks the target.","tokens_in":21802,"tokens_out":2676,"would_cite":false,"duration_ms":24991,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using every LLM as a judge can beat single-judge benchmarks at ranking models by human preference.","keywords":["decentralized evaluation","LLM-as-a-judge","collective intelligence","Elo rating","pairwise comparison","coarse-to-fine ranking","representative question selection","Chatbot Arena correlation"],"falsifier":"Run De-Arena on the same 66 models with different initial Elo values or with judge weights frozen at their first iteration; if the final rankings shift materially, for example by more than a few adjacent swaps among close models, the claim that the weighting loop reliably improves human agreement is falsified. The paper's insertion-order shuffles show low variance, but they do not test initialization sensitivity.","tokens_in":20768,"feed_emoji":"⚖️","tokens_out":5699,"duration_ms":55007,"temperature":0.7,"pith_summary":"The paper tries to establish that language-model evaluation can be made both more human-aligned and more scalable by letting every evaluated model serve as a judge. Instead of relying on one strong judge or a small committee, De-Arena runs democratic pairwise comparisons in which the full pool of models votes, weights each judge's vote by its own estimated skill, and inserts new models via binary search plus local reranking. On 66 LLMs this automated pipeline reaches a 0.974 Spearman correlation with the human-judged Chatbot Arena overall ranking, higher than any benchmark it compares against, while keeping comparison counts sub-quadratic. If the claim holds, model creators could obtain trustworthy, fine-grained leaderboards for thousands of models and new dimensions without expensive human annotation or the biases of a single authority judge.","feed_headline":"All 66 LLMs judge each other, beating existing benchmarks","feed_subtitle":"Democratic pairwise voting reaches 97% agreement with human preferences at lower cost.","key_machinery":"The load-bearing mechanism is a three-part pipeline. First, pairwise comparison: every model in the pool votes on which of two outputs wins, and a model never judges its own output. Second, coarse-to-fine incremental ranking: a new model is inserted by binary search against the current ranking, then re-ranked inside a small sliding window of neighbours, which concentrates judge effort on ambiguous pairs and keeps complexity near $O(k n \\log n)$ rather than $O(n^3 k)$. Third, Elo score generation with judge weighting and style control: after comparisons, Elo scores are fit by logistic regression, each judge's normalized Elo is used as its weight in the loss, and output styles (length, headers, lists, bold) are controlled to reduce style-driven bias. A separate ranking-based question-selection step picks the questions whose per-question rankings most agree with the average ranking, so new evaluation dimensions can be built automatically.","core_discovery":"On the paper's own terms, the central discovery is that the 'wisdom of the crowd' transfers to LLM evaluation: a decentralized system where all evaluated models judge each other produces rankings closer to human preference than single-judge or small-committee automated benchmarks, and it does so without exhaustively comparing every pair. The paper reports Spearman correlations with Chatbot Arena of 0.957, 0.967, and 0.974 for test sets of 15, 30, and 66 LLMs in the overall dimension, with correlation improving as the judge pool grows. It further reports that weighting judge votes by the judges' own Elo ratings adds a small but consistent gain (for example, 0.957 versus 0.949 on MT-Bench), and that its question-selection method outperforms random, perplexity-based, anchor-point, and single-strong-judge selection.","pith_inferences":["A natural extension the paper does not pursue is using the same ranking pipeline to build arbitrary new dimensions from scratch: any question collection could be standardized to open-ended format and scored by ranking consistency, so the barrier to creating a bespoke leaderboard drops to dataset collection.","If judge weighting is doing real work, then a testable consequence is that the method's advantage over unweighted voting should grow as the pool includes larger ability gaps; the paper's reported gap is small, so this may be a modest effect.","The democratic design also suggests a governance interpretation: rankings become a fixed point of a social-choice process rather than a decree of one model, which could be probed by checking whether excluding any single judge materially changes the ranking."],"forward_implications":["Automatic leaderboards can be updated incrementally: adding a new model costs comparisons proportional to the log of the list size plus a small window, rather than a full re-ranking of all pairs.","Fine-grained dimensions such as algebra, geometry, probability, social reasoning, chemistry, biology, physics, and general chat can be constructed automatically from open-source question collections, removing the bottleneck of human-curated data.","As more models participate, agreement with human preference should continue to rise, since the paper observes monotone improvement as the judge pool grows from 8 to 16 to 26 models.","Because judge weights come from the models' own Elo scores, the method adapts to a growing pool without needing a fixed authority judge, which avoids the self-preference bias seen when a single model judges itself and its relatives."],"supporting_citations":[{"why":"Supplies the human-preference leaderboard used as the ground truth for all Spearman-correlation comparisons.","marker":"[8]"},{"why":"Documents single-judge self-enhancement and verbosity biases that motivate replacing one authority judge with a crowd.","marker":"[17]"},{"why":"Provides the AlpacaEval single-judge baseline that De-Arena is compared against.","marker":"[15]"},{"why":"Provides the PRD peer-ranking multi-judge baseline that the democratic scheme extends.","marker":"[36]"},{"why":"Provides the Auto Arena multi-judge committee baseline and the five-judge committee design that De-Arena generalizes to all models.","marker":"[37]"},{"why":"Supplies the Elo rating system used to convert pairwise wins into a final ranking.","marker":"[43]"},{"why":"Supplies the binary search algorithm that gives the incremental insertion its logarithmic complexity.","marker":"[42]"},{"why":"Provides the WildBench real-user-task benchmark used as a comparison baseline.","marker":"[52]"},{"why":"Provides MixEval, a strong benchmark baseline whose ranking-consistency idea the question-selection step adapts.","marker":"[49]"}],"fun_headline_variants":["66 LLMs judge each other: 97% human agreement","Democratic AI evaluation: 66 judges, 97% human match","Collective LLM judgment beats single-model bias","Lower-cost, scalable AI benchmarking via peer vote","LLM self-evaluation with human-level correlation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the assumption that the iterative loop of re-estimating Elo scores and re-weighting each judge by its own Elo converges to a stable, unbiased ranking; the paper gives no proof of convergence or uniqueness for this fixed point.","fun_headline_variants_meta":{"raw":{"variants":["66 LLMs judge each other: 97% human agreement","Democratic AI evaluation: 66 judges, 97% human match","Collective LLM judgment beats single-model bias","Lower-cost, scalable AI benchmarking via peer vote","LLM self-evaluation with human-level correlation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3065,"prompt_tokens":949,"completion_tokens":2116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2038}},"tokens_in":565,"tokens_out":2116,"duration_ms":15713,"temperature":1.0,"reasoning_tokens":2038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:34.734836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run De-Arena on the same 66 models with different initial Elo values or with judge weights frozen at their first iteration; if the final rankings shift materially, for example by more than a few adjacent swaps among close models, the claim that the weighting loop reliably improves human agreement is falsified. The paper's insertion-order shuffles show low variance, but they do not test initialization sensitivity.","supporting_citations":[{"cited_title":"Binary search algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the binary search algorithm that gives the incremental insertion its logarithmic complexity."}],"review_version":1}