{"id":"b43f7fcc-8a01-4c0f-829f-17fa4d09d938","arxiv_id":"2412.06745","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ONEBench treats each benchmark sample as a voter in a Plackett-Luce aggregation, enabling open-ended, capability-specific, and incomplete-data model rankings.","lead":"ONEBench is a new benchmarking system that pools samples from many AI benchmarks and ranks models by aggregating per-sample \"votes\", allowing users to build custom tests for specific capabilities. It claims accurate rankings even when most measurements are missing, potentially reducing evaluation cost by up to 20 times.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PL identifiability is only established for PL-generated data; real-data validation uses a mean-score ground truth derived from the same measurements, leaving the central recovery claim untested under misspecification.","rationale":"I agree with the reader's identification of the PL generative assumption as the weakest link. The concern is not that the aggregation method is internally inconsistent; the PL estimator is well-defined and the code release and synthetic experiments are useful. The issue is evidential: the only synthetic guarantee is generated from the assumed model, and the real-world comparison uses a ground truth that is a function of the same scores used to form the rankings, so it does not independently confirm that PL recovers an underlying ranking when the data are not PL. A non-PL simulation is a direct, low-cost check that would either clear the concern or force a qualified claim. Since this is an addressable empirical gap rather than a demonstrated failure, the reader's conditional verdict remains appropriate.","tokens_in":37180,"tokens_out":6289,"duration_ms":73276,"concrete_test":"Re-run the Appendix D synthetic protocol with a non-PL noise model: set latent qualities β_i, draw per-sample scores s_{ij} = β_i + ε_{ij} with ε_{ij} ~ N(0,1) (a Thurstone–Mosteller model), convert to sample rankings, apply the paper's PL rank-breaking and tie handling, and compute Kendall-τ between the MLE ranking and the true β ranking at the same missing-data fractions (0%, 50%, 90%, 95%, 99%). If Kendall-τ falls materially below the reported 0.9+ values in any realistic dispersion regime, the identifiability claim is misspecification-dependent and the headline recovery guarantee needs to be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1's identifiability guarantee (Property 1) is stated for the Plackett–Luce generative model: if the data are PL, MLE asymptotically recovers the utility scores. Appendix D tests exactly this case by sampling Gumbel scores, which is the PL noise model, so it cannot reveal what happens when real evaluations are generated differently. On the real benchmarks, Section 3.2.1 defines the ground truth as the mean of min-max-normalized per-sample scores, and the sample-level rankings fed to PL are built from those same scores. The reported Kendall-τ therefore measures agreement between two summaries of the same input, not recovery of an independent latent quality. The paper's Limitations section explicitly concedes that \"statistical modeling assumptions ... may not always hold\", but no experiment quantifies the effect of that concession. Because the abstract's promise is \"asymptotically recovering ground-truth scores\", the load-bearing premise is that real measurements are approximately PL or that PL-MLE is rank-robust under misspecification; neither is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ONEBench, a sample-level benchmarking paradigm that pools heterogeneous measurement data across multiple benchmarks and aggregates them with a Plackett-Luce (PL) rank model. The authors argue that converting all model measurements into ordinal rankings and fitting PL parameters via maximum likelihood yields an identifiable, sample-efficient aggregation that recovers ground-truth model scores, remains accurate with up to 95% randomly missing data, and supports open-ended capability queries through semantic and metadata retrieval. The manuscript includes four real-world benchmark suites (HELM, Open LLM Leaderboard, VHELM, LMMs-Eval), comparisons against Elo, Bradley-Terry, Borda, and Dowdall aggregation, synthetic data experiments, and a capability-probing study with manual annotation of retrieval quality.","tokens_in":37402,"tokens_out":5685,"duration_ms":68450,"significance":"If the central claims held as stated, the work would be a valuable contribution to foundation-model evaluation: it provides an open, expanding infrastructure, a unified treatment of heterogeneous metrics, and a concrete demonstration that ordinal aggregation can be more robust than Elo-style ratings. The paper ships two benchmark pools (ONEBench-LLM, ONEBench-LMM), compares several aggregation methods on shared data, and includes capability-probing evaluations with human-annotated retrieval quality, which are useful practical artifacts. The significance of the theoretical claims, however, is currently limited by validation issues: the real-data ground truth is constructed from the same scores used to form the PL input rankings, and the synthetic validation uses the PL noise model itself. These issues do not destroy the framework's practical potential, but they prevent the paper from substantiating its headline guarantee of asymptotically recovering ground-truth scores on real, misspecified data.","major_comments":[{"comment":"The ground truth for real-data evaluation is defined as the mean min-max normalized score computed from the same per-sample measurements that are then converted into the ordinal rankings fed to the PL estimator. The reported Kendall's tau therefore measures agreement between two summaries of the same input data, not recovery of an independently defined latent quality. Under the PL random-utility model, MLE is expected to rank models according to their mean latent scores, so a nontrivial correlation with mean scores is partly built in by construction. The Limitations section concedes that the statistical modeling assumptions 'may not always hold', but no experiment quantifies the effect of this concession. I request an external-validity check: for example, fit PL on a subset of benchmarks or samples and compare against leaderboard rankings from a held-out benchmark, or compare against an independent preference signal such as human arena win rates that were not used in fitting. Without such a check, the real-data results do not test the load-bearing claim of recovering an underlying score independent of the defined ground truth.","section":"Section 3.2.1 and Tables 1-2"},{"comment":"The robustness experiments simulate missingness by uniformly random removal of samples or model measurements. ONEBench's motivating incompleteness, however, is structural: different benchmarks evaluate largely disjoint model subsets, and models enter the pool over time, so the observed measurement matrix contains blocks of missing entries rather than uniform random holes. Uniform subsampling from a fully observed matrix cannot reproduce the sparsity pattern that the framework is designed to handle. The claim of maintaining accurate rankings at 95% missing data should be tested under structured missingness, such as holding out entire benchmarks for subsets of models, or withholding all measurements for models added in a later time period, and reporting Kendall's tau under those patterns.","section":"Section 3.2.3 and Figure 3"},{"comment":"The synthetic experiments generate scores from Gumbel-distributed random variables, which is precisely the noise model underlying the Plackett-Luce distribution assumed in Section 3.1. Table 7 therefore verifies that the MLE recovers the parameters of the assumed generative model; it cannot reveal behavior when real measurements are generated differently (for example, by binary correctness with sample-dependent difficulty, log-normal continuous scores, or heterogeneous per-benchmark noise). Since the real-data validation is circular in the sense described above, these synthetic experiments are the only currently available test of recovery, and they only cover the favorable same-model case. I ask for a misspecification experiment with non-PL generative distributions to quantify the rank-recovery error when the PL assumption is violated.","section":"Appendix D, Synthetic Data Experiments"},{"comment":"The identifiability guarantee is stated for data where all models are comparable through a directed path. The paper does not report whether the real measurement graph in ONEBench-LLM or ONEBench-LMM is connected, nor how many samples each model appears in. If the comparison graph has disconnected components, MLE cannot identify relative scores across components, making a global Kendall's tau ill-defined. Please report the connectivity of the measurement graph, the largest connected component, and per-model sample counts, or restrict global ranking claims to the connected component containing the reference model.","section":"Section 3.1, Property 1"}],"minor_comments":[{"comment":"Please fix typographical errors: 'ONEBENCHto' in the title, 'Placket-Luce' instead of 'Plackett-Luce' in the main text, 'continuosly' in Section 2.1, 'practioners' in Appendix F.2, and 'over-ended' in Appendix G.","section":"Title, Abstract, and throughout"},{"comment":"The statement that PL parameters are identifiable 'up to an arbitrary additive constant' is imprecise: the gamma parameters in Eq. (1) are identifiable only up to a multiplicative constant, or equivalently additive in log-score space. The strict concavity claim for the likelihood also holds only after fixing this gauge, e.g., by setting the baseline model's log-score to zero. Please state this more carefully.","section":"Section 3.1, Eq. (1)"},{"comment":"The min-max normalization used to construct the ground truth is not precisely specified: it should be stated whether normalization is applied per benchmark, per metric, or across the entire pooled data, since different choices produce different ground-truth rankings and affect reproducibility.","section":"Section 3.2.1 and Appendix A"},{"comment":"The number of concepts is reported inconsistently: Section 4.3 mentions 'a curated set of 50 concepts', Table 3 reports 40 for LLM and 50 for LMM, and Appendix F.2 says 'ONEBench-LMM(50 vs 40)', which appears reversed. Please align these counts and clarify which concepts were used for which analysis.","section":"Section 4.3 and Table 3"},{"comment":"For the comparison in Table 1, the paper reports mean and variance over three runs but does not state the source of stochasticity (e.g., random subsampling, rank-breaking randomness, or initialization). Please specify the randomness that produces the variance.","section":"Section 3.2.1, Metrics"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful infrastructure contribution and the capability-probing part is original, but the central validation currently rests on a circular real-data ground truth and a synthetic experiment that matches the PL noise model. These are fixable with additional experiments, so I recommend major revision rather than rejection. The authors should also be asked to reconcile the concept-count inconsistencies and the identifiability statement before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on ONEBench. The new thing is the combination of sample-level rankings, Plackett-Luce aggregation, and explicit handling of heterogeneous metrics and incomplete data. Prior work either aggregated at task level or required complete binary matrices. The capability querying interface is also a nice touch. The paper does a thorough job integrating many benchmarks into ONEBench-LLM and ONEBench-LMM, and the experiments show PL consistently beats Elo and Bradley-Terry at reproducing leaderboard rankings. Robustness to 95% random missingness is impressive, and the retrieval precision numbers are solid.\n\nThe soft spots are real but not fatal. The main one is that the ground truth is the mean of min-max normalized per-sample scores from the same data used to fit PL. So the reported Kendall-tau measures agreement between two summaries of the same measurements, not recovery of an independent latent quality. The identifiability guarantee is for PL-generated data, and the synthetic experiments use Gumbel noise, which is exactly the PL generative model. That doesn't test misspecification. The limitations section concedes this but doesn't quantify how much deviation from PL hurts.\n\nSecond, all missingness is simulated as random. Real incompleteness is structured — models are evaluated on different benchmark subsets, not random samples. The 95% robustness may not hold under structured missingness. The paper should test that.\n\nThird, the theory is cited rather than proved, but that's fine for a systems paper.\n\nThe central argument holds up if you read it as 'PL aggregation reproduces mean-score rankings well and is robust to random missing data.' The stronger claim of 'asymptotically recovering ground-truth scores' is only established in-model. The authors are honest about this in the limitations, but the abstract oversells it.\n\nI'd send this to review. It's a practical contribution to benchmarking methodology, with a large released benchmark and a reasonable evaluation. A good referee would ask for structured missingness experiments and a clearer statement of what the real-data validation does — agreement with a heuristic, not recovery of an independent ground truth.","headline":"ONEBench is a useful, honest benchmarking framework; the recovery claim is only validated against a mean-score ground truth and random missingness, but the contribution stands.","tokens_in":37896,"tokens_out":3790,"would_cite":true,"duration_ms":39476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that converting all sample-level model measurements into ordinal rankings and aggregating them with the Plackett-Luce model recovers the ground-truth model ranking even when up to 95% of measurements are missing, making…","keywords":["open-ended benchmarking","sample-level evaluation","Plackett-Luce model","rank aggregation","ordinal rankings","missing data robustness","capability probing","foundation model evaluation"],"falsifier":"Generate synthetic benchmarks from a non-Plackett-Luce process, such as a Thurstone model where models have unequal score variances or correlated errors, then run the proposed aggregation under 50% to 95% missing data; if the recovered ranking's Kendall tau drops far below the reported values, the recovery guarantee is specific to the assumed model rather than a general property of ordinal sample-level aggregation.","tokens_in":37014,"feed_emoji":"🏆","tokens_out":8268,"duration_ms":74408,"temperature":0.7,"pith_summary":"ONEBench proposes replacing fixed task-specific benchmarks with a continuously expanding pool of sample-level tests, where each test sample votes by ranking the models it was used on. The paper argues that aggregating these partial, heterogeneous ordinal rankings with the Plackett-Luce model recovers the underlying model ranking even when most samples are missing for most models. This matters because it would make benchmarking cheaper—up to 20 times fewer measurements—and would let users construct personalized capability benchmarks from a shared pool. The argument combines theoretical identifiability and sample-efficiency guarantees with empirical correlations on real language and vision-language leaderboards.","feed_headline":"Model rankings survive when 95% of benchmark data is missing","feed_subtitle":"ONEBench turns every test sample into a voter, cutting evaluation cost by up to 20x.","key_machinery":"The load-bearing mechanism is the Plackett-Luce random utility model: each model has a positive score, and the probability of seeing a particular ranking is the product of sequential choice probabilities. Maximum likelihood estimation over these scores from partial rankings is the aggregation step, and its properties—strict concavity, identifiability up to an additive constant, and minimax-optimal sample efficiency when comparisons form an expander graph—are what make sparse, heterogeneous sample-level data recoverable. Rank-breaking converts partial rankings into near-optimal pairwise estimates in the implementation.","core_discovery":"The central discovery is that model evaluation can be reframed as preference aggregation at the sample level: each test item is a voter over the models it was evaluated on, and the Plackett-Luce model turns those sparse votes into a global ranking. The paper claims that this aggregation is identifiable, because it asymptotically recovers ground-truth utility scores, and sample-efficient enough that accurate rankings survive up to 95% missing measurements. On real benchmarks, the recovered rankings correlate strongly with mean-score rankings on homogeneous data, and the approach handles heterogeneous metrics by first converting all measurements to ordinal comparisons.","pith_inferences":["Beyond the paper, the same ordinal aggregation could be applied to other open-ended evaluation settings, such as text-to-image generation or code synthesis, where metrics are heterogeneous and coverage is sparse.","Beyond the paper, an adaptive version that chooses which sample-model measurements to collect next could reach the same ranking accuracy with even less compute than random subsampling.","Beyond the paper, the claimed ground-truth recovery would be strengthened by validating against an independent held-out sample set, rather than only against mean-score rankings from the same leaderboards."],"forward_implications":["A single expanding sample pool can serve many custom benchmarks, because the same sample-level rankings are reused across queries.","Evaluation cost can drop by up to 20 times, because accurate rankings survive about 95% missing measurements.","Model rankings become more comparable across benchmarks that use different metrics, because all measurements are first converted to ordinal comparisons.","Capability-specific queries produce rankings that differ systematically from the global ranking, revealing model specialization.","The theoretical identifiability and sample-efficiency results justify aggregating incomplete and heterogeneous test sets into one ranking."],"supporting_citations":[{"why":"Defines the ranking probability model whose likelihood ONEBench maximizes.","marker":"Plackett, 1975; Luce, 1959"},{"why":"Supplies the fast, accurate MLE algorithm and sample-efficiency bounds used for aggregation.","marker":"Maystre and Grossglauser, 2015"},{"why":"Establishes identifiability of Plackett-Luce utilities from directed-path comparisons and notes why general random-utility MLE is intractable.","marker":"Xia, 2019"},{"why":"Provides consistency and asymptotic normality results for Plackett-Luce estimators under comparison-graph assumptions.","marker":"Han and Xu, 2023"},{"why":"Gives rank-breaking estimators that convert partial rankings into near-optimal pairwise comparisons.","marker":"Soufiani et al., 2014"},{"why":"Shows model rankings are more stable than scores across test sets, supporting the ordinal aggregation choice.","marker":"Recht et al., 2019"},{"why":"Extends the rank-stability evidence to a new dataset, arguing ordinal comparisons are more robust.","marker":"Salaudeen and Hardt, 2024"},{"why":"Documents why Elo-style assumptions fit LLM evaluation poorly, motivating Plackett-Luce instead.","marker":"Boubdir et al., 2023"},{"why":"Provides evidence that pairwise comparisons can beat cardinal scores despite information loss.","marker":"Shah et al., 2014"},{"why":"Inspires using rankings to guard against overfitting and benchmark contamination.","marker":"Hardt and Recht, 2022"}],"fun_headline_variants":["Sample-level voting ranks models despite 95% missing data","Aggregate test items as votes to rank models cheaply","Open-ended benchmarks from a unified, growing sample pool","Incomplete data? Sample aggregation recovers rankings","Cut evaluation cost 20x by pooling sample-level tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on assuming that real benchmark measurements are generated by a Plackett-Luce random utility model; if that generative assumption is violated, the theoretical recovery guarantees no longer apply and the empirical ranking is only matching the mean-score ground truth used to validate it.","fun_headline_variants_meta":{"raw":{"variants":["Sample-level voting ranks models despite 95% missing data","Aggregate test items as votes to rank models cheaply","Open-ended benchmarks from a unified, growing sample pool","Incomplete data? Sample aggregation recovers rankings","Cut evaluation cost 20x by pooling sample-level tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000349,"raw_usage":{"total_tokens":1923,"prompt_tokens":975,"completion_tokens":948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":870}},"tokens_in":591,"tokens_out":948,"duration_ms":10282,"temperature":1.0,"reasoning_tokens":870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:18:46.398216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate synthetic benchmarks from a non-Plackett-Luce process, such as a Thurstone model where models have unequal score variances or correlated errors, then run the proposed aggregation under 50% to 95% missing data; if the recovered ranking's Kendall tau drops far below the reported values, the recovery guarantee is specific to the assumed model rather than a general property of ordinal sample-level aggregation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides consistency and asymptotic normality results for Plackett-Luce estimators under comparison-graph assumptions."}],"review_version":1}