{"id":"cfb4dbc6-4afa-4322-b8c7-0e314f48c378","arxiv_id":"2606.02547","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Introduces a leaderboard mechanism satisfying local stability for heterogeneous preferences using O~(k) comparisons per user and shows standard Bradley-Terry aggregation violates it on LMArena data.","lead":"The paper proposes pluralistic leaderboards for LLMs that use local stability from social choice theory to handle diverse user preferences instead of a single Bradley-Terry ranking. A smart generalist might read it because current LLM evaluations can systematically ignore large groups of users with different needs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Empirical demonstration that BT violates local stability hinges on unstated data aggregation assumptions","rationale":"The reader's weakest assumption directly identifies the same data-representativeness issue that underpins the empirical half of the strongest claim. The theoretical construction is not challenged here because no internal inconsistency is visible from the abstract. The verdict adjustment to CONDITIONAL reflects that the 'in practice' demonstration requires additional verification of the user-fraction estimator before the full claim can be accepted at face value.","tokens_in":1727,"tokens_out":376,"duration_ms":30603,"concrete_test":"Reproduce the LMArena stability check using two different operationalizations of 'fraction of users': (1) treat each battle as an independent sample and threshold aggregate win rates, (2) cluster battles by prompt embedding or user session (if available) and compute per-cluster majorities; if the set of models violating the O(1/k) threshold differs materially between (1) and (2), the empirical claim is sensitive to the aggregation choice.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim includes both a theoretical guarantee for the proposed mechanism and the statement that 'standard Bradley-Terry aggregation can violate local stability in practice' on LMArena data. Local stability requires that no outside model is 'collectively preferred to the top-k set by more than O(1/k) fraction of users.' LMArena supplies anonymous pairwise votes without per-user longitudinal data, so estimating this fraction necessarily involves an auxiliary modeling step (e.g., implicit clustering, win-rate thresholding, or treating each battle as an independent draw from a heterogeneous population). If that step is misspecified, the reported violation could be an artifact rather than evidence that BT fails the definition. The theoretical O(k) sample bound is not directly implicated by this issue.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes pluralistic leaderboards for LLMs that achieve local stability (no outside model preferred to the top-k by more than O(1/k) fraction of users) for heterogeneous preferences, drawing on social choice theory. The mechanism elicits only \tilde{O}(k) pairwise comparisons per user while satisfying the stability property, in contrast to standard Bradley-Terry aggregation which the authors show can violate local stability on LMArena data.","tokens_in":1884,"tokens_out":538,"duration_ms":23076,"significance":"If the local-stability guarantee and the empirical contrast hold under the stated modeling assumptions, the work supplies an efficient, axiomatically grounded alternative to single-score leaderboards that respects preference heterogeneity. The \tilde{O}(k) sample bound and the parameter-free character inherited from social choice are concrete strengths that could influence evaluation practice.","major_comments":[{"comment":"The claim that standard Bradley-Terry aggregation violates local stability on LMArena data (abstract and experimental section) requires an auxiliary model to estimate the 'O(1/k) fraction of users' from anonymous pairwise votes that lack per-user longitudinal structure. The precise aggregation, clustering, or thresholding step used to operationalize collective preference must be stated explicitly and subjected to sensitivity checks; without this, the reported violation risks being an artifact of the auxiliary modeling choice rather than evidence against BT.","section":"Experimental evaluation / data-processing pipeline"},{"comment":"The local-stability guarantee and the \tilde{O}(k) comparison bound (theoretical section) are load-bearing for the central contribution. The manuscript should verify that the mechanism satisfies the exact definition of local stability given in the paper (no outside model collectively preferred by >O(1/k) users) without hidden homogeneity assumptions, and that the bound remains valid under the heterogeneous population model used for the LMArena experiments.","section":"Theoretical analysis / mechanism definition"}],"minor_comments":[{"comment":"Add a short, self-contained paragraph in the introduction that recalls the precise definition of local stability before invoking it in the abstract claim.","section":"Introduction"},{"comment":"Clarify notation for the prefix size k and the hidden constants in the \tilde{O}(k) bound when first introduced.","section":"Preliminaries"}],"recommendation":"major_revision","confidential_remarks":"The manuscript aligns with the journal's scope at the intersection of game theory / social choice and AI evaluation. The citation pattern to social-choice literature appears appropriate and non-circular."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback on the experimental pipeline and theoretical guarantees. We address each major comment below.","responses":[{"response":"We agree that the auxiliary modeling steps used to estimate the fraction of users from anonymous LMArena votes require explicit documentation. In the revised manuscript we will describe the precise aggregation, clustering, and thresholding procedures, and we will add sensitivity analyses over a range of parameter choices to show that the reported violation of local stability by Bradley-Terry aggregation is robust.","revision_made":"yes","referee_comment":"[Experimental evaluation / data-processing pipeline] The claim that standard Bradley-Terry aggregation violates local stability on LMArena data (abstract and experimental section) requires an auxiliary model to estimate the 'O(1/k) fraction of users' from anonymous pairwise votes that lack per-user longitudinal structure. The precise aggregation, clustering, or thresholding step used to operationalize collective preference must be stated explicitly and subjected to sensitivity checks; without this, the reported violation risks being an artifact of the auxiliary modeling choice rather than evidence against BT."},{"response":"The mechanism is taken from social-choice results that hold for arbitrary heterogeneous preference profiles and contain no homogeneity assumptions. We will add a short verification subsection confirming that the mechanism meets the paper's exact local-stability definition and that the \tilde{O}(k) bound continues to hold when the underlying population is the heterogeneous model used in the LMArena experiments.","revision_made":"yes","referee_comment":"[Theoretical analysis / mechanism definition] The local-stability guarantee and the \tilde{O}(k) comparison bound (theoretical section) are load-bearing for the central contribution. The manuscript should verify that the mechanism satisfies the exact definition of local stability given in the paper (no outside model collectively preferred by >O(1/k) users) without hidden homogeneity assumptions, and that the bound remains valid under the heterogeneous population model used for the LMArena experiments."}],"tokens_in":1403,"tokens_out":428,"duration_ms":23510,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core takeaway is that this work imports local stability to handle diverse user preferences in LLM rankings and pairs it with an elicitation scheme that uses only roughly O(k) comparisons per user. They also run an experiment claiming standard Bradley-Terry aggregation violates the property on real data.\n\nWhat stands out as new is the targeted combination: prior Bradley-Terry work on leaderboards did not focus on this stability notion or the corresponding sample bound. The theoretical guarantee appears to rest on existing social choice techniques, which they adapt cleanly. The empirical section at least attempts to show a concrete gap between the two approaches on LMArena battles.\n\nThe soft spot is the empirical claim that BT violates local stability in practice. LMArena supplies anonymous pairwise votes, not longitudinal per-user data, so measuring the fraction of users who collectively prefer an outside model requires some modeling step to turn raw battles into user-level preferences. The stress-test note correctly flags that this step is not automatic and could affect whether the reported violation is robust. The paper would be stronger if it spelled out the exact aggregation choices and checked sensitivity.\n\nLocal stability itself is a reasonable target for pluralistic settings, though it is one among several possible stability notions. The O(k) bound is the main technical payoff and looks like the part worth following up.\n\nThis is for researchers working on preference aggregation, AI evaluation platforms, or mechanism design for heterogeneous populations. A reader who cares about how leaderboards handle user diversity would get concrete value from the mechanism and the comparison to BT. It is worth sending to peer review because the problem is timely and the proposed fix has a clear formal grounding, even if the data analysis needs tightening.","headline":"The paper adapts local stability from social choice to LLM leaderboards and gives an O(k)-query mechanism that meets it, while showing BT can fail on LMArena data.","tokens_in":2359,"tokens_out":421,"would_cite":false,"duration_ms":15514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A mechanism for LLM leaderboards achieves local stability to heterogeneous preferences with O(k) pairwise comparisons per user.","keywords":["pluralistic leaderboards","local stability","Bradley-Terry model","LLM evaluation","pairwise comparisons","social choice theory","heterogeneous preferences","ranking aggregation"],"falsifier":"A preference profile or LMArena subsample in which the mechanism's reported top-k set is collectively preferred over by more than an O(1/k) fraction of users.","tokens_in":2637,"feed_emoji":"📊","tokens_out":643,"duration_ms":25762,"temperature":0.7,"pith_summary":"Standard Bradley-Terry aggregation fits a single latent score to all pairwise comparisons and produces one global ranking. This collapses distinct user preferences into one ordering that can systematically misrepresent groups favoring different model behaviors. The paper adapts local stability from social choice theory, requiring that no model outside the top-k set is preferred over that set by more than an O(1/k) fraction of users. It gives a mechanism that meets this condition while collecting only roughly O(k) comparisons from each user. On LMArena data the standard approach can violate the stability condition, while the new mechanism maintains it.","feed_headline":"Mechanism stabilizes LLM leaderboards for varied users","feed_subtitle":"It guarantees local stability using O(k) comparisons per user, while standard global rankings can misrepresent groups with different prefere","key_machinery":"Local stability, defined as the requirement that no model outside the reported top-k is preferred over the top-k set by more than an O(1/k) fraction of users, and the query-efficient aggregation procedure that enforces it.","core_discovery":"The paper claims that an alternative leaderboard mechanism, built from social choice techniques, satisfies local stability—no model outside the top-k is collectively preferred to the top-k by more than O(1/k) of users—while eliciting only tilde O(k) pairwise comparisons per user, and that this holds on real LMArena comparison data where Bradley-Terry aggregation violates the condition.","pith_inferences":["Leaderboards could report stable sets rather than total orders, allowing different user cohorts to see different top models without extra data collection.","The same local-stability approach might transfer to ranking other AI systems or consumer products that serve heterogeneous populations.","If platforms adopted the method, evaluation costs could drop while still protecting against large-group misrepresentation."],"forward_implications":["The mechanism meets the local stability guarantee for any chosen prefix length k.","Only tilde O(k) comparisons per user are needed to produce the stable ranking.","Bradley-Terry aggregation can and does violate local stability on existing LLM comparison data.","The new procedure supplies measurably stronger stability than single-score aggregation."],"fun_headline_variants":["Social choice stabilizes LLM leaderboards locally","Local stability via O(k) comparisons per user","Pluralistic leaderboards avoid Bradley-Terry issues","Stable top-k rankings for diverse LLM preferences","New mechanism guarantees local stability with few votes"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Local stability is the appropriate stability notion for LLM leaderboards and that LMArena pairwise data sufficiently represents the relevant heterogeneous user populations.","fun_headline_variants_meta":{"raw":{"variants":["Social choice stabilizes LLM leaderboards locally","Local stability via O(k) comparisons per user","Pluralistic leaderboards avoid Bradley-Terry issues","Stable top-k rankings for diverse LLM preferences","New mechanism guarantees local stability with few votes"]},"model":"grok-4.3","cost_usd":0.005581,"raw_usage":{"total_tokens":2669,"prompt_tokens":658,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":55812000,"prompt_tokens_details":{"text_tokens":658,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1946,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":658,"tokens_out":65,"duration_ms":18048,"temperature":1.0,"reasoning_tokens":1946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T11:52:32.542081+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A preference profile or LMArena subsample in which the mechanism's reported top-k set is collectively preferred over by more than an O(1/k) fraction of users.","supporting_citations":[],"review_version":1}