Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Dennis Wei; Jenny Y. Huang; Tamara Broderick; Yunyi Shen

REVIEW 3 major objections 3 minor 1 cited by

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dropping 0.02% of matchups flips the number-one LLM on Chatbot Arena.

desk verdict Real result for tie-free BT fits, but the headline overreaches to the actual Chatbot Arena leaderboard and overstates minimality. read the letter →

arxiv 2508.11847 v4 pith:4EUSQFUI submitted 2025-08-16 stat.ML cs.LG

Jenny Y. Huang , Yunyi Shen , Dennis Wei , Tamara Broderick This is my paper · ORCID

classification stat.MLcs.LG MSC 62F3562J12

keywords Bradley-Terrymodelworst-casedatadroppinginfluencefunctionsLLMleaderboardrobustnessChatbotArenaMT-Benchpreferencerankstability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that widely used language-model leaderboards, built by fitting a Bradley-Terry model to pairwise preferences, can change their top rankings when an adversary removes a tiny, specially chosen subset of evaluations. On tie-free Chatbot Arena data, deleting nine matchups out of roughly 40,000, about 0.02%, swaps the first- and second-ranked models; deleting twenty changes a member of the top five. The paper also argues that MT-Bench, with its expert annotators and carefully constructed prompts, is far more stable, requiring more than 3% of its data to move the top model, and that human-judged and LLM-judged rankings are comparably fragile. This matters because leaderboards are used to decide which models to deploy, train, or trust, and the paper supplies a fast way to check whether a reported top-k ranking survives deletion of a handful of preferences.

What carries the argument

Approximate Maximum Influence Perturbation (AMIP), a first-order Taylor expansion in data-removal weights that replaces the combinatorial search over all $\alpha$-fraction subsets with a ranking of per-observation influence scores. For a candidate pair $(i,j)$, the relevant per-evaluation influence is computed from the classical logistic-regression influence-function formula, i.e. the effect of removing each matchup on the fitted score gap $\hat\theta_i-\hat\theta_j$; the $\lfloor\alpha N\rfloor$ matchups with the largest negative influence are dropped, and the Bradley-Terry model is refit to confirm whether the score gap changes sign. Top-$k$ robustness is reduced, via Proposition 3.1, to p

What would settle it

Run the paper's deletion search on the official Chatbot Arena leaderboard computation that represents each tied match as two weighted win/loss rows; if no 0.02%-scale subset flips the top model under that tie-handling, the reported sensitivity is an artifact of tie removal. As a method check, apply exact combinatorial search or rigorous optimization to a smaller arena, where brute force is feasible, and compare with the AMIP/greedy answer to see whether it ever misses a smaller flipping subset.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that top-k orderings in Bradley-Terry based LLM evaluation arenas are not stable under small worst-case deletions. In tie-free Chatbot Arena data, dropping nine of 39,716 evaluations flips GPT-4-0125-preview from first to second, replaced by GPT-4-1106-preview; dropping twenty evaluations, 0.035%, changes one member of the top five. The same procedure on MT-Bench requires dropping 83 of 2,575 evaluations, 3.22%, to move the top model from GPT-4 to Claude-v1, and 110 evaluations, 4.27%, to alter the top five. The authors further claim that human-judged and LLM-judged rankings are similarly sensitive in both arenas, with no systematic differen

Load-bearing premise

The headline sensitivity number assumes that a tie-free Bradley-Terry fit is a faithful stand-in for Chatbot Arena's actual weighted-tie leaderboard, and that the first-order influence approximation plus the greedy closest-pair search finds every small subset that can flip a top-k ranking; if either fails, the true minimal deletion fraction could be larger.

Editorial extensions

If this is right

If a leaderboard's top position can move with fewer than ten deleted matchups, then quoting that position as a stable fact overstates the information contained in the underlying preferences.
The paper's robustness check is cheap enough to run routinely: for a roughly 50,000-matchup arena, the paper reports top-1 and top-5 checks in under three minutes on a laptop.
Expert-annotated, carefully prompted benchmarks such as MT-Bench are much harder to perturb than crowdsourced ones, giving concrete evidence for investing in annotation quality and prompt construction.
The method returns the exact evaluations responsible for a flip, so platform operators can inspect, remove, or repair the specific votes that carry a leaderboard position.
Replacing crowd raters with LLM judges does not by itself stabilize rankings, because both evaluation sources show similar sensitivity in the paper's tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The same deletion-sensitivity test could be applied to reward models trained with RLHF, not just to leaderboards; if a few preference pairs can move a ranking, they may also move the reward model's behavior.
Editorial inference: The observed concentration of flips in a single repeated model-pair suggests a cheap monitoring rule for platforms: track pair-specific win streaks and flag matchups where one model consistently beats a top competitor.
Editorial inference: Because sensitivity depends on score gaps, newer leaderboards with increasingly close top models may become more deletion-sensitive over time; the paper's numbers are a snapshot, not a constant.
Editorial inference: Re-running the deletion search under Chatbot Arena's official weighted-tie Bradley-Terry pipeline would directly test whether the 0.02% figure is an artifact of omitting ties.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper does something real and mostly does it carefully. It extends AMIP to Bradley-Terry rankings, reduces top-k robustness to pairwise checks, and verifies the reported flips by refitting. For tie-free BT fits on the released Arena data, the headline existence claim—removing 9 of ~40k evaluations flips the top two models—holds up. The code is public and the experiments look reproducible. That is worth a serious referee.

The soft spots are concentrated in how the results are framed. First, the tie-free model is not the Chatbot Arena leaderboard. The platform uses a weighted tie model, and Section 3.1 admits tie handling may change rankings. The paper never measures that gap. So the 0.02% (or 0.003%—the abstract disagrees with the body) describes a proxy, not the leaderboard users see. The finding is still interesting, but the headline oversells the target.

Second, the paper says MT-Bench “requires over 3%” of data to flip the top model. The reported 83 matches is a sufficient subset found by an approximate search, not a certified minimum. It should say “dropping 3.2% suffices,” not “requires over 3%.” The distinction matters for a robustness claim.

Third, the abstract/body percentage mismatch is careless and needs fixing.

None of this breaks the central existence result for tie-free fits. The method is a genuine extension, the refit verification is the right habit, and the human-vs-LLM-judge comparison is a fair secondary point. I just would not take the Chatbot Arena headline as a statement about the platform’s actual rankings until the tie handling is checked.

Verdict: conditional accept. This paper deserves peer review. A good reviewer should push for a re-run on the weighted-tie model (or a clear statement that the result is about tie-free rankings) and for replacing “requires” with “suffices.” I’d bring it to reading group if someone is thinking about leaderboard trust.

Referee Report

3 major / 3 minor

Summary. The paper proposes a computationally efficient method, based on Approximate Maximum Influence Perturbation (AMIP) and influence-function/one-step-Newton approximations, for assessing whether Bradley–Terry (BT) rankings of LLMs are robust to dropping a worst-case small fraction of preference evaluations. The method selects candidate data subsets and verifies each candidate by refitting the BT model. Applied to Chatbot Arena and MT-Bench data, the paper reports that dropping 9 of 39,716 tie-free human evaluations flips the top-1 model from GPT-4-0125-preview to GPT-4-1106-preview, that 20 evaluations can change a top-5 member, that MT-Bench requires over 3% of evaluations to flip its top model, and that human- and LLM-judged rankings are roughly equally sensitive. The paper frames these findings as evidence of fragility in widely used LLM leaderboards.

Significance. If the findings hold, the paper makes a useful contribution: it provides a practical, fast robustness-auditing tool for a widely used ranking model, with public code and a verification step that guards against false positives from influence-function approximation. The reported existence results for the tie-free BT model are supported by refitting, and the method's runtime is plausible. The central caveat is that the headline Chatbot Arena claim is established for the paper's tie-free logistic BT model, not for the platform's actual weighted-tie ranking procedure; Section 3.1 acknowledges this difference but does not quantify it. This is the main load-bearing gap, along with an abstract/body numerical inconsistency and an unsupported minimality claim for MT-Bench.

major comments (3)

[§3.1, §4.2, Table 1] The headline claim that dropping 9 evaluations changes the top-ranked model on Chatbot Arena is computed on tie-free data (9 of 39,716), whereas the Chatbot Arena leaderboard uses a weighted BT model in which each match contributes two rows and ties are encoded as wins for both sides. The objective in Eq. (4) and the influence scores in Eq. (11) differ between these two formulations, so the 9 selected evaluations are not guaranteed to flip the platform's actual weighted-tie leaderboard. Section 3.1 concedes this only qualitatively. The paper should either run the same robustness check on the weighted-tie objective or explicitly relabel the claim as a property of the tie-free BT model rather than of Chatbot Arena.
[Abstract, §4.2, Table 1] The abstract states that dropping 0.003% of preferences changes the top-ranked model on Chatbot Arena, while the body reports 9 out of 39,716, which is 0.0226% (and 0.0156% of the 57,477 total evaluations). 0.003% is off by a factor of roughly 7.5 and appears to be an error. This discrepancy materially changes the strength of the headline claim and must be corrected, with the denominator clearly specified.
[§4.2, Table 1] The statement that MT-Bench 'requires over 3%' of the data to change the top-ranked model is presented as a minimality claim, but the method only demonstrates that dropping 83 out of 2,575 evaluations (3.22%) suffices to produce a flip. The greedy closest-pair search and influence-function/one-step-Newton approximations do not establish that no smaller subset could flip the ranking; they only provide an upper bound. The text should be reframed as 'we found a flip at 3.22%' unless a lower-bound search is performed for all smaller α. The same issue applies to the 'over 4%' statement for top-5.

minor comments (3)

[§3] The notation for the top-k set and pairwise robustness is used somewhat loosely: Eq. (8) defines a pairwise notion of robustness, but the subsequent text sometimes uses 'robust' without specifying whether it refers to pairwise or top-k robustness. Clarifying this distinction would improve readability.
[§4.1] The paper mentions 57,477 total Chatbot Arena evaluations but then reports results on 39,716 tie-free evaluations. The relationship between these numbers and the tie-discarding procedure should be stated explicitly in the main text, not only implicitly.
[§4.5, Table 2] Table 2 presents five dropped prompts and responses for the k=5 flip, but the main text could be clearer that these are the five evaluations mentioned in the caption of Figure 3 and not the full set of 20 evaluations for top-5 sensitivity. A short cross-reference would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline sensitivity result is established by refitting on real preference data, not by definition or by fitting a parameter to the target outcome.

full rationale

The paper's central claim is an empirical existence result: for the tie-free Bradley-Terry fits on the Chatbot Arena and MT-Bench datasets, removing a small subset of matchups and refitting the model changes the top-ranked model (Table 1, Section 4.2). The subset-selection step uses influence-function/AMIP approximations from prior work, including Broderick et al. (2020) and Huang et al. (2024) by the same group, but the load-bearing verification is a direct refit of the BT model on the reduced data (Section 3 and Appendix A: 'Once a candidate subset is identified, however, our method always performs a refitting... to verify whether the non-robustness is true'). The reported flips are therefore properties of the data and the fitted model, not identities. The influence scores in Eq. (11) are standard logistic-regression diagnostics (Pregibon 1981), and the method never fits a parameter to the target ranking flip and then reports it as a prediction; candidate subsets are checked exactly. The self-citations to AMIP and one-step Newton are methodological scaffolding, not an imported uniqueness theorem or a forced ansatz. The acknowledged tie-free proxy vs. weighted-tie leaderboard mismatch (Section 3.1) is a faithfulness threat to the Chatbot Arena headline, and the wording 'requires over 3%' is stronger than the sufficient-subset evidence, but these are correctness risks, not circularity. No derivation step reduces to its own input by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted constants are introduced to obtain the finding; BT scores are the statistic under audit, not hidden parameters. The user-selected settings such as alpha, k, and tie filtering are audit inputs. The main assumptions are model specification, approximation validity, and the tie-free proxy.

assumptions (5)

domain assumption Preference outcomes are generated by the Bradley-Terry model: y_n ~ Bernoulli(sigmoid(theta_in - theta_jn)) (Eq. 1).
Every ranking and robustness claim is computed under this model; if actual user votes follow a different process, the fragility numbers apply to the BT fit, not to true quality.
standard math Maximum likelihood estimate exists and X^T V X is invertible for influence scores (Eqs. 3 and 11).
The influence function formula requires computing (X^T V X)^{-1}; no regularization is mentioned, so separability or collinearity would invalidate the scores.
domain assumption The first-order influence approximation identifies a subset that, after refitting, really flips the ranking.
AMIP is an approximation; the paper verifies each reported flip by refitting, but if the approximation misses flips, negative thresholds such as 'requires over 3%' are overstated. Appendix A.
domain assumption Tie-free Bradley-Terry analysis is an acceptable proxy for Chatbot Arena's weighted-tie leaderboard.
Section 3.1 states tie handling may affect rankings; the headline applies to this proxy rather than the exact platform procedure.
domain assumption Greedy closest-pair search plus early termination finds a non-robust pair if one exists.
Only closely ranked pairs are checked first and the procedure stops on the first non-robust pair; without an exhaustive search, completeness is assumed. Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings." pith.science (2026). https://pith.science/paper/4EUSQFUI

@misc{pith2026250811847,
  author       = {Pith},
  title        = {Pith review of: Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EUSQFUI}},
  note         = {Machine review of arXiv:2508.11847}
}

read the original abstract

We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.

Figures

Figures reproduced from arXiv: 2508.11847 by the authors.

**Figure 2.** Each bar shows the percentage of data points dropped from Chatbot Arena that is [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

**Figure 3.** Player involvement in the most influential matches whose removal caused two models (players), one inside the top-k positions and one outside, to exchange places. Each bar represents the composition of matches dropped in order to exchange the models. The proportions indicate whether neither (green), one (orange), or both (blue) of the affected models were involved in each dropped matchup. For Chatbot Arena (Human-Jud… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The number of times each model appears in a match in Chatbot Arena. The horizontal bar chart shows how frequently each model appeared in any match, with GPT-4 and GPT-3.5 variants being the most represented. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_4.png]

**Figure 5.** Figure 5: The model rankings in MT-Bench. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

**Figure 6.** Figure 6: Robustness of results is correlated with the proximity of the BT scores. Each point [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
cs.CL 2026-08 conditional novelty 6.0 of 10

FinReportBench is a fine-grained, expert-grounded benchmark for institution-grade LLM financial report generation, and its skill-evolution method improves G1 and G2 scores across model families.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · cited by 1 Pith paper

[1]

Fit a BT model on the entire arena

work page
[2]

(11)) (one-step Newton scores for the Additive one-step Newton algorithm) for all matches in the arena

Compute the influence scores (Eq. (11)) (one-step Newton scores for the Additive one-step Newton algorithm) for all matches in the arena

work page
[3]

(13)) is the largest in the negative direction (assuming that player i has a higher estimated BT score than player j on the full data)

Identify the ⌊αN ⌋ matchups for which the difference in influence scores (as given in Eq. (13)) is the largest in the negative direction (assuming that player i has a higher estimated BT score than player j on the full data)

work page
[4]

Approximate impact of dropping these ⌊αN ⌋ matchups by the sum of the influence score approximations

work page
[5]

If the approximation predicts that the relative ranking between playersi and j changed, then refit the model leaving out the identified subgroup. 9 These data-dropping algorithms replace a computationally intractable combinatorial search with an algorithm that costs only O(Analysis + N log(αN ) + N P2 + P 3), where Analysis represents the cost of fitting ...

work page
[1981]

is given by ∂ ˆθj(w) ∂wn w=1N = e⊤ j (X⊤V X)−1xnbpn(1 − bpn) (yn − bpn) , (14) In addition to influence functions, our framework enables a second data-dropping approxi- mation known as the One-step Newton (1sN) approximation, which approximates the effect of dropping a data point on model parameters using a second-order Taylor expansion in 23 parameter sp...

work page 2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Fit a BT model on the entire arena

work page

[2] [2]

(11)) (one-step Newton scores for the Additive one-step Newton algorithm) for all matches in the arena

Compute the influence scores (Eq. (11)) (one-step Newton scores for the Additive one-step Newton algorithm) for all matches in the arena

work page

[3] [3]

(13)) is the largest in the negative direction (assuming that player i has a higher estimated BT score than player j on the full data)

Identify the ⌊αN ⌋ matchups for which the difference in influence scores (as given in Eq. (13)) is the largest in the negative direction (assuming that player i has a higher estimated BT score than player j on the full data)

work page

[4] [4]

Approximate impact of dropping these ⌊αN ⌋ matchups by the sum of the influence score approximations

work page

[5] [5]

If the approximation predicts that the relative ranking between playersi and j changed, then refit the model leaving out the identified subgroup. 9 These data-dropping algorithms replace a computationally intractable combinatorial search with an algorithm that costs only O(Analysis + N log(αN ) + N P2 + P 3), where Analysis represents the cost of fitting ...

work page

[6] [1981]

is given by ∂ ˆθj(w) ∂wn w=1N = e⊤ j (X⊤V X)−1xnbpn(1 − bpn) (yn − bpn) , (14) In addition to influence functions, our framework enables a second data-dropping approxi- mation known as the One-step Newton (1sN) approximation, which approximates the effect of dropping a data point on model parameters using a second-order Taylor expansion in 23 parameter sp...

work page 2020