REVIEW 5 cited by
Improving Your Model Ranking on Chatbot Arena by Vote Rigging
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Chatbot Arena is a popular platform for evaluating LLMs by pairwise battles, where users vote for their preferred response from two randomly sampled anonymous models. While Chatbot Arena is widely regarded as a reliable LLM ranking leaderboard, we show that crowdsourced voting can be rigged to improve (or decrease) the ranking of a target model $m_{t}$. We first introduce a straightforward target-only rigging strategy that focuses on new battles involving $m_{t}$, identifying it via watermarking or a binary classifier, and exclusively voting for $m_{t}$ wins. However, this strategy is practically inefficient because there are over $190$ models on Chatbot Arena and on average only about $1\%$ of new battles will involve $m_{t}$. To overcome this, we propose omnipresent rigging strategies, exploiting the Elo rating mechanism of Chatbot Arena that any new vote on a battle can influence the ranking of the target model $m_{t}$, even if $m_{t}$ is not directly involved in the battle. We conduct experiments on around $1.7$ million historical votes from the Chatbot Arena Notebook, showing that omnipresent rigging strategies can improve model rankings by rigging only hundreds of new votes. While we have evaluated several defense mechanisms, our findings highlight the importance of continued efforts to prevent vote rigging. Our code is available at https://github.com/sail-sg/Rigging-ChatbotArena.
Forward citations
Cited by 5 Pith papers
-
The Leaderboard Illusion
Chatbot Arena's rankings are systematically distorted by undisclosed private testing, selective score reporting, and data access asymmetries that favor large proprietary providers.
-
Ranked Voting based Self-Consistency of Large Language Models
Generating ranked answer lists instead of single answers and aggregating them with instant-runoff, Borda count, or mean reciprocal rank voting improves self-consistency accuracy by roughly 1 to 5 points on six QA benchmarks.
-
PointArena: Probing Multimodal Grounding Through Language-Guided Pointing
A three-stage benchmark consisting of 982 pointing tasks, a live pairwise arena with 4,500 votes, and a robot manipulation study shows that pointing-supervised open models such as Molmo-72B can match proprietary model...
-
Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.
-
The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations
In a public LLM comparison arena, showing users that the larger model consumes more energy caused about 46% of users who preferred it to say they would switch to the smaller model.
Discussion (0). Continue with ORCID to comment.