Pith. sign in

REVIEW 5 cited by

Prompt-to-Leaderboard

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14855 v2 pith:JJEZSPBZ submitted 2025-02-20 cs.LG cs.CL

Evan Frick , Connor Chen , Joseph Tennyson , Tianle Li , Wei-Lin Chiang , Anastasios N. Angelopoulos , Ion Stoica This is my paper · ORCID
classification cs.LGcs.CL
keywords modellanguagearenaaveragingchatbotevaluationevaluationsgithub
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

    cs.AI 2026-04 unverdicted novelty 6.5 of 10

    BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.

  2. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  3. RLAE: Reinforcement Learning-Assisted Ensemble for LLMs

    cs.LG 2025-05 reject novelty 6.0 of 10

    RLAE uses PPO and MAPPO policies to assign per-span ensemble weights across 7B-8B LLMs; it claims up to 3.3% accuracy improvement over prior ensemble baselines but underperforms on several tested tasks.

  4. Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference

    cs.MA 2025-09 reject novelty 5.0 of 10

    MoMA routes each query to a specialized agent or to the cost-optimal LLM, using judge-trained performance scores, a Pareto cost frontier, and TOPSIS selection.

  5. Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks

    cs.IT 2025-08 unverdicted novelty 5.0 of 10

    A routing policy that weighs predicted answer quality against wireless and compute cost is claimed to cut response latency by 5-15% and large-model invocations by 10-20% without losing benchmark accuracy.

Pith tools