REVIEW 11 cited by
MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Evaluating large language models (LLMs) is challenging. Traditional ground-truth-based benchmarks fail to capture the comprehensiveness and nuance of real-world queries, while LLM-as-judge benchmarks suffer from grading biases and limited query quantity. Both of them may also become contaminated over time. User-facing evaluation, such as Chatbot Arena, provides reliable signals but is costly and slow. In this work, we propose MixEval, a new paradigm for establishing efficient, gold-standard LLM evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks. Based on MixEval, we further build MixEval-Hard, which offers more room for model improvement. Our benchmarks' advantages lie in (1) a 0.96 model ranking correlation with Chatbot Arena arising from the highly impartial query distribution and grading mechanism, (2) fast, cheap, and reproducible execution (6% of the time and cost of MMLU), and (3) dynamic evaluation enabled by the rapid and stable data update pipeline. We provide extensive meta-evaluation and analysis for our and existing LLM benchmarks to deepen the community's understanding of LLM evaluation and guide future research directions.
Forward citations
Cited by 11 Pith papers
-
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
SPOT shows that state-of-the-art AI models detect fewer than one in five known errors in full scientific papers, with precision below 7%.
-
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
Shortcut neuron patching suppresses benchmark-contamination shortcuts in LLMs and yields evaluation scores that strongly correlate with the external MixEval benchmark.
-
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
A fully automatic LLM evaluation framework where all evaluated models serve as judges for one another reaches 97% Spearman correlation with human preference rankings while keeping cost sub-quadratic.
-
WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training
A 50-million-conversation synthetic chat dataset built from 54 open-weight models, plus an SFT mix that beats Tulu-3's mix with fewer samples.
-
How to Select Datapoints for Efficient Human Evaluation of NLG Models?
Selecting human-evaluation items by metric variance, metric consistency, output diversity, or IRT-based informativeness matches random-sampling ranking accuracy with roughly 70% of the annotation budget in WMT23 and SummEval.
-
Tuning LLM Judge Design Decisions for 1/1000 of the Cost
A multi-fidelity, multi-objective search finds cheap open-weight LLM judges that match or outperform prior judge designs on several benchmarks.
-
Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change
Fine-tuned GPT-3.5-turbo agrees with expert climate coders on social media claims as often as the experts agree with each other (alpha=0.89), but the study's open-source benchmark is weakened by a flawed prompt and ra...
-
Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference
Automatic LLM rankers align well with humans on broad leaderboards but degrade sharply when ranking close-performing models, and instance-level judge accuracy does not predict system-level bencher quality.
-
C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation
C2LEVA is a bilingual, multi-task LLM benchmark that combines passive test-data renewal with active data watermarking to reduce contamination risk, and ranks 15 models.
-
Challenges in Trustworthy Human Evaluation of Chatbots
Open chatbot leaderboards like Chatbot Arena can have their model rankings moved by several positions with only 10% low-quality or adversarial votes.
-
LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
LARPO, an iterative preference optimization method that adapts information retrieval techniques such as listwise ranking losses, hard negatives, and candidate lists, is claimed to substantially improve LLM alignment o...
Discussion (0). Continue with ORCID to comment.