From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

Banghua Zhu; Evan Frick; Ion Stoica; Joseph E. Gonzalez; Lisa Dunlap; Tianhao Wu; Tianle Li; Wei-Lin Chiang

arxiv: 2406.11939 · v2 · pith:KWEAOFPEnew · submitted 2024-06-17 · 💻 cs.LG · cs.AI· cs.CL

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

Tianle Li , Wei-Lin Chiang , Evan Frick , Lisa Dunlap , Tianhao Wu , Banghua Zhu , Joseph E. Gonzalez , Ion Stoica This is my paper

Pith reviewed 2026-05-23 23:58 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL

keywords benchmark curationLLM evaluationArena-Hard-AutoBenchBuilderLLM-as-a-Judgecrowdsourced datamodel separationhuman preference alignment

0 comments

The pith

BenchBuilder automates extraction of 500 hard prompts from crowdsourced data to produce a benchmark with three times the model separation of MT-Bench and 98.6 percent correlation to human rankings at twenty dollars cost.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to address the lag between fast LLM progress and slow manual benchmark creation by introducing BenchBuilder, an LLM-driven pipeline that pulls challenging open-ended prompts from large crowdsourced collections such as Chatbot Arena and WildChat-1M. A sympathetic reader would value this because it promises ongoing, low-cost benchmark refreshes that keep evaluations aligned with human judgments and better able to distinguish model capabilities. The authors apply the pipeline to generate Arena-Hard-Auto and introduce metrics for separation power and human alignment to validate the result. They report that the new benchmark outperforms MT-Bench on separation while matching human preferences at high correlation and minimal expense.

Core claim

BenchBuilder is an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large crowd-sourced datasets such as Chatbot Arena and WildChat-1M and then uses an LLM-as-a-Judge for automatic model evaluation. When applied, it yields Arena-Hard-Auto consisting of 500 challenging prompts. This benchmark delivers three times higher separation of model performances than MT-Bench, reaches 98.6 percent correlation with human preference rankings, and can be produced for twenty dollars.

What carries the argument

BenchBuilder pipeline, which uses LLMs both to select challenging prompts from crowdsourced data and to judge model responses automatically.

Load-bearing premise

An LLM acting as judge can produce model rankings that stay aligned with human preferences without introducing systematic bias from the judge model itself.

What would settle it

A side-by-side human evaluation on the same 500 Arena-Hard-Auto prompts that produces model rankings differing substantially from the LLM-judge rankings.

Figures

Figures reproduced from arXiv: 2406.11939 by Banghua Zhu, Evan Frick, Ion Stoica, Joseph E. Gonzalez, Lisa Dunlap, Tianhao Wu, Tianle Li, Wei-Lin Chiang.

**Figure 2.** Figure 2: BenchBuilder Pipeline. Starting with a live data source of crowdsourced user prompts, we first cluster their embeddings to form topic clusters. An LLM annotator then assigns quality scores based on the required skills. Clusters with low quality scores are filtered out, and we sample from the remaining high-quality clusters to create a diverse and challenging dataset of benchmark prompts. The final agreemen… view at source ↗

**Figure 3.** Figure 3: Win-rate of three model pairs (GPT-4-0613 vs Llama-2-70b-chat, Claude-3-Sonnet [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 5.** Figure 5: Comparison between Arena-HardAuto (Green) and MT-Bench (Grey). The former offers significantly better separability between models and tighter confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

**Figure 6.** Figure 6: A more complete selection of mean scores of various topic clusters in descending order. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

read the original abstract

The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

BenchBuilder gives a workable automated pipeline for turning crowd data into hard LLM benchmarks, but the headline numbers rest on details that need checking.

read the letter

The main thing to know is that the paper describes BenchBuilder, an LLM-driven pipeline that pulls challenging open-ended prompts from crowd-sourced sources like Chatbot Arena and WildChat, then uses an LLM judge to score models. They release Arena-Hard-Auto (500 prompts) and report 3x better model separation than MT-Bench plus 98.6% correlation with human Arena votes, all at roughly $20 cost. This is a concrete, practical step on a real problem: keeping evaluation sets current as models advance without constant manual work. The new alignment and separation metrics are a useful addition for judging whether a benchmark is actually good. The automation without human-in-the-loop is the clearest advance over prior arena-style and LLM-judge work. The approach is straightforward and the released benchmark is immediately usable. The soft spots sit in the validation. The abstract states the numbers cleanly, but the provided text gives no concrete filtering rules, judge prompt, judge model identity, or ablation results. That makes it hard to judge whether the separation and correlation are independent or partly driven by overlapping biases between the judge and the original crowd data. The stress-test concern about shared length or style preferences is reasonable to raise until those details appear. If the full paper has held-out judges or sensitivity checks, that would strengthen the claims; right now the evidence is thin on that axis. This paper is for people who build or maintain LLM evaluation infrastructure and want scalable ways to refresh test sets. It shows honest engagement with the practical constraints and cites the relevant prior work on arenas and judges. It deserves a serious referee because the problem matters and the method is new enough to warrant review, even if the experiments will likely need expansion and clarification.

Referee Report

3 major / 2 minor

Summary. The paper introduces BenchBuilder, an LLM-driven pipeline to automatically curate challenging open-ended prompts from large crowd-sourced datasets (Chatbot Arena, WildChat-1M). It releases Arena-Hard-Auto, a 500-prompt benchmark evaluated via LLM-as-a-Judge, claiming 3× higher model separation than MT-Bench and 98.6% correlation with human preference rankings from Chatbot Arena, all at ~$20 cost. The work positions this as a scalable, human-free framework for ongoing benchmark creation.

Significance. If the separation and correlation metrics are robustly supported, the pipeline offers a concrete, low-cost route to continuously refreshed benchmarks that track human preferences more closely than static suites like MT-Bench. The public release of Arena-Hard-Auto and the curation code would constitute a reusable artifact for the community.

major comments (3)

[§4] §4 (Experiments) and the metric definitions: the separation metric yielding the '3×' claim is not accompanied by its exact formula, variance estimates, or the precise MT-Bench baseline numbers used for the ratio; without these, the factor-of-three improvement cannot be independently verified from the reported tables.
[LLM-as-a-Judge subsection] LLM-as-a-Judge subsection: no ablation is presented that swaps the judge model (or holds it out from the evaluated model pool) while recomputing the 98.6% correlation; this leaves open whether the reported alignment partly reflects shared biases between judge and the original crowd-sourced votes rather than independent validation.
[BenchBuilder pipeline description] BenchBuilder pipeline description: the prompt-filtering and difficulty-ranking steps lack explicit exclusion rules, temperature settings, and the precise prompt templates fed to the curator LLM, rendering the 500-prompt extraction non-reproducible from the stated data sources.

minor comments (2)

[Tables] Table captions should explicitly state the number of models and the exact human-vote subset used for the correlation computation.
[Cost analysis] The cost figure of $20 should be broken down by API calls (curator vs. judge) with token counts for transparency.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to improve clarity, reproducibility, and validation where the points are valid.

read point-by-point responses

Referee: §4 (Experiments) and the metric definitions: the separation metric yielding the '3×' claim is not accompanied by its exact formula, variance estimates, or the precise MT-Bench baseline numbers used for the ratio; without these, the factor-of-three improvement cannot be independently verified from the reported tables.

Authors: We agree that the separation metric requires an explicit formula, variance estimates, and the precise MT-Bench baseline values for independent verification. These details will be added to Section 4 of the revised manuscript. revision: yes
Referee: LLM-as-a-Judge subsection: no ablation is presented that swaps the judge model (or holds it out from the evaluated model pool) while recomputing the 98.6% correlation; this leaves open whether the reported alignment partly reflects shared biases between judge and the original crowd-sourced votes rather than independent validation.

Authors: This is a valid concern about potential bias in the correlation metric. We will add an ablation study in the revised manuscript that recomputes the correlation using a held-out judge model distinct from those involved in the original data collection. revision: yes
Referee: BenchBuilder pipeline description: the prompt-filtering and difficulty-ranking steps lack explicit exclusion rules, temperature settings, and the precise prompt templates fed to the curator LLM, rendering the 500-prompt extraction non-reproducible from the stated data sources.

Authors: We acknowledge that these implementation details are necessary for reproducibility. The revised Section 3 will include the explicit exclusion rules, temperature settings, and full prompt templates, with the code release updated accordingly. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims rest on external human validation

full rationale

The paper's central results (3x separation vs MT-Bench and 98.6% correlation with human rankings) are empirical measurements obtained by applying an LLM judge to curated prompts and then comparing the resulting model rankings against independent human preference votes from Chatbot Arena. These comparisons are defined externally to the BenchBuilder curation pipeline and do not reduce to fitted parameters or self-referential definitions within the paper. No equations or steps equate a derived quantity to its own inputs by construction, and the validation metrics are not produced by the same process that generates the benchmark. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the domain assumption that LLM-based curation and judging can substitute for human effort while preserving alignment; no free parameters or invented entities are described in the abstract.

axioms (1)

domain assumption LLM-as-a-Judge produces rankings that correlate strongly with human preferences
Invoked to justify both the automatic evaluation and the 98.6% correlation claim.

pith-pipeline@v0.9.0 · 5759 in / 1323 out tokens · 30015 ms · 2026-05-23T23:58:11.162398+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose a novel data curation pipeline, BenchBuilder, to automatically construct high-quality benchmarks from crowdsourced data... Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We leverage the LLM-as-a-Judge framework... Bradley & Terry model to produce model’s the final model scores

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 34 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Evaluating Large Language Models in Scientific Discovery
cs.AI 2025-12 unverdicted novelty 8.0

The SDE benchmark shows LLMs lag on scientific discovery tasks relative to general science tests, with diminishing scaling returns and shared weaknesses across models.
DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows
cs.AI 2026-05 unverdicted novelty 7.0

DecisionBench supplies a fixed task suite, model pool, delegation interface, and multi-axis metrics to evaluate emergent delegation, showing similar quality across awareness conditions but 15-31 point headroom under p...
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
cs.AI 2026-05 unverdicted novelty 7.0

SCICONVBENCH is a new benchmark evaluating LLMs on multi-turn disambiguation and inconsistency resolution for task formulation in computational science, with frontier models reaching only 52.7% success on fluid mechan...
Green Shielding: A User-Centric Approach Towards Trustworthy AI
cs.CL 2026-04 unverdicted novelty 7.0

Green Shielding introduces CUE criteria and the HCM-Dx benchmark to demonstrate that routine prompt variations systematically alter LLM diagnostic behavior along clinically relevant dimensions, producing Pareto-like t...
Sell More, Play Less: Benchmarking LLM Realistic Selling Skill
cs.CL 2026-04 conditional novelty 7.0

SalesLLM provides an automatic evaluation framework for LLM sales dialogues that correlates 0.98 with human experts and shows top models approaching human performance while weaker ones lag.
Convex Optimization for Alignment and Preference Learning on a Single GPU
cs.LG 2026-05 unverdicted novelty 6.0

COALA applies convex optimization reformulations of neural networks to direct preference optimization, claiming single-GPU training with ~18% of DPO's TFLOPs and competitive performance on multiple datasets and models...
General Preference Reinforcement Learning
cs.LG 2026-05 unverdicted novelty 6.0

GPRL carries k-dimensional skew-symmetric preference structure into policy updates via per-dimension advantages and context-dependent eigenvalues, yielding 56.51% length-controlled win rate on AlpacaEval 2.0 from Llam...
General Preference Reinforcement Learning
cs.LG 2026-05 unverdicted novelty 6.0

GPRL carries a k-dimensional skew-symmetric preference structure into policy updates with per-dimension advantages and a drift monitor, yielding 56.51% length-controlled win rate on AlpacaEval 2.0 from Llama-3-8B-Inst...
General Preference Reinforcement Learning
cs.LG 2026-05 unverdicted novelty 6.0

GPRL applies a k-dimensional preference model with per-dimension normalized advantages and a drift monitor to LLM post-training, reporting 56.51% length-controlled win rate on AlpacaEval 2.0 and gains on other benchma...
Evaluating Multi-turn Human-AI Interaction
cs.HC 2026-05 unverdicted novelty 6.0

Introduces the TCR framework to evaluate educational LLM assistants on transparency, consistency, and refinement in multi-turn interactions, complementing aggregate metrics.
Dynamic Model Merging Made Slim
cs.LG 2026-05 unverdicted novelty 6.0

DiDi-Merging achieves dynamic model merging performance matching or exceeding prior methods while using only 1.24x to 1.4x the parameters of a single fine-tuned model.
FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
cs.CL 2026-05 unverdicted novelty 6.0

FINESSE-Bench is a new hierarchical benchmark suite combining certification-style exams, trading tasks, and a Russian olympiad set to evaluate LLMs on financial competencies at multiple difficulty levels.
FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
cs.CL 2026-05 unverdicted novelty 6.0

FINESSE-Bench is a hierarchical benchmark suite of eight datasets with 3,993 questions for evaluating LLMs on financial domain knowledge, technical analysis, and professional competencies.
From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs
cs.CL 2026-05 unverdicted novelty 6.0

LogiHard hardens reasoning benchmarks by transforming 0-order selection into 2-order judgment, causing 31-56% accuracy drops in 12 frontier LLMs and a 47% drop on zero-shot MMLU, revealing a combinatorial reasoning ga...
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
cs.AI 2026-05 unverdicted novelty 6.0

Deployment-relevant AI alignment cannot be inferred from model-level evaluations alone, as benchmark audits show missing interaction support and cross-model tests reveal model-dependent scaffold effects.
Hybrid Policy Distillation for LLMs
cs.CL 2026-04 unverdicted novelty 6.0

Hybrid Policy Distillation unifies existing knowledge distillation methods for LLMs into a reweighted log-likelihood objective and introduces a hybrid forward-reverse KL approach with mixed data sampling to improve st...
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
cs.CL 2025-06 unverdicted novelty 6.0

LongWriter-Zero applies RL from a base model with specialized rewards for length, quality, and structure to outperform SFT baselines and larger models on long-writing benchmarks.
Process Reinforcement through Implicit Rewards
cs.LG 2025-02 conditional novelty 6.0

PRIME enables online process reward model updates in LLM RL using implicit rewards from rollouts and outcome labels, yielding 15.1% average gains on reasoning benchmarks and surpassing a stronger instruct model with 1...
Qwen2.5-1M Technical Report
cs.CL 2025-01 accept novelty 6.0

Qwen2.5-1M models reach 1M token context with improved long-context performance, no short-context loss, and 3-7x prefill speedup via open inference optimizations.
LoCar: Localization-Aware Evaluation of In-Vehicle Assistants through Fine-Grained Sociolinguistic Control
cs.CL 2026-05 unverdicted novelty 5.0

LoCar is a localization-aware evaluation framework for in-vehicle assistants that identifies unstable Korean honorific control and weaker performance on strategic metrics like clarification and proactivity in current LLMs.
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
cs.CL 2026-05 conditional novelty 5.0

EngGPT2MoE-16B-A3B matches or exceeds other Italian open-source LLMs on most international benchmarks while remaining competitive on ITALIC, though it trails some top international models.
Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
cs.AI 2026-04 unverdicted novelty 5.0

An LLM-as-a-judge evaluation framework for math reasoning outperforms symbolic methods by accurately assessing diverse answer representations and formats.
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
cs.SE 2026-04 unverdicted novelty 5.0

LLM judges for code tasks show high sensitivity to prompt biases that systematically favor certain options, changing accuracy and model rankings even when code is unchanged.
Proximal Supervised Fine-Tuning
cs.LG 2025-08 unverdicted novelty 5.0

PSFT modifies supervised fine-tuning by incorporating trust-region ideas from RL to constrain policy changes, yielding better out-of-domain generalization in math and human-value tasks without entropy collapse.
Kimi K2: Open Agentic Intelligence
cs.LG 2025-07 unverdicted novelty 5.0

Kimi K2 is a 1-trillion-parameter MoE model that leads open-source non-thinking models on agentic benchmarks including 65.8 on SWE-Bench Verified and 66.1 on Tau2-Bench.
Qwen3 Technical Report
cs.CL 2025-05 unverdicted novelty 5.0

Pith review generated a malformed one-line summary.
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
cs.CL 2024-08 unverdicted novelty 5.0

WildFeedback extracts preference pairs from in-situ user feedback in LLM conversations to fine-tune models for better alignment with real user preferences.
Submodular Benchmark Selection
cs.AI 2026-05 unverdicted novelty 4.0

Submodular maximization under a Gaussian model selects small benchmark subsets that outperform random selection for imputing leaderboard scores, with mutual information better than entropy at small sizes.
Ministral 3
cs.CL 2026-01 unverdicted novelty 4.0

Ministral 3 releases 3B/8B/14B parameter-efficient language models with base, instruction, and reasoning variants derived via iterative pruning and distillation, including image understanding capabilities.
Phi-4-reasoning Technical Report
cs.AI 2025-04 unverdicted novelty 4.0

A 14B reasoning model trained via supervised fine-tuning on selected prompts and o3-mini traces, plus outcome RL, outperforms larger open models like DeepSeek-R1-Distill-Llama-70B on math, coding, planning and related...
Reinforcement Learning for Scalable and Trustworthy Intelligent Systems
cs.LG 2026-05 unverdicted novelty 3.0

Reinforcement learning is advanced for communication-efficient federated optimization and for preference-aligned, contextually safe policies in large language models.
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
cs.CL 2026-05 unverdicted novelty 3.0

EngGPT2MoE-16B-A3B matches or beats other Italian models on most international benchmarks but trails top international models such as GPT-5 nano and Qwen3-8B.
Sustainability via LLM Right-sizing
cs.CL 2025-04 unverdicted novelty 3.0

Empirical comparison shows smaller open-weight LLMs achieve strong performance on everyday work tasks, supporting task-aware selection over always using the largest models for sustainability and cost reasons.
Qwen2.5 Technical Report
cs.CL 2024-12 unverdicted novelty 3.0

Qwen2.5 LLMs scale pre-training data to 18 trillion tokens and apply multistage reinforcement learning, achieving competitive performance on benchmarks with models up to 5 times larger.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 30 Pith papers · 2 internal anchors

[1]

InternLM2 Technical Report

AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zon...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/d15-1075 2024
[2]

Holistic Evaluation of Language Models

Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.97. URL https://aclanthology.org/2021.emnlp-main.97. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Ro...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2021.emnlp-main.97 2021
[3]

Specificity: Does the prompt ask for a specific, well-defined output without leaving any ambiguity? This allows the AI to demonstrate its ability to follow instructions and generate a precise, targeted response

work page
[4]

Domain Knowledge: Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains? The prompt must demand the AI to have a strong prior knowledge or mastery of domain- specific concepts, theories, or principles

work page
[5]

Complexity: Does the prompt have multiple components, variables, or levels of depth and nuance? This assesses the AI’s capability to handle complex, multi-faceted problems beyond simple queries

work page
[6]

Problem-Solving: Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution? Note active problem-solving is not simply reciting facts or following a fixed set of instructions

work page
[7]

Creativity: Does the prompt require a creative approach or solution? This tests the AI’s ability to generate novel ideas tailored to the specific needs of the request or problem at hand

work page
[8]

Technical Accuracy: Does the prompt require an answer with a high degree of technical accuracy, correctness and precision? This assesses the reliability and truthfulness of the AI’s outputs

work page
[9]

Criteria Satisfied: [1, 2, 4, 6, 7]

Real-World Application: Does the prompt relate to real-world applications? This tests the AI’s ability to provide practical and actionable information that could be implemented in real-life scenarios. After analyzing the input prompt based on these criteria, you must list the criteria numbers that the prompt satisfies in the format of a Python array. For ...

work page
[10]

Assistant A is significantly better: [[A>>B]]

work page
[11]

Assistant A is slightly better: [[A>B]]

work page
[12]

Tie, relatively the same: [[A=B]]

work page
[13]

Assistant B is slightly better: [[B>A]]

work page
[14]

My final verdict is tie: [[A=B]]

Assistant B is significantly better: [[B>>A] Example output: "My final verdict is tie: [[A=B]]". 24 D A DDITIONAL EXAMPLES User: Use ABC notation to write a melody in the style of a folk tune. GPT-4: X:1 T:Untitled Folk Tune M:4/4 L:1/8 K:G|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|A2B2|c2B2|A2F2|E4|D2E2|F2E2 |D2B,2|C4:| GPT-3.5-Turbo: Sure! Here’s a simple m...

work page

[1] [1]

InternLM2 Technical Report

AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zon...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/d15-1075 2024

[2] [2]

Holistic Evaluation of Language Models

Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.97. URL https://aclanthology.org/2021.emnlp-main.97. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Ro...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2021.emnlp-main.97 2021

[3] [3]

Specificity: Does the prompt ask for a specific, well-defined output without leaving any ambiguity? This allows the AI to demonstrate its ability to follow instructions and generate a precise, targeted response

work page

[4] [4]

Domain Knowledge: Does the prompt test the AI’s knowledge and understanding in a specific domain or set of domains? The prompt must demand the AI to have a strong prior knowledge or mastery of domain- specific concepts, theories, or principles

work page

[5] [5]

Complexity: Does the prompt have multiple components, variables, or levels of depth and nuance? This assesses the AI’s capability to handle complex, multi-faceted problems beyond simple queries

work page

[6] [6]

Problem-Solving: Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution? Note active problem-solving is not simply reciting facts or following a fixed set of instructions

work page

[7] [7]

Creativity: Does the prompt require a creative approach or solution? This tests the AI’s ability to generate novel ideas tailored to the specific needs of the request or problem at hand

work page

[8] [8]

Technical Accuracy: Does the prompt require an answer with a high degree of technical accuracy, correctness and precision? This assesses the reliability and truthfulness of the AI’s outputs

work page

[9] [9]

Criteria Satisfied: [1, 2, 4, 6, 7]

Real-World Application: Does the prompt relate to real-world applications? This tests the AI’s ability to provide practical and actionable information that could be implemented in real-life scenarios. After analyzing the input prompt based on these criteria, you must list the criteria numbers that the prompt satisfies in the format of a Python array. For ...

work page

[10] [10]

Assistant A is significantly better: [[A>>B]]

work page

[11] [11]

Assistant A is slightly better: [[A>B]]

work page

[12] [12]

Tie, relatively the same: [[A=B]]

work page

[13] [13]

Assistant B is slightly better: [[B>A]]

work page

[14] [14]

My final verdict is tie: [[A=B]]

Assistant B is significantly better: [[B>>A] Example output: "My final verdict is tie: [[A=B]]". 24 D A DDITIONAL EXAMPLES User: Use ABC notation to write a melody in the style of a folk tune. GPT-4: X:1 T:Untitled Folk Tune M:4/4 L:1/8 K:G|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|A2B2|c2B2|A2F2|E4|D2E2|F2E2 |D2B,2|C4:| GPT-3.5-Turbo: Sure! Here’s a simple m...

work page