Dynamically Allocating Evaluation Effort for Model Ranking

Alon Lavie; Julia Kreutzer; Matt Post; Mrinmaya Sachan; Ond\v{r}ej Bojar; Tom Kocmi; Vil\'em Zouhar

REVIEW 4 major objections 6 minor 9 references

Dynamically Allocating Evaluation Effort for Model Ranking

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that when the goal is a top-weighted model ranking, the optimal way to spend a fixed evaluation budget is to sample models in proportion to the square root of their rank-based importance, which with inverse-squared-rank…

desk verdict The empirical allocation result is likely real, but Theorem 3's optimality proof has an algebra error, so the paper is a solid empirical contribution with an unproven headline theorem. read the letter →

arxiv 2608.03437 v1 pith:DUK57DYZ submitted 2026-08-04 cs.CL cs.LG

Vilém Zouhar , Julia Kreutzer , Alon Lavie , Tom Kocmi , Matt Post , Ondřej Bojar , Mrinmaya Sachan This is my paper · ORCID

classification cs.CLcs.LG

keywords humanevaluationbest-armidentificationmulti-armedbanditadaptiveallocationmodelrankingrank-weightedcorrelationtranslationinverse-ranksampling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human evaluation of many competing models is expensive because standard protocols annotate every model on every item. The paper reframes the choice of which model to evaluate next as a best-arm identification problem in a multi-armed bandit, where pulling an arm means spending one annotation on that model. Its central theoretical result is that, under a Gaussian noise model, sampling models proportionally to the square root of their rank-weight minimizes weighted ranking uncertainty; with the weights the paper recommends, this reduces to sampling inversely to the model's current estimated rank. The same framework proves consistency of such policies as the budget grows, and shows that the unrestricted optimization is NP-hard, so the paper turns to adaptive heuristics. Simulation on large-scale translation evaluation data indicates that inverse-rank sampling recovers top-model rankings more accurately than uniform sampling and approaches an oracle that sees all scores.

What carries the argument

The load-bearing identity is the variance-allocation formula: with sample means of variance $\sigma^2/|R_m|$, the weighted variance loss $\sum_m \omega_m^2 \sigma^2/|R_m|$ is minimized under a budget by setting $|R_m|$ proportional to $\omega_m$ (the square root of the importance weight). The paper proves this by Lagrangian differentiation (Theorem 3). The other piece of machinery is the weighted rank correlation $\tau_\omega$, a modified rank-correlation score in which disagreements involving high-weight models count more, which lets the paper define the objective as 'get the top of the leaderboard right' and choose $\omega_m = 1/\text{rank}^2_m$. The additive noise model $r_{x,m} = q_m + d_x + \varepsilon_{x,m}$ then supports a linear estimator that removes bias when evaluation items are ordered by difficulty rather than at random.

What would settle it

Compute the per-model evaluation variance on a benchmark and solve the Lagrangian minimization with those individual variances; if the resulting optimal allocation differs materially from inverse-rank sampling on the same data, then the theorem's recommendation is not the optimum once the equal-variance premise is relaxed.

Watch

Extended reading notes

Core claim

The paper's central claim is that evaluation effort should be allocated according to the square root of the importance weight assigned to each model's rank position, and that for weights $\omega_m = 1/\text{rank}^2_m$ this means sampling the next model to evaluate proportionally to $1/\text{rank}_m$. This is established as Theorem 3: assuming the estimated mean score of each model is an independent Gaussian with variance $\sigma^2/|R_m|$, the Lagrange-optimal allocation that minimizes the weighted sum $\sum_m \omega_m^2 \sigma^2/|R_m|$ under a fixed budget is $|R_m| \propto \omega_m$. Since uniform sampling is recovered when all weights are equal, the theorem places the common baseline as the flat-weight case of the same rule. The paper further proves that any policy that never lets a model's selection probability drop below a positive constant will converge to the true ranking (Theorem 2), and that exact optimal allocation remains NP-hard even with oracle access (Theorem 1). Because the equal-variance assumption fails on real translation data, the paper adds a confusion-minimization algorithm that uses per-model variance estimates to maximize pairwise ranking certainty.

Load-bearing premise

The optimality result depends on the assumption that every model's estimated mean score is an independent Gaussian with the same variance, so that shifting samples toward lower-ranked models provably buys the most ranking certainty; the paper's own measurements show per-model variances differ, which is why the authors introduce a separate confusion-minimization heuristic.

Editorial extensions

If this is right

With inverse-rank sampling, annotation campaigns can concentrate human effort on the few models that compete for the top spots, cutting the budget needed to identify the best model.
Uniform sampling, the current baseline, is exactly the optimal policy when all rank positions are weighted equally, so the framework generalizes rather than replaces standard practice.
The consistency theorem guarantees that adaptive allocation eventually recovers the true ranking as the budget grows, so the method is safe to use at any scale.
The same allocation logic transfers to automatic evaluation where each model-output judgment costs compute or API calls rather than annotator time.
Dynamic model selection can be combined with smart item ordering (e.g., difficulty-prioritized items) when score estimates are corrected with an additive difficulty model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The inverse-rank rule implies an expected evaluation count that falls off as $1/\text{rank}$; a direct extension would be a closed-form budget estimator that predicts, before annotation starts, how much budget is needed to retrieve the top-$k$ set at a target probability.
Because the optimality proof assumes equal variance across models while the paper's own data show better models have lower variance, a variance-aware analogue (allocating $|R_m| \propto \sigma_m \omega_m$) would be the natural next theorem and could outperform $1/\text{rank}$ sampling on real data.
The introductory 95%-at-40%-budget claim is framed as a retrieval probability, while the reported experiments use averaged weighted rank correlation; a dedicated experiment measuring exact top-3 retrieval probability would directly test the practical promise.
The additive quality-plus-difficulty model assumes a single quality per model; in settings where a model's relative quality varies by item type (e.g., language pairs), the same bandit policy could be run per cluster of items to produce specialized rankings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The empirical core of this paper deserves to be taken seriously. Running rank-based adaptive allocation on 290k WMT human evaluations is a sensible and reproducible way to ask whether we can spend less annotation budget on inferior models and still recover the top of the ranking. The results in Figures 5-6 and Tables 1-3 are plausible: rank sampling and confusion minimization beat uniform sampling under the weighted Kendall tau objective, and they approach the greedy oracle. That is a useful, concrete finding for benchmark organizers, and the paper ships code and uses public WMT data, which makes it verifiable. The synthetic extensions are also a good check on the limits of the WMT simulations. The problem is the central theorem. Theorem 3 states that sampling according to sqrt(omega_m) minimizes weighted ranking uncertainty. The proof does not establish that. The Lagrange step is internally inconsistent: if the surrogate loss is sum omega_m^2 sigma^2 / |R_m|, stationarity gives |R_m| proportional to omega_m, not sqrt(omega_m). The proof then quietly uses a denominator sum of sqrt(omega_k) and concludes sqrt(omega_m). Getting sqrt(omega_m) requires a surrogate loss linear in omega_m, not quadratic. The proof also never shows that minimizing this variance proxy minimizes the expected weighted Kendall tau from the paper's own Equation 6. So as written, Theorem 3 is not proved. The assumptions behind the theorem are also in tension with the paper's own model. Lemma 2 explicitly says items share difficulty, so model estimators are correlated; and Appendix D Figure 11 shows variance decreases for stronger models, contradicting the common-sigma assumption. The authors acknowledge this and introduce confusion minimization to cope, which is honest and reasonable, but it means the optimality theorem cannot explain why rank sampling works in their own setting. The practical algorithm may still be right, but the proof is not. There are smaller soft spots worth naming. The introduction's claim that top-3 among 20 models can be recovered with 95% probability at 40% budget is not tied to any figure or table. Table 2 shows the gains over uniform largely shrink for unweighted Kendall's tau, which is relevant to practitioners who care about full rankings. The warmup and exploration hyperparameters are numerous, though the main trends look stable. Who should read this? Evaluation researchers, WMT/IWSLT organizers, and anyone allocating human annotation budget for model comparison. It is a useful empirical study with a broken theoretical badge. I would send it to peer review rather than desk reject, with the expectation that the authors fix Theorem 3, connect or remove the 95%/40% claim, and discuss metric dependence. With those repairs it is publishable; without them, the theoretical selling point is unsupported but the empirical contribution likely survives.

Referee Report

4 major / 6 minor

Summary. The paper proposes a bandit-style framework for allocating human evaluation effort across competing NLP models, with the goal of recovering top-model rankings under a fixed budget. It formalizes evaluation allocation as best-arm identification, proves NP-hardness of the oracle allocation problem, and proposes several sampling policies, including rank-weighted sampling and a confusion-minimization policy. The main theoretical claim is Theorem 3, which states that sampling a model with probability proportional to the square root of its objective weight is optimal for minimizing weighted ranking uncertainty. The empirical section simulates these policies on 290k WMT human evaluations across 37 campaigns, reporting that rank-based sampling and confusion minimization improve weighted Kendall tau over uniform allocation and approach the greedy oracle.

Significance. If the theoretical and empirical claims held, the paper would offer a practical recipe for reducing the cost of large-scale human evaluation: focus annotation effort on competitive models while preserving top-k ranking fidelity. The empirical study is substantial and well-grounded: 100-seed simulations on real WMT data, comparison against a greedy oracle, ablation over item ordering, and synthetic robustness checks. The paper also ships code, which aids reproducibility. However, the central optimality theorem is not established by the proof as written, and the headline 95%/40% claim in the introduction is unsupported by the reported experiments. The practical heuristic value of rank-based sampling may survive, but the paper's core theoretical contribution currently does not.

major comments (4)

[Appendix B, Theorem 3, Eqs. (43)-(49)] The Lagrange step in the proof of Theorem 3 is algebraically inconsistent. Stationarity in Eq. (45) gives |R_m|^2 = ω_m²σ²/λ, hence |R_m| = ω_m σ/√λ. Substituting into the budget constraint yields |R_m|* = B·ω_m / Σ_k ω_k, i.e. proportional to ω_m, not √ω_m. The conclusion |R_m| ∝ √ω_m only follows if the loss in Eq. (43) were Σ_m ω_m σ²/|R_m| rather than Σ_m ω_m² σ²/|R_m|. As written, the proof does not prove the stated theorem.
[Section 3.2 and Appendix B, Theorem 3] The proof replaces the weighted Kendall tau objective of Eq. (6) with the surrogate loss L = Σ_m ω_m² σ²/|R_m| without establishing that minimizing this surrogate minimizes the expected value of τ_ω. Weighted Kendall tau is a nonlinear, pairwise function of the estimated means, so reducing individual variances need not reduce the expected number of weighted inversions. Without an explicit bound or monotonicity argument connecting L to E[τ_ω], the optimality claim concerns only a variance proxy, not the stated ranking objective.
[Appendix B, Eq. (40); Appendix D, Fig. 11; Section 4.2] The assumptions behind Theorem 3 — independent estimators with common variance σ² across models — are contradicted by the paper's own correlated-arms model (Lemma 2, Eq. (29)) and by the empirical heteroscedasticity reported in Figure 11. The paper concedes this in Section 4.2 when it introduces Confusion Minimization specifically to address the violation. Consequently, Theorem 3 cannot justify the claimed optimality of inverse-rank sampling for the WMT experiments; at best the algorithm is an empirically motivated heuristic under those conditions.
[Section 1 and Section 5] The introduction claims that the ordering of the top 3 among 20 translation models can be reliably (p=95%) retrieved with only 40% of the budget, but no figure or table reports this quantity. Figure 5 shows average weighted Kendall tau, which is not a retrieval probability, and no top-3 hit-rate result appears elsewhere in the provided text. This headline claim should either be removed or supported with the corresponding experiment, e.g. top-3 retrieval frequency over the 100 seeds.

minor comments (6)

[Section 3.4] The word 'concider' should be 'consider'; similar typographical errors appear elsewhere (e.g. 'Bolzmann' in Section 4.1 and Table 2, 'Rearrranging' in Eq. (46), 'Experimnents' in Limitations, 'cummulative' in Section 4.2).
[Appendix B, Lemma 1] The proof of Lemma 1 contains a circular equation: it writes Var(μ̂_repeat) = Var(μ̂_repeat) + Cov(r1,r2), which is not a comparison. The argument should directly show Var(μ̂_repeat) = Var(μ̂_distinct) + positive covariance terms.
[Figure 5] The caption says numbers in the legend show areas under the curves, but no numerical AUC values are visible in the provided figure. Please clarify whether these are omitted or intended to appear in the legend.
[Table 2] The rows 'Greedy oracle rank1' and 'Greedy oracle rank top 3' are not defined in the text; please specify which objective each oracle variant optimizes.
[Section 4.3, footnote 5] The statement that the greedy objective is not submodular and can be arbitrarily worse than optimum is important and should be moved into the main text rather than relegated to a footnote.
[Section 5.1] The text says 'Numbers in legend show areas under curves' but also refers to 'Other algorithms are included in Appendix C.1'; consider adding explicit pointers to Table 2 when discussing Figure 5 and Table 1.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 3's optimality proof optimizes a surrogate variance loss with weights ω²_m and contains an algebraic jump (Σω_k replaced by Σ√ω_k), so the claimed √ω-optimality of rank sampling is constructed from the surrogate, not derived from the τ_ω objective.

self definitional [Sec. 4.1 (Eq. 12) and Appendix B, Theorem 3 proof (Eqs. 40–49)]
"Sampling according to √ω_m is the optimal strategy for minimizing the weighted (according to ω_m) ranking uncertainty. ... we define a differentiable loss function that proxies the discrete ranking metric τ_ω ... ∑_{m∈ℳ} ω²_m ⋅ Var(μ̂_m) = ∑_{m∈ℳ} ω²_m σ²/|R_m| ... |R_m|* = (B/Σ_k ω_k) ω_m ... The optimal number of samples |R_m|* is then proportional to the square root of the importance weight ω_m. ... Notably, this intentionally bears resemblance to weights in optimization objective (Equation 6)."

The paper's stated objective is the weighted Kendall tau of Eq. 6, whose pairwise terms are weighted by ω_i·ω_j. The proof never connects the surrogate loss Eq. 43 to Eq. 6; it instead minimizes Σ ω²_m σ²/|R_m|. The Lagrange stationarity condition (Eqs. 45–46) yields |R_m| ∝ ω_m, and Eq. 49 states |R_m|* = B·ω_m/Σ_k ω_k. The theorem statement turns this into 'sampling according to √ω_m' only by calling ω_m² the importance weight, so |R_m| ∝ ω_m = √(ω_m²), and by the erroneous step Eq. 48 where Σ_k ω_k is replaced with Σ_k √ω_k. With the paper's actual choice ω_m = 1/rank², the algebra would give sampling proportional to 1/rank², not the advertised 1/rank.

full rationale

The main theoretical claim of the paper is Theorem 3, 'Sampling according to √ω_m is the optimal strategy for minimizing the weighted (according to ω_m) ranking uncertainty.' On inspection, the proof does not derive this from the weighted Kendall tau objective defined in Eq. 6. It introduces a differentiable proxy loss with squared weights ω²_m (Eq. 43) and minimizes that proxy under a budget. The solution of that minimization is |R_m| ∝ ω_m (Eqs. 46 and 49), which is already different from the theorem's √ω_m conclusion; the proof bridges the gap with an algebraic inconsistency in Eq. 48 (substituting Σ√ω_k for Σω_k). The conclusion 'proportional to the square root of the importance weight' only works if the importance weight in the surrogate is ω_m², so the theorem's content is equivalent to the definition of the surrogate and the square-notation, not to any property of the paper's actual ranking objective. This is a self-definitional circularity in the central theoretical result: the optimality of inverse-rank sampling is constructed from the surrogate loss chosen after the fact, and the proposed sampler in Eq. 12 is explicitly said to 'intentionally bear resemblance' to the objective weights. The empirical evaluation on WMT data is self-contained and does not reduce to a fitted prediction; no load-bearing self-citation chain or imported uniqueness theorem appears. The introduction's 95%/40% claim is not connected to a displayed figure or table, but that is an unsupported-claim issue rather than circularity. Overall, the empirical contribution may stand, but the paper's advertised proof of optimality is circular in the sense that the theorem's conclusion is built into the surrogate objective and the notation.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central claim rests mainly on (a) an additive item-model error model, (b) unbiased-ordering and constant-cost assumptions that are relaxed in the experiments, and (c) an independence/homoscedasticity noise assumption for the optimality theorem that the paper itself shows is violated. The optimality theorem's loss function is additionally mis-specified relative to its conclusion. No new physical or formal entities are introduced; the linear estimator and confusion-minimization objective are algorithms, and correlated arms are inherited from Gupta et al. (2021).

free parameters (6)

warmup items C = 5
All algorithms evaluate every model on the first C=5 items before adaptive allocation starts (Section 5.1); the value is stated once and not swept.
epsilon-greedy epsilon = 0.5
Fixed ε = 50% exploration for the epsilon-greedy variant (Section 5.1); an unusually high exploration rate, not tuned per campaign.
Boltzmann temperature = 1
Fixed temperature of 1 for Boltzmann sampling (Section 5.1).
UCB exploration constant gamma = sqrt(2)
Fixed γ = √2 for the UCB variant (Section 5.1).
ranking objective weight omega_m = 1/rank^2
The weighted Kendall tau objective uses ω_m = 1/rank² for the main experiments (Section 3.2). The paper's headline gains depend on this hand-chosen weighting, and by the logic of Theorem 3 it also fixes the optimal sampler (1/rank).
synthetic data generative parameters = mu=0.7, sigma=0.25, d ~ N(0,1)
The synthetic experiments (Appendix C.4) fix the generative process for all four conditions; these are simulation settings, not fitted to the target claim, but the claimed scale-up beyond WMT is only shown under these choices.

assumptions (7)

domain assumption Additive item-model score model: r_{x,m} = q_m + d_x + ε_{x,m}, with d_x and ε_{x,m} independent noise terms.
Introduced in Section 3.4 (Eq. 7) to justify the linear estimator that lifts Assumptions 1 and 2, and used in Lemmas 1 and 2 in Appendix B. Standard in psychometrics (Cronbach et al.), but if item effects are not additive, the 'same items for all models' variance argument in Lemma 2 weakens.
domain assumption Assumption 1: the item ordering ≺ is unbiased for the basic theorems.
Section 3.1 states the running-mean estimator is only unbiased under unbiased item ordering. Theorem 2 and the implementation of Theorem 3 depend on this; the paper later relaxes it via the linear estimator.
domain assumption Assumption 2: constant evaluation cost, cost(x) = 1 for all items.
Section 3.1 assumes constant cost; the paper relaxes it later by ordering items by utility per cost, but the main experiments treat cost as uniform.
ad hoc to paper Estimated model means are independent Gaussians with common variance: μ̂_m ~ N(μ_m, σ²/|R_m|) with the same σ for all models.
Eq. 40 in the proof of Theorem 3. Independence contradicts the correlated-arms setup the paper itself adopts (shared item difficulty, Lemma 2), and homoscedasticity is contradicted by Appendix D, Figure 11. This is the load-bearing premise for the optimality claim.
domain assumption Evaluating a model on its next unevaluated item from a shared global ordering is lossless (Appendix A, Observations 1 and 2).
Appendix A reduces model-plus-item selection to model-only selection by forbidding repeated annotations and enforcing overlapping item sets; Lemma 2 justifies overlap, but real campaigns may not satisfy the subset-containment condition that maximizes overlap.
ad hoc to paper In the NP-hardness reduction, oracle scores can be assigned per model-item pair so each model group occupies its own score interval I_n and a target model's mean crosses its group threshold only at exactly 1 + w_n evaluations.
Appendix B, Theorem 1 proof. The construction assigns disjoint score intervals to model groups, which is not obviously compatible with a shared additive item difficulty d_x across models; the feasibility of the reduction under the paper's own correlated-arms model is not shown.
standard math Strong Law of Large Numbers and Borel-Cantelli lemmas in the consistency proof.
Theorem 2 proof: a lower-bounded selection probability implies every model is sampled infinitely often and mean estimates converge. Standard probability, no issue.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamically Allocating Evaluation Effort for Model Ranking." pith.science (2026). https://pith.science/paper/DUK57DYZ

@misc{pith2026260803437,
  author       = {Pith},
  title        = {Pith review of: Dynamically Allocating Evaluation Effort for Model Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUK57DYZ}},
  note         = {Machine review of arXiv:2608.03437}
}

read the original abstract

While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Figures

Figures reproduced from arXiv: 2608.03437 by the authors.

**Figure 1.** Classical evaluation vs our proposed approach: we evaluate the same number of items but in dynamic evaluation we automatically focus on top-performing models. This informs a better choice of closematching state-of-the-art models. instead of strategically allocating evaluation effort where it is most needed (e.g. top performing and close matching models). We frame allocation of evaluation effort as a search proble… view at source ↗

**Figure 2.** Constrained evaluation allocation problem. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** “Missing” values are predicted with a linear [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Timeline of evaluation based on WMT25 English [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Ranking 𝜏𝜔 (average) across varying budgets (proportion of all item+model evaluations in the data), averaged across all languages and 100 seeds. Shaded areas correspond to 95% t-distribution confidence interval. Numbers in legend show areas under curves (i.e. average …

**Figure 6.** Figure 6: Resulting allocation of evaluation items from [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 9.** Figure 9: Score distribution for randomly selected items. Overlaying curve shows best matching Beta distribution. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

**Figure 10.** Figure 10: Model mean distribution across datasets. Overlaying curve shows best matching Normal distribution. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Variance and mean of model scores. The trend shows slight heteroscedasticity: better models tend to [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 5 canonical work pages

[2]

ambiguity

Since 𝜇̂𝑚,𝐵 converges to 𝑞𝑚, there exists a finite budget 𝐵† such that for all 𝐵 > 𝐵†, |𝜇̂𝑚,𝐵 − 𝑞𝑚| < 𝛿 2 for all 𝑚, implying 𝑅̂𝐵 = 𝑅∗. ∎ We now introduce Lemma 4 which is used for proving Theorem 3. Lemma 4 (Asymptotic allocation convergence): Let 𝜋 be a sampling policy where at each step 𝑡, a model 𝑚 is selected with probability 𝑃𝑡(𝑚) proportional to it...

work page 2010
[6]

In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

Best Arm Identification for Prompt Learning under a Limited Budget . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. 2025. The L...

arXiv 2024
[8]

Advances in neural information pro cessing systems 36:46595–46623

Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information pro cessing systems 36:46595–46623. Jin Peng Zhou, Christian K. Belardi, Ruihan Wu, Travis Zhang, Carla P. Gomes, Wen Sun, and Kilian Q. Weinberger. 2025. On Speeding Up Language Model Evaluation. ArXiv: 2407.06172 [cs.AI]. Vilém Zouhar, Peng Cui, and Mrinmaya Sachan....

arXiv 2025
[2002]

Machine learning 47:235–256

Finite-time analysis of the multiarmed bandit problem. Machine learning 47:235–256. James E Baker. 1985. Adaptive selection methods for genetic algorithms. In Proceedings of the 1st Inter national Conference on Genetic Algorithms , pages 101–111, Pittsburgh, PA, USA. Esma Balkır, Alice Pernthaller, Marco Basaldella, José Hernández-Orallo, and Nigel Coll...

arXiv 1985
[2006]

Journal of Machine Learning Research 7(39):1079–1105

Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems . Journal of Machine Learning Research 7(39):1079–1105. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translati...

work page 2021
[2013]

In Proceedings of the 30th International Conference on Machine Learning, pages 1238–1246, Atlanta, Georgia, USA

Almost Optimal Exploration in Multi-Armed Bandits. In Proceedings of the 30th International Conference on Machine Learning, pages 1238–1246, Atlanta, Georgia, USA. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstan- tin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundk...

work page arXiv 2025
[2023]

In Proceedings of the Eighth Conference on Machine Translation , pages 756–767, Singapore

MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task . In Proceedings of the Eighth Conference on Machine Translation , pages 756–767, Singapore. Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. 2012. PAC Subset Selection in Sto- chastic Multi-armed Bandits. In Proceedings of the 29th International Conference on Machin...

work page 2023
[2024]

In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria

tinyBenchmarks: evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating Machine Translation Difficulty . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24...

work page arXiv 2025

Show all 9 references

[2025]

ArXiv: 2505.03814 [stat.ML]

Cer-Eval: Certifiable and Cost-Efficient Eval - uation Framework for LLMs . ArXiv: 2505.03814 [stat.ML]. Zhuo Wang, Wen Wu, Guoqing Wang, Guangze Ye, and Zhenxiao Cheng. 2026. MetaEval: Measuring the Discrimination of Benchmarks for Efficient LLM Evaluation. Proceedings of the...

2026 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2]

ambiguity

Since 𝜇̂𝑚,𝐵 converges to 𝑞𝑚, there exists a finite budget 𝐵† such that for all 𝐵 > 𝐵†, |𝜇̂𝑚,𝐵 − 𝑞𝑚| < 𝛿 2 for all 𝑚, implying 𝑅̂𝐵 = 𝑅∗. ∎ We now introduce Lemma 4 which is used for proving Theorem 3. Lemma 4 (Asymptotic allocation convergence): Let 𝜋 be a sampling policy where at each step 𝑡, a model 𝑚 is selected with probability 𝑃𝑡(𝑚) proportional to it...

work page 2010

[2] [6]

In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

Best Arm Identification for Prompt Learning under a Limited Budget . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. 2025. The L...

arXiv 2024

[3] [8]

Advances in neural information pro cessing systems 36:46595–46623

Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information pro cessing systems 36:46595–46623. Jin Peng Zhou, Christian K. Belardi, Ruihan Wu, Travis Zhang, Carla P. Gomes, Wen Sun, and Kilian Q. Weinberger. 2025. On Speeding Up Language Model Evaluation. ArXiv: 2407.06172 [cs.AI]. Vilém Zouhar, Peng Cui, and Mrinmaya Sachan....

arXiv 2025

[4] [2002]

Machine learning 47:235–256

Finite-time analysis of the multiarmed bandit problem. Machine learning 47:235–256. James E Baker. 1985. Adaptive selection methods for genetic algorithms. In Proceedings of the 1st Inter national Conference on Genetic Algorithms , pages 101–111, Pittsburgh, PA, USA. Esma Balkır, Alice Pernthaller, Marco Basaldella, José Hernández-Orallo, and Nigel Coll...

arXiv 1985

[5] [2006]

Journal of Machine Learning Research 7(39):1079–1105

Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems . Journal of Machine Learning Research 7(39):1079–1105. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translati...

work page 2021

[6] [2013]

In Proceedings of the 30th International Conference on Machine Learning, pages 1238–1246, Atlanta, Georgia, USA

Almost Optimal Exploration in Multi-Armed Bandits. In Proceedings of the 30th International Conference on Machine Learning, pages 1238–1246, Atlanta, Georgia, USA. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstan- tin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundk...

work page arXiv 2025

[7] [2023]

In Proceedings of the Eighth Conference on Machine Translation , pages 756–767, Singapore

MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task . In Proceedings of the Eighth Conference on Machine Translation , pages 756–767, Singapore. Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. 2012. PAC Subset Selection in Sto- chastic Multi-armed Bandits. In Proceedings of the 29th International Conference on Machin...

work page 2023

[8] [2024]

In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria

tinyBenchmarks: evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating Machine Translation Difficulty . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24...

work page arXiv 2025