Pith. sign in

REVIEW 2 major objections 5 minor 14 references

BayesAME stops evaluating a model once its performance estimate and uncertainty both settle, and active item selection beats random sampling when reference scores distinguish items well.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 13:30 UTC pith:D6WWVLL6

load-bearing objection Solid systems paper on auto-stopping coreset eval; fixed-n gains look real, but the Pareto curves partly compare apples-to-oranges uncertainty scales. the 2 major comments →

arxiv 2607.27023 v1 pith:D6WWVLL6 submitted 2026-07-29 cs.LG cs.AIstat.ML

BayesAME: Bayesian Active Model Evaluation

Paula Cordero Encinar , Taylan Cemgil , Arnaud Doucet , Virginia Aglietti , Silvia Chiappa This is my paper
classification cs.LG cs.AIstat.ML
keywords active model evaluationcoreset selectionBayesian sequential designbenchmark efficiencyinformation gainmulti-target evaluationlog-likelihood scores
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Evaluating large generative models on full benchmarks is expensive. BayesAME treats full-benchmark performance as a random variable built from latent abilities on groups of items that share the same scores from historical reference models. A joint Gaussian prior encodes the belief that the target model behaves like those references. The method grows a coreset by picking items that most reduce uncertainty in performance, updates a closed-form posterior, and stops when both the estimate's recent fluctuation and the width of its credible interval fall below user thresholds. A multi-target extension shares latent factors across several models so correlated targets need fewer evaluations. Across standard leaderboards the method beats sequential versions of prior estimators, shows that non-random selection helps once reference vectors are sufficiently unique, and shows that continuous log-likelihood scores give markedly tighter estimates than binary correct/incorrect labels.

Core claim

When reliable estimation matters more than a fixed budget, a sequential Bayesian model of bucket-level abilities, driven by an information-gain selection rule and a two-threshold stopping rule on estimate stability and credible-interval width, yields lower error at smaller coresets than sequential adaptations of existing methods; non-random selection is advantageous precisely when reference score vectors are nearly unique, and continuous scores substantially outperform binary scores.

What carries the argument

Bucket-level latent abilities with a squared-exponential Gaussian prior on reference-score similarity (Eq. 1), whose posterior mean and variance supply both the performance estimator (Eq. 5) and the information-gain acquisition function (Eq. 6), together with a dual stopping criterion on estimate range and 95% credible-interval width.

Load-bearing premise

The target model is assumed to behave like the reference models on items that look similar under those references; if that similarity structure is badly wrong, both the prior and the active-selection rule can mislead.

What would settle it

On a benchmark whose reference score vectors are nearly unique, replace active information-gain selection with pure random selection while keeping the same posterior estimator and stopping thresholds; if the cost-accuracy Pareto front of BayesAME no longer dominates the random and mean baselines, the claim that non-random selection is advantageous fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can set two plain thresholds (estimate stability and tolerated uncertainty width) and obtain an automatically sized coreset without guessing a budget in advance.
  • When several correlated models must be ranked or compared, joint coregionalization can shrink the total number of item-model evaluations needed for the same accuracy.
  • Switching from binary correctness to continuous response log-likelihoods is a high-leverage, low-overhead way to tighten performance estimates.
  • Claims that random coresets and the plain sample mean are hard to beat do not hold once reference representations are rich enough and evaluation runs across the full range of coreset sizes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same bucket-and-prior construction could be reused for multi-dimensional or preference-based evaluation scores if the likelihood is replaced while keeping the information-gain and stopping logic.
  • If public leaderboards routinely published continuous log-likelihoods rather than only binary marks, active coreset methods would become substantially more sample-efficient by default.
  • A practical deployment path is to start with a small random pilot to learn correlations, then switch to active multi-target selection only when the estimated cross-model weights are large.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. BayesAME is a sequential Bayesian method for estimating a target model’s full-benchmark mean performance from a coreset, with automatic coreset-size determination. Items are grouped into buckets by identical reference-model score vectors; each bucket has a latent ability with a squared-exponential Gaussian prior (Eq. 1). Scores are modeled as Gaussian given abilities (bucket-size-scaled noise); the posterior yields a performance estimator (Eq. 5), a credible-interval width (Eq. 2), and an information-gain acquisition rule (Eq. 6). The coreset grows until estimate fluctuation and interval width fall below user thresholds ε1, ε2 (Alg. 1). A multi-target LMC extension shares latent factors across targets. Experiments on seven benchmarks (binary and continuous scores; interpolation/extrapolation; 10/50/90% references; multi-target low/high correlation) compare against sequentialized baselines via RMSE Log/Gain, evaluation-weighted integrated RMSE, and cost–accuracy Pareto curves over 200 seeds. The paper claims consistent outperformance, that active selection beats random when reference representations are sufficiently unique, and that continuous log-likelihood scores substantially improve estimation.

Significance. Efficient evaluation of large generative models is a genuine bottleneck; automatic budget determination when reliability is prioritized is under-addressed relative to fixed-budget methods. The empirical program is a clear strength: multiple benchmarks, interpolation vs. extrapolation splits in the spirit of Zhang et al., binary vs. continuous scores, active vs. random ablation (BayesAME vs. BayesAME-RS), and multi-target correlation regimes, with metrics that average over 200 seeds. The finding that continuous response scores markedly reduce error, and that non-random selection helps once buckets are nearly unique, directly engages recent skepticism and is of practical value. Closed-form Gaussian posteriors and O(min(n,B)²) updates make the method usable. If the auto-stopping comparisons are placed on firmer footing, the work is a solid contribution to data-efficient model evaluation.

major comments (2)
  1. [§3.1 Stopping Criterion; §5.1 Metrics; Fig. 2; Eq. (2) vs Eq. (9)] The Cost–Accuracy Trade-Off (Fig. 2; App. Figs. 7–12) is the main evidence for automatic coreset-size determination, yet every sequential method is stopped on the same ε2 grid applied to method-specific Var_R quantities that are not on a common scale or semantics: BayesAME/ProEval use Bayesian Var(R|s_C) for the finite-benchmark mean (Eq. 2); RS-Mean/Seq-APW use design-based sampling variance with FPC (Eq. 9); Bayes-AIPW propagates a ridge posterior through an AIPW correction. The paper does not report frequentist coverage of the nominal 95% intervals for true R★ across seeds/targets, nor any recalibration or width-normalization. Fixed-n metrics (RMSE Log/Gain, Table 1) already support estimator quality and the active-selection claim without calibrated variances; the Pareto frontier, however, may partly reflect uncertainty scaling rather than pure estimation quality. Please either (i) re
  2. [Abstract; §3.1 Selection Strategy; §5.1.1; Table 1; §6] Active selection is disabled whenever B < N−N0 (N0=10) or continuous scores round to non-unique vectors, defaulting to random (§3.1; Alg. 1). Table 1 and the text correctly note that BayesAME then coincides with BayesAME-RS on several binary benchmarks (e.g., MMLU-Pro, ARC-Challenge, MuSR). The abstract and §6 still state broadly that “non-random coreset selection is advantageous over random selection.” That claim is supported primarily in the nearly-unique / continuous-score regime (and scales with richer reference pools). Please state the scope of the active-selection advantage up front (abstract and contributions) in terms of the uniqueness condition, so readers do not over-generalize from the binary, coarse-bucket cases where the method intentionally does not select actively.
minor comments (5)
  1. [§4; §5.1 baseline list] ProEval’s estimator is defined as the average of latent abilities rather than of scores; the paper notes it need not recover R★ at full evaluation (§4, §5.1). Flagging this more visibly in figure legends would help readers interpret its RMSE curves.
  2. [Appendix C.1] Hyperparameter defaults (σ² = 10^{-4} vs 10^{-1} by uniqueness regime, F=50, W=30, N0=10) are listed in App. C.1; a short sensitivity note in the main text or appendix would strengthen robustness claims.
  3. [§3.2; Fig. 3] Multi-target initialization evaluates all targets on a random 10% of items before IG (§3.2). Clarify whether that cost is included in the coreset-size axis of Fig. 3 / Fig. 14.
  4. [Throughout (PDF text extraction / formatting)] Typographical/spacing issues appear in several places (e.g., “WefurthershowthatBayesAME”, “Evaluatinglargegenerativemodels”). A full copy-edit pass is needed.
  5. [§3.1; Appendix B.3] Appendix B.3’s logit-normal ablation (Fig. 5) is useful; citing it briefly in §3.1 when defending the Gaussian score model would preempt modeling objections.

Circularity Check

0 steps flagged

No significant circularity: estimators and selection rules are standard Bayesian constructions validated against external full-benchmark ground truth R★, not quantities forced by the fit.

full rationale

BayesAME’s load-bearing chain is: (i) bucket latent abilities with a squared-exponential Gaussian prior on reference-score similarity (Eq. 1); (ii) Gaussian likelihood and closed-form posterior mean/variance of the finite-benchmark mean R (Eqs. 2–8); (iii) information-gain item selection (Eq. 6) or random default when B < N−N0; (iv) optional multi-target LMC; (v) stop when estimate range ≤ ε1 and credible-interval width ≤ ε2. Hyperparameters α, β (and multi-target weights) are fit by coreset marginal likelihood, but every reported success metric—RMSE Log/Gain, Evaluation-Weighted Integrated RMSE, Cost-Accuracy Trade-Off, Spearman rank—compares ˆR to the held-out true mean R★ = (1/N)Σ s_i on external leaderboard benchmarks (Open LLM, HELM, Zhang et al.). Nothing in that comparison is defined by the fitted prior or forced by the stopping thresholds. The prior and the non-unique-regime random default are modeling choices, not self-definitional identities; active-vs-random and continuous-vs-binary claims are empirical ablations. No uniqueness theorem, self-citation chain, or renamed known identity carries the central claim. Interval-calibration differences across baselines (skeptic concern) affect fairness of the shared-ε2 Pareto plot but are a correctness/comparability issue, not circular reduction of a prediction to its inputs. Score 0; steps empty.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The method rests on standard Gaussian-process-style conditioning plus domain modeling choices (reference-score buckets, similarity prior, Gaussian noise on possibly binary scores) and several hand-set procedural constants for stopping and regime switching. No new physical entities; latent abilities are standard hierarchical-model constructs.

free parameters (6)
  • prior covariance scale α and length-scale β = initialized α0=β0=1; refit online
    Learned every F iterations by minimizing negative log marginal likelihood of coreset scores; control the strength of reference-driven correlation.
  • observation noise σ² = 10^{-4} or 10^{-1} (single-target); learned (multi-target)
    Fixed in single-target (10^{-4} if nearly unique refs else 10^{-1}); learned in multi-target. Directly scales posterior variance and stopping.
  • uniqueness margin N0 = 10
    Hand-set threshold deciding active vs random selection and Bayes vs control-variate estimator.
  • stopping thresholds ε1, ε2 and window W = ε1=0.005; ε2∈{0.005..0.03}; W=30
    User-defined; experiments sweep ε2 and fix ε1=0.005, W=30. They define when the coreset is 'done' and thus drive cost-accuracy claims.
  • hyperparameter update frequency F = 50
    Controls how often α,β (and multi-target weights) are re-optimized.
  • multi-target LMC weights w^t_l and kernel mix = online; w init N(0,0.01)
    Learned with AdamW; L=K_t with half SE and half Matérn-3/2 kernels.
axioms (6)
  • domain assumption Items sharing the same reference score vector form exchangeable buckets with a single latent ability θ_b for the target.
    §3.1 partitioning into B buckets; reduces item-level GP cost and is central to the model.
  • domain assumption Joint prior on abilities is Gaussian with mean average reference score and SE covariance of reference vectors (Eq. 1).
    Encodes 'target behaves like references'; drives posterior mean, variance, and IG.
  • ad hoc to paper Scores are conditionally Gaussian given θ with bucket-size-scaled noise, even for binary or [0,1] scores.
    §3.1 explicitly prefers this over support-correct logit/Bernoulli models for closed-form IG and better empirics (App. B.3).
  • ad hoc to paper When B < N−N0 (or continuous scores round to non-unique), random selection is used instead of IG.
    §3.1 selection strategy; protects against coarse buckets but weakens the 'active selection helps' claim in those regimes.
  • domain assumption Multi-target abilities follow a linear model of coregionalization with L=K_t latent GPs.
    §3.2; standard LMC (Álvarez et al.) applied to target correlation.
  • standard math Gaussian conditioning and differential-entropy IG formulas are valid under the stated joint Gaussians.
    Appendix A.1–A.2; standard multivariate normal identities.
invented entities (1)
  • Bucket-level latent abilities θ_b (and multi-target θ^t) no independent evidence
    purpose: Induce a random full-benchmark performance R and closed-form posterior estimators, uncertainty, and information gain.
    Standard hierarchical latent-variable construct, not a new physical object; independent evidence is only indirect via predictive RMSE on held-out full benchmarks.

pith-pipeline@v1.2.0-daily-grok45 · 44894 in / 3620 out tokens · 72092 ms · 2026-07-30T13:30:10.797487+00:00 · methodology

0 comments
read the original abstract

Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 9 linked inside Pith

  1. [2]

    Recall that𝐻C and 𝐷C denote the submatrices of𝐻 and 𝐷 formed by the rows corresponding toC

    has mean𝜇 1+Σ 12Σ−1 22(𝑥 2−𝜇 2)and covarianceΣ 11−Σ 12Σ−1 22 Σ21. Recall that𝐻C and 𝐷C denote the submatrices of𝐻 and 𝐷 formed by the rows corresponding toC. The joint distribution of(𝜃,𝑠C)can be written as: 𝜃 𝑠C ∼N 𝜇𝜃 𝐻C𝜇𝜃 , Σ𝜃 Σ𝜃𝐻⊤ C 𝐻C Σ𝜃 𝐻C Σ𝜃𝐻⊤ C+𝐷C . Applying the Gaussian conditioning rule, we obtain: 𝜇𝜃 |C =𝜇 𝜃+Σ 𝜃𝐻⊤ C(𝐻C Σ𝜃𝐻⊤ C+𝐷C)−1(𝑠C−𝐻C𝜇𝜃), Σ𝜃 ...

    1995

  2. [5]

    We observe that the overall performance of these batch approaches is similar to that of the single-item selection strategy. Table 2 compares theEvaluation-Weighted Integrated RMSEachieved byBayesAMEwith single- candidate selection, random selection, bath selection with|𝐴|= 2using both myopic (Myo) and non-myopic (Non-Myo) criteria, andBayes-AIPWacross the...

    arXiv

  3. [8]

    Z. Liu, J. Zhang, C. Liu, and Y. Zhu. Active testing of large language models via approximate Neyman allocation. arXiv preprint arXiv:2605.10075,

    Pith/arXiv arXiv

  4. [10]

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Pith/arXiv arXiv

  5. [30]

    (1)) and the rest are Matérn-3/2

    Of the𝐿 covariances, half are squared exponential (Eq. (1)) and the rest are Matérn-3/2. As described in Appendix A.4, optimization is performed using AdamW, with𝛼𝑙 and 𝛽𝑙 initialized to1, each𝑤𝑡 𝑖 to a sample fromN( 0, 0.01), and𝜎2 to0 .1for binary scores and to0.01 for continuous scores. We use𝐹= 50, and learning rate𝜂= 0.01for binary scores and𝜂= 0.001...

    2025

  6. [2006]

    However, we are interested in the posterior of𝜃=𝜎(𝑓)=(1+𝑒 −𝑓)−1, conditioned on𝑠C

    to compute the posterior. However, we are interested in the posterior of𝜃=𝜎(𝑓)=(1+𝑒 −𝑓)−1, conditioned on𝑠C. The posterior mean can be computed as: 𝔼[𝜃𝑏|𝑠C]= ∫ 𝜎(𝑓 𝑏)𝑝(𝑓|𝑠 C)d𝑓, which does not admit a closed-form solution because of the nonlinearity of𝜎. We consider three strategies for approximating this integral. • Monte Carlo estimation.The main drawba...

    1992

  7. [2012]

    A. M. Bean, N. Seedat, S. Chen, and J. R. Schwarz. Scales++: Compute efficient evaluation subset selection with cognitive scales embeddings.arXiv preprint arXiv:2510.26384,

    Pith/arXiv arXiv

  8. [2018]

    Fisch, D

    A. Fisch, D. Deutsch, J. Maynez, A. Agarwal, J. Berant, W. Cohen, A. Globerson, and J. Eisenstein. CollabEval: Statistically efficient collaborative model evaluation via matrix completion.arXiv preprint arXiv:2607.05046,

    Pith/arXiv arXiv

  9. [2019]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  10. [2021]

    Hsu and S

    C.-Y. Hsu and S. Shekhar. Efficient sequential evaluation of large language models.arXiv preprint arXiv:2607.17409,

    Pith/arXiv arXiv

  11. [2023]

    L. Liao, Q. Zhang, R. Wu, and G. Fang. Toward a unified framework for data-efficient evaluation of large language models.arXiv preprint arXiv:2510.04051,

    arXiv

  12. [2024]

    G. Wang, Z. Chen, B. Li, and H. Xu. Cer-Eval: Certifiable and cost-efficient evaluation framework for LLMs. arXiv preprint arXiv:2505.03814,

    Pith/arXiv arXiv

  13. [2025]

    Bowyer, A

    S. Bowyer, A. Locatelli, and K. Cao. Efficient benchmarking is just feature selection and multiple regression. arXiv preprint arXiv:2605.25773,

    Pith/arXiv arXiv

  14. [2026]

    Jiang, S

    H. Jiang, S. Kwon, J. Luo, Z. Xiao, and S. Zhang. Can we trust item response theory for AI evaluation?arXiv preprint arXiv:2607.15190,

    Pith/arXiv arXiv