Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

Fluid Benchmarking claims that evaluating language models with an adaptive item response theory method improves validity, variance, and saturation, often with far fewer items.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:04 UTC pith:DHJJXNWH

load-bearing objection FLUIDBENCHMARKING is a well-executed, reproducible study that makes a genuine contribution to LM evaluation, but the lack of statistical rigor and the unexamined weak-ability region keep me from endorsing the strong claims as-is. the 3 major comments →

arxiv 2509.11106 v1 pith:DHJJXNWH submitted 2025-09-14 cs.CL cs.AIcs.LG

Fluid Language Model Benchmarking

Valentin Hofmann , David Heineman , Ian Magnusson , Kyle Lo , Jesse Dodge , Maarten Sap , Pang Wei Koh , Chun Wang
show 2 more authors
Hannaneh Hajishirzi Noah A. Smith
This is my paper
classification cs.CL cs.AIcs.LG
keywords language model evaluationitem response theoryadaptive testingbenchmark refinement2PL modelFisher informationevaluation validitybenchmark saturation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Fluid Benchmarking, a method for evaluating language models that treats benchmark questions as psychometric test items with estimated difficulty and discrimination. It claims that scoring models in a latent ability space and dynamically choosing the most informative items for each model improves evaluation quality on all four measured dimensions: efficiency, validity, step-to-step variance, and saturation. The authors demonstrate the gains during pretraining of six LMs on six standard benchmarks, reporting e.g. better validity and lower variance on MMLU with fifty times fewer items than random item sampling. A sympathetic reader would take away that static evaluation leaves substantial quality on the table; making evaluation adaptive could become standard practice.

Core claim

The central discovery is that combining two psychometric components yields evaluation quality gains that neither component achieves alone. First, item response theory (specifically a two-parameter logistic model fitted to existing leaderboard results) converts each model's raw accuracy into a latent ability estimate, correcting for item difficulty and discrimination. Second, a Fisher-information-based selection rule dynamically chooses the subset of items that is most informative for the current ability estimate, as in computerized adaptive testing. In experiments with six pretraining LMs and six benchmarks, this combination outperforms random item sampling, static IRT-based subsets, and oth

What carries the argument

A two-parameter logistic (2PL) item response model fitted to existing LM evaluation results, giving each benchmark item a difficulty and a discrimination parameter. Ability is estimated via maximum a posteriori in that latent ability space, and items are selected greedily by maximizing Fisher information given the current ability estimate, analogous to computerized adaptive testing. This replaces the standard accuracy metric with a latent ability score and the static benchmark subset with a per-model dynamic subset.

Load-bearing premise

The item difficulty and discrimination parameters, estimated from 102 fully trained leaderboard LMs, are assumed to remain accurate for much weaker intermediate checkpoints that fall far outside the ability range of the training set.

What would settle it

Take a very early checkpoint (e.g., 1% of training) from one of the six LMs, fit the 2PL model to the leaderboard LMs only, and compare Fluid Benchmarking's ability estimate against a held-out criterion benchmark. If the rank distance under dynamic selection is not lower than under random item selection, or if the selected item set degenerates to always choosing the same easiest items without tracking ability, the claimed advantage for weak models is not real.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • IRT-based scoring alone improves validity but can worsen step-to-step variance; dynamic Fisher-information selection is what reduces variance, resolving earlier criticisms that IRT increases evaluation noise.
  • Fluid Benchmarking with only 50 items outperforms full-benchmark accuracy on validity, variance, and saturation, so the gains are not merely a matter of efficiency.
  • Mislabeled items are nearly two orders of magnitude less likely to appear in Fluid Benchmarking selections than in random subsets, because low-discrimination items carry little Fisher information.
  • Ability-space scoring keeps training curves informative even where accuracy saturates, as shown for OLMo2-7B on HellaSwag during the final 30% of training.
  • The method supports dynamic stopping: evaluation can terminate once the standard error of the ability estimate falls below the average rank-adjacent ability gap, varying the number of items per checkpoint.
  • Separate unidimensional IRT models per benchmark are preferred over a single cross-benchmark model; multidimensional IRT is an open variant that may suit multimodal evaluation.
  • Unless item parameters are regularly refreshed, the method cannot distinguish difficulty among items that no training LM answered correctly, so Fluid Benchmarking needs a living reference standard as model capabilities grow.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural operationalization of this work is a rolling leaderboard where the IRT model is continuously refit as new model results arrive, which would extend the performance gains to frontier models whose ability exceeds the current training set.
  • The decomposition into IRT-for-validity and adaptation-for-variance suggests that existing efficient-evaluation methods could be upgraded in a modular way: static IRT subsets could be made adaptive, and random sampling could be paired with IRT aggregation.
  • Because the method only needs evaluation results to fit the item model, it is directly transferable to other modalities and languages, e.g., vision-language leaderboards, as the paper notes.
  • A testable extension would be to measure whether the same two-component decomposition holds for posttraining evaluation, where ability distributions differ and calibration of item parameters may be more challenging.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces FLUIDBENCHMARKING, an adaptive LM evaluation method that combines two-parameter logistic item response theory (IRT) ability estimation with Fisher-information-based dynamic item selection. The authors formalize benchmark refinement as a selection/scoring/aggregation problem, define four quality dimensions (efficiency, validity, variance, saturation), and evaluate the method on six pretraining runs (61–94 checkpoints each) across six Open LLM Leaderboard benchmarks, reporting 2,802 checkpoint–benchmark combinations and over 13 million item-level evaluations. They compare against random sampling, random sampling plus IRT ability estimation, ANCHORPOINTS, TINYBENCHMARKS, METABENCH, SMART, and MAGI. The central claim is that FLUIDBENCHMARKING outperforms all baselines across all dimensions and sample sizes, and that the two components have complementary effects: IRT improves validity while dynamic selection reduces variance. Code and data are released.

Significance. If the central claim holds, this is a substantial contribution to LM evaluation methodology. It provides a unifying framework, a large-scale empirical study with held-out LMs and checkpoints, and a clear decomposition of the benefits of IRT aggregation versus adaptive item selection. The paper also directly addresses the prior 'false promise of IRT' critique by showing that variance reduction requires dynamic selection, not just IRT-based scoring. Strengths include the breadth of the experimental design, the use of held-out LMs for item-parameter estimation, publicly released code and data, and the specific falsifiable prediction that adaptivity improves evaluation quality. The main risks are uncertainty about item-parameter extrapolation to low-ability checkpoints and the lack of uncertainty quantification for the headline comparisons.

major comments (3)
  1. [§5, Tables 1–2] The headline claim that FLUIDBENCHMARKING 'outperforms all baselines across all dimensions and sample sizes' rests on single mean values over 2,802 checkpoint–benchmark combinations. These observations are not independent (they are nested within LMs, benchmarks, and checkpoints), yet no confidence intervals, significance tests, or effect-size distributions are reported. This is load-bearing for the central claim. Please provide bootstrap confidence intervals clustered by LM/benchmark (or by checkpoint), paired comparisons per benchmark and LM, and state whether observed differences are stable across these groupings.
  2. [§3.1, §4.1, §6] Item parameters (a_j, b_j) in Eq. (2) are estimated from 102 final-checkpoint leaderboard LMs. The evaluation targets include early pretraining checkpoints whose abilities likely fall below the range spanned by the training LMs. For the easiest items, responses from the training LMs are near-ceiling, so difficulty and discrimination are weakly identified and largely prior-driven. If these parameters are miscalibrated for low-ability checkpoints, both the MAP ability estimate in Eq. (3) and the Fisher-information item selection in Eq. (5) can be biased. Section 6 explicitly addresses extrapolation to stronger LMs, but not the weak-ability end. Because Tables 1–2 aggregate over all checkpoints, a systematic low-end bias would propagate into every reported dimension. This risk is directly testable: please add posterior predictive checks or diagnostic plots for the earliest checkpoints, and/
  3. [§4.2] The validity metric is not fully specified. 'Rank distance' is computed between predicted performance on one benchmark and accuracy on another, but the population over which ranks are computed is not defined. Are ranks computed across all checkpoints and LMs pooled, within each LM's training run, or across final checkpoints only? This choice changes the interpretation (cross-sectional versus longitudinal validity) and is needed for reproducibility. Please state the exact protocol and justify it.
minor comments (4)
  1. [Figure 2 and Appendix B] The 'simulated trajectory' of ability from -7 to +7 is mentioned but no simulation details are provided. Please specify the exact simulation setup (number of checkpoints, relationship with training progress, noise, etc.) so the figure is reproducible.
  2. [Appendix D] The inclusion criteria for the 102 training LMs are described, but the full list of LMs is not provided. Please include the complete list or a link to it for transparency and reproducibility.
  3. [Eq. (6)] The normalized total variation divides by |x_n - x_1|. If this denominator is zero or very small, the measure becomes unstable or undefined. Please clarify how such cases are handled and whether results are sensitive to this choice.
  4. [Abstract and §1] The abstract states 'fifty times fewer items' without giving the exact baseline item count. Please specify the comparison (e.g., 500 vs. 10 items) so the claim is precise.

Circularity Check

0 steps flagged

No significant circularity: IRT parameters are fit to held-out leaderboard LMs, and the reported validity, variance, and saturation gains come from new checkpoint evaluations, not from construction.

full rationale

The paper's derivation chain is not circular. The 2PL item parameters (a_j, b_j) are estimated from 102 Open LLM Leaderboard LMs after explicitly excluding the six test LMs and related models (Section 4.1 and Appendix D). The ability estimates for the 2,802 checkpoint-benchmark combinations are then computed from newly collected item-level evaluations (Section 4.1: 'We then evaluate all checkpoints of the six selected LMs on the six benchmarks'), so the reported scores are not fitted to their own targets. The validity metric compares ability ranks on one benchmark against accuracy ranks on a different benchmark (Section 4.2), an external criterion rather than the fitted response matrix. The variance reduction is consistent with Fisher-information-based selection (Eq. 5) and psychometric theory (Lord, 1983), but it is measured on actual training curves, and the ablations show that RANDOMIRT (same IRT aggregation, random selection) does not reduce variance, so the effect is not a definitional artifact. The saturation improvement is an empirical property of the adaptive test on held-out checkpoints, not entailed by the definition of ABILITY. There is no load-bearing self-citation: the methodological components (2PL IRT, MCMC, Fisher information, computerized adaptive testing) are cited from external psychometric and NLP literature, and the authors' own prior work appears only as benchmark/test data. The acknowledged limitation in Section 6 concerns extrapolation to LMs more capable than any training LM; the analogous extrapolation risk at the low-ability end is a genuine correctness concern but does not make the derivation circular. Overall, the central claim is supported by out-of-sample measurements and the paper is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 8 axioms · 1 invented entities

The central claim rests on IRT item parameters fit to leaderboard data, the unidimensional 2PL model, the assumption that these parameters generalize to held-out checkpoints, and the validity of using a paired benchmark's full accuracy as ground truth. The method itself introduces no new physical entities; the latent ability is a measurement construct with partial external validation.

free parameters (3)
  • Item difficulty and discrimination parameters (b_j, a_j) for every item in the six Open LLM Leaderboard benchmarks = Estimated via MCMC 2PL on 102 LMs (values not listed in paper)
    These are the core fitted parameters of the method; every downstream ability estimate and item selection depends on them.
  • Hyperparameters of hierarchical priors in the MCMC estimation (prior means/variances for b_j and a_j) = Set by MCMC procedure (Natesan et al. 2016)
    The 2PL model uses hierarchical priors on all parameters as suggested by Natesan et al. (2016); these hyperparameters affect the estimated item parameters.
  • Dynamic stopping threshold (standard error below average ability gap between rank-adjacent LMs) = Computed from Open LLM Leaderboard (specific number not stated)
    Used in the dynamic stopping demonstration (Section 6, Figure 5); a heuristic choice that affects the number of items administered.
axioms (8)
  • domain assumption 2PL logistic item response function (Equation 2)
    Invoked in Section 3.1; assumes a specific monotone relationship between latent ability and probability of a correct response for each item.
  • domain assumption Local independence of item responses conditional on ability
    Stated in Section 3.1 'we assume local independence'; required for the likelihood factorization in Equation 3.
  • domain assumption Unidimensionality of each benchmark
    They fit separate unidimensional IRT models per benchmark (Section 3.1, Appendix E); assumes one latent trait per benchmark.
  • domain assumption Generalization of item parameters from 102 leaderboard LMs to test LMs and their checkpoints
    The IRT models are fit on Open LLM Leaderboard LMs (Section 4.1); the method applies them to checkpoints of six held-out LMs, including early checkpoints likely far outside the leaderboard ability range (discussed partially in Section 6).
  • domain assumption Accuracy on a paired benchmark is a valid ranking ground truth
    Section 4.2 defines validity as rank distance to accuracy-based rank on a second benchmark; assumes full-benchmark accuracy correctly orders LMs on the target capability.
  • domain assumption Model capability increases monotonically during pretraining
    The saturation metric (Section 4.2) interprets higher Spearman correlation between checkpoint index and performance as better, assuming the true underlying capability increases monotonically during pretraining.
  • standard math Fisher information is the appropriate optimality criterion for item selection
    Section 3.2 uses Fisher information (Equation 4) from psychometric theory (Reckase 2009); standard result that higher Fisher information yields more precise ability estimates.
  • domain assumption MAP estimation assumes a prior distribution on ability (not specified in the paper)
    Section 3.1 uses maximum a posteriori estimation (Birnbaum 1969) but does not specify the prior; the choice affects ability estimates and downstream item selection.
invented entities (1)
  • Latent ability theta_i for each LM and checkpoint independent evidence
    purpose: To represent LM performance on a common latent scale instead of raw accuracy, enabling adaptive item selection and IRT-based aggregation.
    A standard psychometric latent variable, newly applied as a benchmark score; the paper provides external validity evidence by showing that ability ranks predict accuracy ranks on paired benchmarks.

pith-pipeline@v1.3.0-alltime-deepseek · 17229 in / 13225 out tokens · 150319 ms · 2026-08-04T17:04:17.585226+00:00 · methodology

0 comments
read the original abstract

Language model (LM) benchmarking faces several challenges: comprehensive evaluations are costly, benchmarks often fail to measure the intended capabilities, and evaluation quality can degrade due to labeling errors and benchmark saturation. Although various strategies have been proposed to mitigate these issues, they tend to address individual aspects in isolation, neglecting broader questions about overall evaluation quality. Here, we introduce Fluid Benchmarking, a new evaluation approach that advances LM benchmarking across multiple dimensions. Inspired by psychometrics, Fluid Benchmarking is based on the insight that the relative value of benchmark items depends on an LM's capability level, suggesting that evaluation should adapt to each LM. Methodologically, Fluid Benchmarking estimates an item response model based on existing LM evaluation results and uses the inferred quantities to select evaluation items dynamically, similar to computerized adaptive testing in education. In our experiments, we compare Fluid Benchmarking against the common practice of random item sampling as well as more sophisticated baselines, including alternative methods grounded in item response theory. We examine four dimensions -- efficiency, validity, variance, and saturation -- and find that Fluid Benchmarking achieves superior performance in all of them (e.g., higher validity and less variance on MMLU with fifty times fewer items). Our analysis shows that the two components of Fluid Benchmarking have distinct effects: item response theory, used to map performance into a latent ability space, increases validity, while dynamic item selection reduces variance. Overall, our results suggest that LM benchmarking can be substantially improved by moving beyond static evaluation.

Figures

Figures reproduced from arXiv: 2509.11106 by Chun Wang, David Heineman, Hannaneh Hajishirzi, Ian Magnusson, Jesse Dodge, Kyle Lo, Maarten Sap, Noah A. Smith, Pang Wei Koh, Valentin Hofmann.

Figure 1
Figure 1. Figure 1: (a) Given a benchmark Q, we train an IRT model on publicly available LM evaluation results, providing useful information about individual items (specifically, about difficulty and discrimination). The figure illustrates this with results for four LMs and difficulty, symbolized by item darkness. In practice, we use more than a hundred LMs. (b) FLUID BENCHMARKING leverages the IRT-enriched benchmark in two w… view at source ↗
Figure 2
Figure 2. Figure 2: Fisher information (Equa￾tion 4) of HellaSwag items as a func￾tion of training progress. Lower panel: simulated trajectory of LM ability, which evolves linearly from θ 1 i = −7 to θ 50 i = +7; upper panel: Fisher information of HellaSwag items. The HellaSwag items with highest Fisher information change drastically during training (see Ap￾pendix B for more details). It can be shown that items with higher Fi… view at source ↗
Figure 3
Figure 3. Figure 3: FLUID BENCHMARKING of OLMo1-7B (HellaSwag/50 items). The figure shows items (stacked along y-axis) selected for FLUID BENCHMARKING as a function of different checkpoints. Items are ordered by difficulty bj . Items selected for FLUID BENCHMARKING are colored by time of selection; brighter colors reflect earlier appearance during evaluation. The bright line close to y = 0 represents the first item, which is … view at source ↗
Figure 4
Figure 4. Figure 4: Training curves of OLMo2-7B (HellaSwag/500 items) with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FLUID BENCHMARKING with dynamic stopping on OLMo1- 7B/HellaSwag (see text for details). Dynamic Stopping. A further advantage of FLUID BENCHMARKING is its support for dynamic stopping. In [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example item characteristic curves. The x-axis shows the ability parameter θi ; the greater θi , the higher the success probability p(uij = 1). The difficulty parameter bj indicates the value of θi at which p(uij = 1) = 0.5, reflected by the location of the curve (compare q1 vs. q3). The discrimination parameters indicates how sharply p(uij = 1) changes when θi is close to bj . aj is proportional to the sl… view at source ↗
Figure 7
Figure 7. Figure 7: Fisher information of HellaSwag items halfway through the simulated training [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Variance and saturation results. The figure shows pairwise comparisons measuring [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GAUGE: Grading Agent-Built Financial Models Without a Golden Answer

    cs.LG 2026-07 conditional novelty 7.0

    Single-expert grading of financial models is invalid because professionals disagree; GAUGE’s practice-envelope benchmark finds the best agent above students but below seniors, with a large mechanical–judgment gap.

  2. Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

    cs.LG 2026-05 unverdicted novelty 7.0

    IRSL applies IRT to reduce scaling law estimation from O(M×N) to O(M+N) parameters, enabling reliable estimates with only 50 questions per benchmark after calibration and generalizable ability scores across related be...

  3. Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks

    cs.LG 2026-05 unverdicted novelty 6.0

    Cross-entropy method sampling reduces inferences needed to estimate five-nines LLM reliability by up to 156x on parameterized GSM8K templates, revealing reliability differences hidden by saturated accuracy scores.

  4. Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

    cs.AI 2026-05 unverdicted novelty 6.0

    Dynamic Boundary Evaluation adaptively identifies each LLM's performance boundary on a shared difficulty scale using a calibrated item bank and a search algorithm.

  5. Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

    cs.AI 2026-05 unverdicted novelty 6.0

    Dynamic Boundary Evaluation locates each LLM's performance boundary at ~50% pass probability via a calibrated item bank and Skill-Guided Boundary Search algorithm to enable unified, adaptive evaluations across safety,...

  6. Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

    cs.CL 2026-04 unverdicted novelty 6.0

    A fixed-parameter multidimensional IRT calibration approach allows extending LLM benchmark suites over time, predicting full performance within 2-3 points and preserving rankings (Spearman ρ ≥ 0.9) using only 100 anch...

Reference graph

Works this paper leans on

21 extracted references · 12 linked inside Pith · cited by 5 Pith papers

  1. [5]

    Passonneau, Megan Ung, and Adina Williams

    Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using SMART filtering of benchmark datasets. arXiv:2410.20245,

    Pith/arXiv arXiv

  2. [6]

    C Checkpoint Details We provide details about the selected LM checkpoints

    B Fisher Information of HellaSwag Items For illustrative purposes, Figure 7 shows the Fisher information of HellaSwag items halfway through the simulated training run, whenθ t i =0. C Checkpoint Details We provide details about the selected LM checkpoints. For Amber-6.7B, we select 73 check- points. For OLMo1-7B, we select 83 checkpoints. For OLMo2-7B, we...

    2025

  3. [9]

    Lalor, Hao Wu, and Hong Yu

    John P . Lalor, Hao Wu, and Hong Yu. Learning latent parameters without human response patterns: Item response theory with artificial crowds. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,

    2019

  4. [14]

    Efficient benchmarking (of language models)

    Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024a. 13 Published as a con...

    Pith/arXiv arXiv 2024

  5. [15]

    Chal- lenging BIG-Bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG-Bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023,

    2023

  6. [16]

    Do large language model benchmarks test reliability?arXiv:2502.03461,

    Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability?arXiv:2502.03461,

    Pith/arXiv arXiv

  7. [18]

    14 Published as a conference paper at COLM 2025 −3 −2 −1 0 1 2 3 LM ability θi 0.0 0.2 0.4 0.6 0.8 1.0Success probability p(uij =

    2025

  8. [100]

    A-7B: Amber-7B; K-65B: K2-65B; O1-7B: OLMo1-7B; O2-7B: OLMo2-7B; P-3B: Pythia-2.8B; P-7B: Pythia-6.9B

    16 Published as a conference paper at COLM 2025 Language model Measure Method A-7B K-65B O1-7B O2-7B P-3B P-7B Validity RANDOM25.5 5.1 10.7 7.1 23.1 28.1 Rank distance↓RANDOMIRT20.2 5.2 8.0 6.1 8.3 15.3 FLUIDBENCHMARKING19.3 2.1 6.7 3.4 8.1 11.6 Variance RANDOM21.8 14.7 10.2 15.4 35.7 20.9 Total variation↓RANDOMIRT16.2 27.0 11.4 12.4 26.1 13.7 FLUIDBENCHM...

    2025

  9. [500]

    We find that full-benchmark accuracy performs worse than FLUIDBENCHMARKINGacross all three evaluation dimensions, despite using substantially more items

    with full-benchmark accuracy, using the same LMs and benchmarks as in our main experiments (see §4). We find that full-benchmark accuracy performs worse than FLUIDBENCHMARKINGacross all three evaluation dimensions, despite using substantially more items. This holds for validity (9.1 vs. 8.3 for FLUIDBENCHMARKING), variance (23.8 vs. 4.9 for FLUIDBENCH- MA...

    2025

  10. [1983]

    Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes

    Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks.arXiv:2406.10229,

    Pith/arXiv arXiv

  11. [2010]

    Are we done with MMLU?arXiv:2406.04127,

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with MMLU?arXiv:2406.04127,

    Pith/arXiv arXiv

  12. [2015]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv:1803.05457,

    Pith/arXiv arXiv

  13. [2016]

    Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu

    John P . Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. Understanding deep learning performance through an examination of test set difficulty: A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

    2018

  14. [2018]

    Training verifiers to solve math word problems.arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv:2110.14168,

    Pith/arXiv arXiv

  15. [2019]

    Pardos, Patrick C

    Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Zachary A. Pardos, Patrick C. Kyllonen, Jiyun Zu, Qingyang Mao, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Shijin Wang, and Enhong Chen. From static benchmarks to adaptive testing: Psychometrics in AI evaluation.arXiv:2306.10512,

    Pith/arXiv arXiv

  16. [2020]

    Lalor, Hao Wu, and Hong Yu

    John P . Lalor, Hao Wu, and Hong Yu. Building an evaluation scale using item response theory. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,

    2016

  17. [2021]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William ...

    Pith/arXiv arXiv

  18. [2022]

    12 Published as a conference paper at COLM 2025 Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu...

    Pith/arXiv arXiv 2025

  19. [2023]

    LLM360 K2: Building a 65B 360-open-source large language model from scratch

    Zhengzhong Liu, Bowen Tan, Hongyi Wang, Willie Neiswanger, Tianhua Tao, Haonan Li, Fajri Koto, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Liqun Ma, Liping Tang, Nikhil Ranjan, Yonghao Zhuang, Guowei He, Renxi Wang, Mingkai Deng, Robin Algayres, Yuanzhi Li, Zhiqiang Shen, Preslav Nakov, and Eric Xing. LLM360 K2: Building a 65B...

    Pith/arXiv arXiv

  20. [2024]

    OLMES: A standard for language model evaluations.arXiv:2406.08446,

    11 Published as a conference paper at COLM 2025 Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations.arXiv:2406.08446,

    Pith/arXiv arXiv 2025

  21. [2025]

    Lalor and Hong Yu

    John P . Lalor and Hong Yu. Dynamic data selection for curriculum learning via ability estimation. InFindings of the Association for Computational Linguistics: EMNLP 2020,

    2020