Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Statistical multi-metric evaluation and visualization of LLM system predictive performance

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a framework that automatically chooses the right significance test, aggregates p-values across metrics and datasets, and visualizes the comparisons; on CrossCodeEval, every pair of 15 LLMs is statistically distinct after…

desk verdict Useful significance-aware leaderboard comparison framework made of standard parts; main caveat is the unaddressed dependence in benchmark items, which likely undermines the all-pairs-significant demonstration. read the letter →

arxiv 2501.18243 v1 pith:USSZWKAX submitted 2025-01-30 stat.AP cs.CLcs.LG

Samuel Ackerman , Eitan Farchi , Orna Raz , Assaf Toledo This is my paper · ORCID
classification stat.APcs.CLcs.LG
keywords statisticalsignificancetestingLLMevaluationp-valueaggregationharmonicmeaneffectsizemultiplecomparisonsCrossCodeEvalvisualization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to make LLM system evaluation statistically sound without requiring the user to know which test to run. Its framework takes per-instance scores from any set of systems, datasets, and metrics, selects the appropriate hypothesis test from data modality and pairing, adjusts for multiple comparisons, and aggregates results both within a dataset (by standardizing and averaging metrics) and across datasets (by combining p-values with the harmonic mean). The demonstration on the multilingual code benchmark CrossCodeEval shows that, once aggregation is done, all 15 model pairs are significantly different according to the p-value criterion. If this holds, leaderboard rankings and model upgrade decisions could be grounded in formal significance rather than raw score averages.

What carries the argument

The carrying mechanism is the score sample list collection and the two-stage aggregation it enables. First, per-instance metric values are standardized with the pooled metric mean and standard deviation, multiplied by a direction sign, and averaged across metrics for each system to form an aggregate score sample; this part requires paired observations. Second, when lists are unpaired (e.g., different language datasets), pairwise test results are aggregated: p-values via Wilson's harmonic mean p-value with multiplicity adjustment, effect sizes via inverse-variance weighting. Test selection follows a fixed table: Welch's t or paired t for numeric data, Z test or McNemar for binary data, with Cohen's d/h effect sizes; p-values are multiplicity-adjusted with the Holm-Šídák step-down procedure.

What would settle it

Re-analyze the CrossCodeEval data accounting for clustering by source repository or prompt template (e.g., by block bootstrap at the repository level or by mixed-effects models) and recompute the adjusted harmonic-mean p-values; if some system pairs stop being significant, the framework's aggregation has hidden dependence. Alternatively, run the framework on a benchmark with known item correlation and compare its p-values to those from a cluster-robust test.

Watch

Extended reading notes

Core claim

The central claim is that the framework automatically performs the correct statistical tests, properly aggregates statistical results across metrics and datasets, and visualizes the results. The key demonstration: by the p-value criterion aggregated across languages using the adjusted harmonic mean, all 15 system pairs are statistically significantly different. Within each language dataset, metric scores are standardized by pooled mean and standard deviation, direction-adjusted, and averaged element-wise into an aggregate metric; across languages, pairwise p-values are combined with Wilson's adjusted harmonic mean p-value, and effect sizes are combined by inverse-variance weighting. The framework also produces connected graphs, cliques of statistically similar systems, heatmaps, and a paired-p-values plot so that the significance structure is readable at a glance.

Load-bearing premise

The significance claims rest on the assumption that the per-instance scores for each system-dataset-metric combination are independent and identically distributed, so that t-tests, McNemar tests, and the p-values built on them are valid; if benchmark items are correlated (e.g., code files from one repository), the p-values become anti-conservative and the claim that all 15 pairs differ would be unsupported.

Editorial extensions

If this is right

  • Leaderboard-style rankings can be accompanied by significance groups (cliques), telling users which systems are statistically indistinguishable rather than merely close in average.
  • Model upgrade decisions, such as whether a new LLM version's score gain is real, can be made with formal p-value and effect-size criteria instead of raw score deltas.
  • The same pipeline applies to any predictive system, not only LLMs, since it consumes post-hoc scores.
  • Weighted aggregations allow a user to emphasize metrics or datasets they care about without losing significance testing.
  • If the CrossCodeEval result generalizes, averaged leaderboard differences that look small may still be statistically significant, and vice versa.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The all-pairs-significant result on CrossCodeEval should not be read as a property of the models: it depends on the independence assumption for per-instance scores; code completion items from the same repository are plausibly correlated, which would make the reported p-values anti-conservative.
  • The framework's harmonic-mean aggregation treats each metric-dataset list as an independent source of evidence; a user could extend this to a hierarchical model that accounts for within-benchmark correlation, which the paper does not address.
  • One testable extension is to compare the framework's p-value groupings against bootstrap percentile intervals on the same data; where they disagree, the aggregation assumptions would be the likely cause.
  • Because the framework ranks systems by a standardized mean, it inherits some of the same blind spots as mean-based MCDM; an inference is that for safety-critical selections, the user should combine it with per-metric inspection of conflicting directions, which the heatmap already hints at.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper presents a framework and open-source implementation for statistically comparing LLM-based systems across multiple evaluation metrics and datasets. It automatically selects between paired and unpaired tests depending on the data structure (Table 1), computes adjusted p-values and effect sizes for pairwise system comparisons, aggregates these results across metrics using the harmonic mean p-value method (Section 6.2), and provides several visualizations including connected graphs, heatmaps, and pairwise p-value plots. The framework is demonstrated on the CrossCodeEval benchmark with three metrics and 15 LLMs, and the main empirical claim is that all system pairs are statistically significantly different by the aggregated p-value criterion (Section 7.3).

Significance. This is a timely and useful contribution to the practice of LLM evaluation. The paper addresses a real gap: most leaderboards report raw averages without significance testing, and the framework provides a concrete, automated pipeline for paired/unpaired testing across metrics and datasets, with a public implementation. The demonstration on CrossCodeEval with 15 models and three metrics is a strength, as is the external sanity check against TOPSIS and other MCDM methods. The paper also explicitly cautions against over-reliance on p-values and includes effect sizes. However, several statistical choices in the aggregation layer—notably the use of adjusted p-values as inputs to the harmonic mean p-value procedure and the non-standard standardization in Section 6.3—need rigorous justification before the framework can be recommended as a generally valid tool. The independence assumption is also a substantive concern for the benchmark application.

major comments (5)
  1. [4.1 (Table 1), 7.3] The tests selected in Table 1 (Welch's t-test, paired t-test, Z-test, McNemar's test) all assume that the per-instance scores within each system-dataset-metric combination are independent and identically distributed. This assumption is not stated in Section 4.1 and is not verified in the CrossCodeEval demonstration. In CrossCodeEval, the instances are code completion tasks drawn from software repositories, and scores from the same repository are likely positively correlated; positive correlation reduces the effective sample size relative to the nominal number of instances, making the standard errors too small and the p-values anti-conservative. The claim in Section 7.3 that all 15 system pairs are significantly different on the aggregated p-value criterion is therefore potentially unreliable. The authors should state the i.i.d. assumption explicitly and provide a sensitivity analysis using cluster-robust standard errors or a cluster bootstrap (e.g., clustering by repository) to show that the conclusion is unchanged, or should qualify the claim.
  2. [6.2, 4.2] The harmonic mean p-value (HMP) aggregation in Section 6.2 is applied to p-values p_{a,b,j} that, according to Section 4.2, are already adjusted for multiple comparisons by the Holm-Sidak procedure. Wilson's HMP and its adjusted p-value formula are designed for raw p-values that are valid (uniformly distributed under the null); applying the HMP to adjusted p-values and then applying an additional multiplicity correction (L = |Omega| x K) double-counts the correction, so the aggregated p-values no longer carry the FWER guarantee claimed in the paper. The sentence 'Because the metric aggregate p-values are not raw, aggregated results cannot be further aggregated' acknowledges that the outputs are not raw, but does not address that the inputs are not raw either. The authors should feed raw per-metric p-values into the HMP with a single correct L, or provide a rigorous justification for the current double adjustment.
  3. [6.3] The standardization in Section 6.3 is not a standard z-score transformation. The denominator \hat{S}_{. ,j} sqrt(sum_b 1/n_{b,j}) is the standard error used in the denominator of a two-sample t-test for the difference of two means, not the standard error of a single group mean. Using this quantity to standardize each observed sample mean \bar{V}_{b,j} makes the rescaled values \tilde{\bar{V}}_{b,j} dependent on the number of systems B and on the between-system variation, and the weighted sum \dot{V}_b in step 3 has no clear statistical interpretation as a standardized aggregate mean. Since \dot{V}_b is the basis for the system ranking in Section 7.1 and the TOPSIS comparison in Section 7.4, the authors should justify this standardization or replace it with a more conventional effect-size or meta-analytic formulation.
  4. [6.2] The paper aggregates effect sizes across metrics by inverse-variance weighting and states that 'the interpretation thresholds for the various Cohen effect sizes metrics are the same.' This is not self-evidently true: Cohen's d is a standardized mean difference in standard-deviation units, while Cohen's h is a difference of arcsine-transformed proportions and is scaled in terms of the variance-stabilizing transform for binomial data. The commonly cited thresholds of 0.2, 0.5, and 0.8 are calibrated for d and are not universally regarded as equivalent for h. Because effect sizes are used for significance classification in the visualizations and are aggregated in Section 6.2, the authors should either restrict the pooling to comparable effect size metrics or provide a reference or derivation supporting the comparability.
  5. [7.3] The statement 'all system pairs are statistically significantly different' in Section 7.3 is stated without reporting effect sizes. Since the CrossCodeEval datasets are large, very small differences can become statistically significant; the paper itself notes in Section 4.1 that p-values are sensitive to large sample sizes and that effect sizes are needed to assess practical significance. A table of the smallest effect size among the 105 pairs, or a histogram of effect sizes, would make the practical implications of the all-significant finding much clearer. Without this, the headline result is vulnerable to the same criticism the paper levels at naive leaderboards.
minor comments (4)
  1. [1] The phrase 'performs performs statistical comparisons' is duplicated in the second paragraph of Section 1.
  2. [4.2] The reference to 'statmodels's multipletests' contains a typo; the package name is 'statsmodels'.
  3. [6.2] The notation p_{a,b, .} and e_{a,b, .} is introduced without an explicit definition of the dot subscript; please define it as shorthand for the vector of values across j.
  4. [7.4] The text says that 'the results for the other MCDM algorithms were very similar,' but no quantitative measure of agreement (e.g., rank correlation) is reported for TOPSIS or the other methods; adding a correlation coefficient would strengthen the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the framework's statistical tests and aggregations operate on external benchmark data, and the only self-citation (Ackerman et al. 2024) is a non-load-bearing footnote distinguishing the scope of the present work.

full rationale

The paper's derivation chain is input-to-output: given score samples from external evaluations (Mishra et al. 2024 on CrossCodeEval), the framework applies standard hypothesis tests chosen by Table 1, adjusts p-values (Holm-Šídák), aggregates score samples by standardization and weighted averaging (Section 6.1), and aggregates test results via Wilson's harmonic mean p-value and inverse-variance effect-size weighting (Section 6.2). Each of these operations is defined in terms of the input samples; none of the outputs are used to define the inputs or to fit the method. The headline result in Section 7.3 ('all system pairs are statistically significantly different') is a computed consequence of applying these published procedures to the external data, not a quantity that was fitted into the framework. The only self-citation, Ackerman et al. (2024) in footnote 1, explicitly separates the prior factorial-design work from this framework ('the current work does not try to optimally explore the search space of system hyperparameter values'), so it is not load-bearing. The TOPSIS/MCDM comparison in Section 7.4 is an external sanity check, not an input to the framework. The independence assumption underlying the tests (Section 4.1, Table 1) is a statistical-validity concern about the data-generating process, not circularity: the paper does not define significance in terms of that assumption's conclusion. Thus no step reduces to its own input or to a self-citation chain.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework introduces no invented physical or mathematical entities and fits no parameters. It does rely on domain assumptions about data independence and several ad-hoc statistical aggregation choices that are not rigorously justified in the paper.

assumptions (4)
  • domain assumption Per-instance score samples are independent and identically distributed within each score sample.
    All tests in Table 1 assume independent observations; dependencies would inflate significance. Invoked in Section 4.1 and Table 1.
  • ad hoc to paper The harmonic mean p-value method can be validly applied to p-values that are already adjusted by Holm-Sidak.
    Wilson's HMP is designed for raw p-values; using it on adjusted p-values is not standard and may alter the FWER guarantee. Invoked in Section 6.2.
  • ad hoc to paper Cohen's d and Cohen's h effect sizes have comparable interpretation thresholds and can be pooled.
    Section 6.2 aggregates effect sizes across metric types assuming identical thresholds, which is a simplification given d and h are different metrics.
  • ad hoc to paper The standardization of unpaired sample means in Section 6.3 yields comparable cross-dataset scores.
    The procedure divides each observed mean by the pooled standard deviation times sqrt(sum 1/n), which is not a standard z-score and may not properly account for differing sample sizes between systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Statistical multi-metric evaluation and visualization of LLM system predictive performance." pith.science (2026). https://pith.science/paper/USSZWKAX

@misc{pith2026250118243,
  author       = {Pith},
  title        = {Pith review of: Statistical multi-metric evaluation and visualization of LLM system predictive performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USSZWKAX}},
  note         = {Machine review of arXiv:2501.18243}
}
read the original abstract

The evaluation of generative or discriminative large language model (LLM)-based systems is often a complex multi-dimensional problem. Typically, a set of system configuration alternatives are evaluated on one or more benchmark datasets, each with one or more evaluation metrics, which may differ between datasets. We often want to evaluate -- with a statistical measure of significance -- whether systems perform differently either on a given dataset according to a single metric, on aggregate across metrics on a dataset, or across datasets. Such evaluations can be done to support decision-making, such as deciding whether a particular system component change (e.g., choice of LLM or hyperparameter values) significantly improves performance over the current system configuration, or, more generally, whether a fixed set of system configurations (e.g., a leaderboard list) have significantly different performances according to metrics of interest. We present a framework implementation that automatically performs the correct statistical tests, properly aggregates the statistical results across metrics and datasets (a nontrivial task), and can visualize the results. The framework is demonstrated on the multi-lingual code generation benchmark CrossCodeEval, for several state-of-the-art LLMs.

Figures

Figures reproduced from arXiv: 2501.18243 by the authors.

Figure 1
Figure 1. Illustration of terms (Section 2.1) for paired data. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Boxplots of system value and rank distribution for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Heatmap of p-values for all metrics for Cross [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Top: Connected graph using p-values for ID [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 5
Figure 5. Figure 5: Connected graph using p-values on CrossCodeE [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Scatterplots of TOPSIS and mean metric aggregate [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Condition-and-compare analysis for an example dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantifying Ranking Uncertainty in LLM Benchmarks

    cs.LG 2026-06 conditional novelty 5.0 of 10

    MMLU ranking uncertainty is dominated by subject-level variability; rank confidence intervals widen substantially when subjects are treated as the sampling unit.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Using Combinatorial Optimization to Design a High quality LLM Solution

    Samuel Ackerman, Eitan Farchi, Rami Katan, and Orna Raz. Using combinatorial optimization to design a high quality llm solution. arXiv preprint arXiv:2405.13020, 2024

    work page Pith review arXiv 2024

  2. [2]

    Simultaneous confidence intervals for ranks with application to ranking institutions

    Diaa Al Mohamad, Jelle J Goeman, and Erik W van Zwet. Simultaneous confidence intervals for ranks with application to ranking institutions. Biometrics, 78 0 (1): 0 238--247, 2022

    work page 2022

  3. [3]

    Statistical Power Analysis for the Behavioral Sciences

    Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, 2 edition, 1988. URL https://www.utstat.toronto.edu/ brunner/oldclass/378f16/readings/CohenPower.pdf

    work page 1988

  4. [4]

    The spotis rank reversal free method for multi-criteria decision-making support

    Jean Dezert, Albena Tchamova, Deqiang Han, and Jean-Marc Tacnet. The spotis rank reversal free method for multi-criteria decision-making support. In 2020 IEEE 23rd International Conference on Information Fusion (FUSION), pages 1--8. IEEE, 2020

    work page 2020

  5. [5]

    CrossCodeEval : A diverse and multilingual benchmark for cross-file code completion

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. CrossCodeEval : A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  6. [6]

    Multiobjective optimization in river basin development

    Lucien Duckstein and Serafim Opricovic. Multiobjective optimization in river basin development. Water resources research, 16 0 (1): 0 14--20, 1980

    work page 1980

  7. [7]

    Sensitivity of decisions to probability estimation errors: A reexamination

    Peter C Fishburn, Allan H Murphy, and Herbert H Isaacs. Sensitivity of decisions to probability estimation errors: A reexamination. Operations Research, 16 0 (2): 0 254--267, 1968

    work page 1968

  8. [8]

    Performances are plateauing, let's make the leaderboard steep again, 2024

    Clementine Fourrier, Nathan Habib, Konrad Szafer Alina Lozovskaya, and Thomas Wolf. Performances are plateauing, let's make the leaderboard steep again, 2024. URL https://huggingface.co/spaces/open-llm-leaderboard/blog

    work page 2024

Show all 24 references
  1. [9]

    Choosing between methods of combining-values

    Nicholas A Heard and Patrick Rubin-Delanchy. Choosing between methods of combining-values. Biometrika, 105 0 (1): 0 239--246, 2018

    2018

  2. [10]

    Methods for Multiple Attribute Decision Making, pages 58--191

    Ching-Lai Hwang and Kwangsun Yoon. Methods for Multiple Attribute Decision Making, pages 58--191. Springer Berlin Heidelberg, Berlin, Heidelberg, 1981. ISBN 978-3-642-48318-9. doi:10.1007/978-3-642-48318-9_3. URL https://doi.org/10.1007/978-3-642-48318-9_3

    1981 doi

  3. [11]

    pymcdm—the universal library for solving multi-criteria decision-making problems

    Bart omiej Kizielewicz, Andrii Shekhovtsov, and Wojciech Sa abun. pymcdm—the universal library for solving multi-criteria decision-making problems. SoftwareX, 22: 0 101368, 2023

    2023

  4. [12]

    Russell V. Lenth. emmeans: Estimated Marginal Means, aka Least-Squares Means, 2024 a . URL https://CRAN.R-project.org/package=emmeans. R package version 1.10.4

    2024

  5. [13]

    Russell V. Lenth. Comparisons and contrasts in emmeans, 2024 b . URL https://cran.r-project.org/web/packages/emmeans/vignettes/comparisons.html

    2024

  6. [14]

    Mangiafico

    Salvatore S. Mangiafico. Summary and analysis of extension program evaluation in R . Rutgers Cooperative Extension, 2016. URL http://rcompanion.org/handbook/

    2016

  7. [15]

    Granite code models: A family of open foundation models for code intelligence

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:2405.0...

    2024 arXiv

  8. [16]

    New effect size rules of thumb

    Shlomo S Sawilowsky. New effect size rules of thumb. Journal of modern applied statistical methods, 8: 0 597--599, 2009

    2009

  9. [17]

    Statsmodels: Econometric and statistical modeling with python

    Skipper Seabold and Josef Perktold. Statsmodels: Econometric and statistical modeling with python. In Proceedings of the Python in Science Conference, page 57. SciPy, 2010

    2010

  10. [18]

    A multiple criteria decision making method based on relative value distances

    Huan-jyh Shyur, Liang Yin, Hsu-shih Shih, and Chi-bin Cheng. A multiple criteria decision making method based on relative value distances. Foundations of Computing and Decision Sciences, 40 0 (4): 0 299--315, 2015

    2015

  11. [19]

    Comparative analysis of some prominent mcdm methods: A case of ranking serbian banks

    Dragisa Stanujkic, Bojan Djordjevic, and Mira Djordjevic. Comparative analysis of some prominent mcdm methods: A case of ranking serbian banks. Serbian journal of management, 8 0 (2): 0 213--241, 2013

    2013

  12. [20]

    Using effect size—or why the p value is not enough

    Gail M Sullivan and Richard Feinn. Using effect size—or why the p value is not enough. Journal of graduate medical education, 4 0 (3): 0 279--282, 2012

    2012

  13. [21]

    Calculating and synthesizing effect sizes

    III Turner, Herbert M and Robert M Bernard. Calculating and synthesizing effect sizes. Contemporary issues in communication science and disorders, 33 0 (Spring): 0 42--55, 2006

    2006

  14. [22]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J. van der Walt , Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nels...

    2020

  15. [23]

    The harmonic mean p-value for combining dependent tests

    Daniel J Wilson. The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences, 116 0 (4): 0 1195--1200, 2019

    2019

  16. [24]

    harmonicmeanp tutorial, 2024

    Daniel J Wilson. harmonicmeanp tutorial, 2024. URL https://cran.r-project.org/web/packages/harmonicmeanp/vignettes/harmonicmeanp.html

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.