Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that single-run evaluations and mean-only leaderboards are statistically unsound, and that paired comparisons with uncertainty bars are necessary for trustworthy model rankings.

desk verdict A useful, honest experience report about running a community eval repository; the statistical section is the weak link—it prescribes a rule without showing it works. read the letter →

arxiv 2507.06893 v1 pith:FCLK7U2D submitted 2025-07-09 cs.CL cs.AI

Alexandra Abbas , Celia Waggoner , Justin Olive This is my paper · ORCID
classification cs.CLcs.AI
keywords AIevaluationslargelanguagemodelbenchmarkinguncertaintyquantificationpairedstatisticalanalysisresamplingmethodologyopen-sourceevaluationrepositoryleaderboardreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper uses eight months of maintaining an open-source repository of more than 70 AI evaluations to argue that AI evaluation needs its own statistical and operational discipline, not just adapted software practice. Its central claim is that single evaluation runs and leaderboards that compare only mean scores are methodologically unsound, because LLM outputs are nondeterministic and small score gaps can be pure noise. To fix this, it proposes a resampling rule for deciding how many runs are enough and paired statistical tests that exploit the correlation between models scored on the same questions. It also reports that structured volunteer cohorts with staged reviews and automated checks can scale community contributions while keeping quality. The reason to care is that these are the practices that would let published benchmark results, safety checks, and model rankings actually support the conclusions drawn from them.

What carries the argument

The statistical machinery is a resampling criterion $\mathbb{E}[\sigma_i^2]/K \ll \mathrm{Var}(\bar{x})$: after a pilot of 100 random samples and 10 epochs, keep adding runs $K$ until the expected within-sample variance divided by $K$ is much smaller than the variance of the sample means, at which point extra runs give diminishing returns. The comparison machinery is paired analysis, which takes per-question score differences $d_i$ between two models, computes $\bar{d} = \frac{1}{N}\sum_i d_i$, and uses the standard error $\mathrm{SD}_d/\sqrt{N}$ to build a 95% confidence interval; this removes question-difficulty variance that widens unpaired intervals. The operational machinery is the structured cohort: 5-week volunteer cohorts, one technical project manager per 5–10 engineers, a reviewed Benchmark Development Plan before implementation, peer verification on data subsets, CI unit and integration tests, and manual log review to catch tool failures and reward hacking.

What would settle it

Run an agentic benchmark where most samples score zero and a few score one, estimate the variance quantities from a 100-sample pilot, and then repeat the full evaluation many times: if the rule calls for a small number of runs while full-run rankings flip sign across repetitions, or if simulated 95% paired confidence intervals cover the true mean difference noticeably less than 95% of the time, the central methodological claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that reliable AI evaluation requires three practices the field currently treats as optional: repeated runs with a principled stopping point, uncertainty quantification on every reported score, and paired comparison on matched samples rather than raw means. The authors state the result directly: "common practices like single evaluation runs and mean-only leaderboard comparisons are methodologically unsound," while "paired statistical analysis and proper uncertainty quantification are essential for reliable model comparisons." They also report that a five-week cohort model with technical project managers, benchmark development plans, peer verification, CI testing, and ±5% reference-result checks kept the repository's 70+ evaluations trustworthy, and that shared infrastructure—automated validation, trusted private test sets, and provenance-tracked result logs—will be needed for the field to solve reproducibility and contamination together.

Load-bearing premise

The advice depends on a variance-based rule telling evaluators when extra runs stop mattering, and the authors assume it holds for AI evaluation scores without demonstrating it for sparse, uneven tasks such as agentic benchmarks.

Editorial extensions

If this is right

  • Single-run benchmark numbers should be treated as noise-prone point estimates, and evaluation budgets should include enough repeated runs to reach the resampling criterion.
  • Leaderboards should report standard errors or paired confidence intervals; otherwise they present measurement noise as capability differences.
  • Evaluation repositories can scale community contributions through cohort programs with design reviews, peer verification, and automated tests, rather than relying on ad hoc individual contributions.
  • Shared infrastructure—automated validation, trusted private test sets, and a provenance-tracked evaluation-result log—would reduce redundant computation and reduce contamination risk.
  • Cost differences between frontier and open-weight models are large enough that rigorous repeated evaluation is affordable for many comparisons, so cost alone is not a reason to skip uncertainty quantification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper's stopping rule is applied to benchmarks whose per-sample scores are heavily skewed or mostly zero, as in agentic tasks, it may understate the number of runs needed; a bootstrap or stratified-resampling variant would be a natural test of the paper's advice.
  • The case against mean-only leaderboards points beyond the paper toward tournament-style pairwise comparisons with multiple-comparison corrections for multi-model rankings, which the authors list as an open statistical challenge.
  • A trusted shared log of evaluation outcomes would let smaller groups reuse statistical evidence from prior runs, turning repeated evaluation into a public good instead of duplicated private compute.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports lessons from eight months of maintaining inspect_evals, an open-source repository of 70+ community-contributed LLM evaluations. The authors describe a structured cohort management program for volunteer contributors, a quality-assurance workflow including benchmark development plans, peer verification, and CI testing, and two statistical prescriptions: an optimal resampling rule (Section 3.5) and paired versus unpaired confidence intervals for cross-model comparison (Section 3.6). The central conclusion is that single evaluation runs and mean-only leaderboard comparisons are methodologically unsound, and that the field needs shared infrastructure, private test-set protocols, and automated validation. The paper is primarily an experience report; it contains no experiments, no data, and no quantitative evaluation of the proposed methods.

Significance. If the claims were substantiated, the paper would provide a useful operational blueprint for community-maintained evaluation repositories and would reinforce the growing consensus that uncertainty quantification is necessary in LLM benchmarking. The qualitative descriptions of cohort management, QA workflows, and cost trade-offs are plausible and consistent with prior work such as Miller (2024). The paper also makes a concrete contribution by naming specific infrastructure gaps (private test sets, trusted evaluation logs, automated validation) that the community can act on. However, the statistical centerpiece is asserted rather than demonstrated: no data, results, or reproducible artifacts are provided to back the conclusion that single runs and mean-only leaderboards are unsound, and the resampling rule is imported from a citation without validation on the repository's heterogeneous evaluations. The significance is therefore conditional on a substantial empirical supplement.

major comments (3)
  1. [Section 3.5, 'Optimal resampling determination']
  2. [Section 5, first paragraph]
  3. [Section 3.3, 'Result validation']
minor comments (5)
  1. [Throughout]
  2. [Appendix A]
  3. [Section 3.6.2]
  4. [References]
  5. [Section 3.5]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the statistical prescriptions are cited from an external source (Miller 2024) and standard formulas, and no fitted value is renamed as a prediction.

full rationale

The paper's central statistical claims are not derived from its own inputs. Section 3.5 invokes Miller's equation E[sigma_i^2]/K << Var(xbar) as an external stopping rule and cites Abbas (2025b) only as the application protocol (100 samples, 10 epochs); the optimal K is not reported and is not an input to any prediction. Section 3.6.2 uses the standard unpaired and paired standard-error formulas (Miller 2024; Abbas 2025a), which do not depend on fitted values from this paper. The conclusion that single evaluation runs and mean-only leaderboards are unsound follows from elementary variance decomposition rather than from a quantity fitted in the paper. The self-authored Colab notebooks are cited as methodological references, but no load-bearing argument reduces to an unverified self-citation: the variance formulas are externally checkable and standard, and the paper presents no numerical 'prediction' that is equivalent by construction to its pilot estimates. Concerns about the small pilot sample and the lack of a demonstrated validity of Miller's rule for sparse agentic evaluations are correctness/evidence issues, not circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted constants or new entities. Its quantitative operational rule (Miller's equation) is cited from prior literature, and the pilot sample size used to estimate variances is chosen by hand without justification. The central claims are qualitative experience assertions rather than derivations.

free parameters (1)
  • Pilot variance estimation size = 10 epochs over 100 random samples
    Chosen by hand in Section 3.5 to compute E[sigma_i^2] and Var(x_bar); no power analysis or justification is given, and the optimal K depends on these pilot statistics.
assumptions (3)
  • domain assumption Miller's equation E[sigma_i^2]/K << Var(x_bar) identifies the optimal number of resampling iterations
    Invoked in Section 3.5 without proof; the paper assumes this stopping rule transfers to LLM evaluation score distributions.
  • domain assumption Unelicited model responses are a fairer basis for cross-model comparison than elicitation-optimized responses
    Section 3.6.1 chooses unelicited responses while acknowledging prompt biases favor some models; the fairness claim is assumed, not measured.
  • standard math Paired per-question differences yield valid standard errors for model score comparisons
    Standard paired-difference statistics underlie Section 3.6.2; this is textbook material and not at issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights." pith.science (2026). https://pith.science/paper/FCLK7U2D

@misc{pith2026250706893,
  author       = {Pith},
  title        = {Pith review of: Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCLK7U2D}},
  note         = {Machine review of arXiv:2507.06893}
}
abstract

AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining $inspect\_evals$, an open-source repository of 70+ community-contributed AI evaluations. We identify key challenges in implementing and maintaining AI evaluations and develop solutions including: (1) a structured cohort management framework for scaling community contributions, (2) statistical methodologies for optimal resampling and cross-model comparison with uncertainty quantification, and (3) systematic quality control processes for reproducibility. Our analysis reveals that AI evaluation requires specialized infrastructure, statistical rigor, and community coordination beyond traditional software development practices.

Figures

Figures reproduced from arXiv: 2507.06893 by the authors.

Figure 1
Figure 1. Cohort management structure showing one Technical Project Manager overseeing multiple Software Engineers with bidirectional review processes for coordinated evaluation imple￾mentation and scalability. Outside of the benchmarks cohort program, we create GitHub issues to signal to the open-source community which evaluations we would be particularly excited to see implemented in inspect evals. We provide support to ind… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [1]

    Pairwise analysis of model performance

    Abbas, A. Pairwise analysis of model performance. Google Colab Notebook, 2025 a . URL https://colab.research.google.com/drive/1dgJEjbjuyYB1FlKQqN2d1wtYQbcE54OK?usp=drive_link

    work page 2025

  2. [2]

    Calculating optimal resampling for model evaluation

    Abbas, A. Calculating optimal resampling for model evaluation. Google Colab Notebook, 2025 b . URL https://colab.research.google.com/drive/1N0LQcXI0YSLQdyHXBWy-qX_FMkor6dnp?usp=drive_link

    work page 2025

  3. [3]

    The AI evaluation substack, 2024

    AI Evaluation Digest . The AI evaluation substack, 2024. URL https://aievaluation.substack.com/. Substack newsletter

    work page 2024

  4. [4]

    AgentHarm : A benchmark for measuring harmfulness of LLM agents

    Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., Winsor, E., Wynne, J., Gal, Y., and Davies, X. AgentHarm : A benchmark for measuring harmfulness of LLM agents. arXiv preprint, arXiv:2410.09024, 2024. doi:10.48550/arXiv.2410.09024. URL https://arxiv.org/abs/2410.09024

    arXiv doi:10.48550/arxiv.2410.09024

  5. [5]

    Autonomous systems evaluation standard

    Black, S., Arbeid, J., Bailey, J., Cooney, A., Sourbut, O., Schmatz, M., and Wang, J. Autonomous systems evaluation standard. Technical report, UK AI Security Institute , 2024. URL https://ukgovernmentbeis.github.io/as-evaluation-standard/. GitHub repository: https://github.com/UKGovernmentBEIS/as-evaluation-standard

    work page 2024

  6. [6]

    GAIA : a benchmark for general ai assistants

    Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. GAIA : a benchmark for general ai assistants. arXiv preprint, arXiv:2311.12983, 2023. doi:10.48550/arXiv.2311.12983. URL https://arxiv.org/abs/2311.12983

    arXiv doi:10.48550/arxiv.2311.12983 2023

  7. [7]

    Adding error bars to evals: A statistical approach to language model evaluations

    Miller, E. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint, arXiv:2411.00640, 2024. doi:10.48550/arXiv.2411.00640. URL https://arxiv.org/abs/2411.00640

    arXiv doi:10.48550/arxiv.2411.00640 2024

  8. [8]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A graduate-level google-proof Q&A benchmark. arXiv preprint, arXiv:2311.12022, 2023. doi:10.48550/arXiv.2311.12022. URL https://arxiv.org/abs/2311.12022

    arXiv doi:10.48550/arxiv.2311.12022 2023

Show all 13 references
  1. [9]

    inspect\_ai: A framework for large language model evaluations, 2024

    UK AI Security Institute . inspect\_ai: A framework for large language model evaluations, 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai. GitHub repository

    2024

  2. [10]

    Research agenda

    UK AI Security Institute . Research agenda. https://www.aisi.gov.uk/research-agenda, 2025

    2025

  3. [11]

    inspect\_evals: Collection of evals for Inspect AI , 2024

    UK AI Security Institute , Arcadia Impact , and Vector Institute . inspect\_evals: Collection of evals for Inspect AI , 2024. URL https://anonymous.4open.science/r/inspect_evals-57FD. GitHub repository

    2024

  4. [12]

    K., Perry, N., Dulepet, R., Ji, J., Menders, C., Lin, J

    Zhang, A. K., Perry, N., Dulepet, R., Ji, J., Menders, C., Lin, J. W., Jones, E., Hussein, G., Liu, S., Jasper, D. J., Peetathawatchai, P., Glenn, A., Sivashankar, V., Zamoshchin, D., Glikbarg, L., Askaryar, D., Yang, H., Zhang, A., Alluri, R., Tran, N., Sangpisit, R., Oseleon...

    2024 doi arXiv

  5. [13]

    M., Moros‑Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J

    Zhou, L., Pacchiardi, L., Martínez‑Plumed, F., Collins, K. M., Moros‑Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J. E., Li, Z., Sánchez‑García, P., Chen, K. J., Casares, P. A. M., Zu, J., Burden, J., Mehrbakhsh, B., Stillwell, D., Cebrian, M., Wang, J., Henders...

    2025 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.