Pith. sign in

REVIEW 5 cited by

Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.03814 v1 pith:MA54ZLBF submitted 2025-05-02 stat.ML cs.AIcs.CLcs.LG

Ganghua Wang , Zhaorun Chen , Bo Li , Haifeng Xu This is my paper · ORCID
classification stat.MLcs.AIcs.CLcs.LG
keywords evaluationtestcer-evalcertifiableframeworkllmsmodelspoints
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DAPRO provides the first dynamic, theoretically guaranteed way to allocate interaction budgets across test cases for bounding time-to-event in multi-turn LLM evaluations, achieving tighter coverage than static conform...

  2. BayesAME: Bayesian Active Model Evaluation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A sequential Bayesian method automatically grows a coreset until performance estimate and uncertainty stabilize, outperforming adapted baselines and showing active selection beats random when reference signals are rich.

  3. SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SocialMaze is a six-task benchmark that claims to evaluate LLM social reasoning along deep reasoning, dynamic interaction, and information uncertainty dimensions.

  4. ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Partial LLM-agent evaluations can support a promote/reject decision after 15–25% of tasks on some benchmarks, but not others; reports should state the decision rule and unresolved count.

  5. ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

    cs.AI 2025-12 reject novelty 5.0 of 10

    LLM reasoning benchmark scores vary substantially across repeated runs under the same model, strategy, and task, so single-run evaluation can misrank systems.

Pith tools