Pith. sign in

REVIEW 7 cited by

Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.21878 v2 pith:DT7KGEF3 submitted 2025-03-27 cs.AI cs.LGstat.ML

Audrey Huang , Adam Block , Qinghua Liu , Nan Jiang , Akshay Krishnamurthy , Dylan J. Foster This is my paper · ORCID
classification cs.AIcs.LGstat.ML
keywords performanceinference-timealignmentcoveragerewardcomputationcomputehacking
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Inference-time computation offers a powerful axis for scaling the performance of language models. However, naively increasing computation in techniques like Best-of-N sampling can lead to performance degradation due to reward hacking. Toward a theoretical understanding of how to best leverage additional computation, we focus on inference-time alignment, which we formalize as the problem of improving the quality of responses drawn from a pre-trained policy, given a prompt of interest and access to an imperfect reward model. We analyze the performance of inference-time alignment algorithms in terms of (i) response quality, and (ii) compute, and provide new results that highlight the importance of the pre-trained policy's coverage over high-quality responses for performance and compute scaling: 1. We show that Best-of-$N$ alignment with an ideal choice for $N$ can achieve optimal performance under stringent notions of coverage, but provably suffers from reward hacking when $N$ is large, and fails to achieve tight guarantees under more realistic coverage conditions. 2. We introduce $\texttt{InferenceTimePessimism}$, a new algorithm which mitigates reward hacking through deliberate use of inference-time compute, implementing the principle of pessimism in the face of uncertainty via rejection sampling; we prove that its performance is optimal and does not degrade with $N$, meaning it is scaling-monotonic. We complement our theoretical results with an experimental evaluation that demonstrate the benefits of $\texttt{InferenceTimePessimism}$ across a variety of tasks and models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding Reasoning from Pretraining to Post-Training

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A joint scaling law: post-RL chess and math performance is predictable from pretraining loss, RL improvement rate grows with pretraining tokens, and RL both amplifies and discovers moves depending on difficulty.

  2. Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

    stat.ML 2025-07 conditional novelty 6.0 of 10

    Smoothed Best-of-N has finite-sample KL and regret bounds under imperfect reward models, and tuning its temperature can make its regret bound beat hard Best-of-N in the overoptimization regime.

  3. Energy-Based Transformers are Scalable Learners and Thinkers

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...

  4. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  5. Test-time reward-guided alignment of language models by importance sampling on pre-logit space

    cs.LG 2025-10 conditional novelty 5.0 of 10

    AISP is a training-free decode-time alignment method: Gaussian-perturb LLM pre-logits, score sampled responses with a reward model, and iteratively shift the perturbation mean by reward-weighted importance sampling.

  6. Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment

    cs.LG 2025-10 reject novelty 5.0 of 10

    A fixed ReLU penalty and a semantically labeled cost model are proposed to make RLHF safer, but the 'certifiable' guarantee is not fully supported.

  7. Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    ADAPT, a diversity-aware prefix fine-tuning method, improves best-of-N sampling efficiency for a 1.5B reasoning model, reaching 80% accuracy at N=32 versus N=256 for the baseline.

Pith tools