REVIEW 16 cited by
Scaling Test-Time Compute Without Verification or RL is Suboptimal
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite substantial advances in scaling test-time compute, an ongoing debate in the community is how it should be scaled up to enable continued and efficient improvements with scaling. There are largely two approaches: first, distilling successful search or thinking traces; and second, using verification (e.g., 0/1 outcome rewards, reward models, or verifiers) to guide reinforcement learning (RL) and search algorithms. In this paper, we prove that finetuning LLMs with verifier-based (VB) methods based on RL or search is far superior to verifier-free (VF) approaches based on distilling or cloning search traces, given a fixed amount of compute/data budget. Further, we show that as we scale test-time compute (measured as the output token length) and training data, suboptimality of VF methods scales poorly compared to VB when the base pre-trained LLM presents a heterogeneous distribution over correct solution traces (e.g., different lengths, styles, etc.) and admits a non-sharp distribution over rewards on traces sampled from it. We formalize this condition using anti-concentration [Erd\H{o}s, 1945]. This implies a stronger result that VB methods scale better asymptotically, with the performance gap between VB and VF methods widening as test-time budget grows. We corroborate our theory empirically on both didactic and math reasoning problems with 3/8/32B-sized pre-trained LLMs, where we find verification is crucial for scaling test-time compute.
Forward citations
Cited by 16 Pith papers
-
Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration
Training-free LLM collaboration gains are bounded by the fixed pool's oracle gap and then by signal coverage, fidelity, and harm, measurable with a small labeled audit.
-
Asking LLMs to Verify First is Almost Free Lunch
Prompting an LLM to first verify a random/trivial candidate answer before generating its solution consistently improves accuracy over chain-of-thought prompting at small extra token cost.
-
Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online
A history-aware verifier that scores candidate actions using past interactions cuts failure rates in ambiguous robot manipulation tasks compared to using the generator alone.
-
e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
-
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Best-of-n sampling provably needs about 1/Δ samples versus 1/Δ² for self-consistency, and a constructed Transformer can route among experts using verifier feedback to reach near-optimal final responses.
-
Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.
-
Test-Time Scaling for Small VLMs on Multilingual Visual MCQ
On EXAMS-V, token budget and parseability dominate chain count, PRM-guided search, and selectors for small VLMs; the policy model itself yields the largest gain.
-
RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction
Robot policies trained on human interventions that rewind to a familiar state and then correct the mistake achieve higher long-horizon success and better data efficiency than imitation on full demonstrations alone.
-
OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
A 2.5M-example code reasoning dataset with critique traces enables Qwen2.5-based models to surpass prior open-weight distilled models on LiveCodeBench via test-time self-critique selection.
-
Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
ADAPT, a diversity-aware prefix fine-tuning method, improves best-of-N sampling efficiency for a 1.5B reasoning model, reaching 80% accuracy at N=32 versus N=256 for the baseline.
-
Faster and Better LLMs via Latency-Aware Test-Time Scaling
Test-time scaling should be tuned for wall-clock latency rather than token count; branches plus speculative decoding win under time budgets.
-
Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought
LLM chain-of-thought length converges to an optimum because of an underfitting-overfitting tradeoff, formalized in a continuous reasoning-space framework and tested with RL.
-
Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
A test-time training method that fine-tunes LoRA adapters on verifier-selected high-confidence pseudo-labels, reporting large gains on math benchmarks, but evaluated on the same queries it adapts on.
-
Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models
A probabilistic saturation model for test-time scaling is proposed and fitted to reasoning benchmarks, but the plateau 'prediction' is computed from the same per-problem data used to measure it.
- Risk-Guided Diffusion: Toward Deploying Robot Foundation Models in Space, Where Failure Is Not An Option
Discussion (0). Continue with ORCID to comment.