REVIEW 25 cited by
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions. We formalize the problem of optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute. This perspective enables us to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use a notion of cumulative regret over output tokens as a way to measure the efficacy of test-time compute. Akin to how RL algorithms can best tradeoff exploration and exploitation over training, minimizing cumulative regret would also provide the best balance between exploration and exploitation in the token stream. While we show that state-of-the-art models do not minimize regret, one can do so by maximizing a dense reward bonus in conjunction with the outcome 0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in the output stream, quantified by the change in the likelihood of eventual success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL.
Forward citations
Cited by 25 Pith papers
-
Formalizing Learning from Language Feedback with Provable Guarantees
Introduces a formal framework for learning from language feedback, a transfer eluder dimension complexity measure, and HELiX, a no-regret algorithm whose regret scales with this dimension.
-
ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.
-
BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens
A control-token insertion and two-stage training method that lets LLMs adhere to user-specified reasoning token budgets while preserving math accuracy.
-
Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
A dual-penalty RL method that compresses chain-of-thought traces by separately penalizing internal semantic stagnation and external post-answer continuation reduces reasoning length by about 40% while preserving accur...
-
Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model
Suppressing 'thinking tokens' in a 1.5B reasoning model preserves accuracy while cutting tokens, and the proposed DuP-PO RL method improves both accuracy and efficiency over GRPO.
-
Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancement
Large reasoning models already contain the ability to reason concisely; steering their internal activations or rewarding brevity can cut reasoning length by up to half without hurting accuracy.
-
e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
-
How Far Are We from Optimal Reasoning Efficiency?
The authors define a reasoning efficiency frontier and a gap metric (REG), then train models with REO-RL to shrink the gap by at least 50% with only small accuracy losses.
-
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Best-of-n sampling provably needs about 1/Δ samples versus 1/Δ² for self-consistency, and a constructed Transformer can route among experts using verifier feedback to reach near-optimal final responses.
-
AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
A stochastic scheduling framework that modulates slow-to-fast reasoning in large reasoning models at test time, improving accuracy while reducing token usage.
-
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM, a per-step progress reward model trained with LCS-based self-annotated labels, improves RL-trained GUI agent success rates on WikiHow relative to outcome reward models.
-
Structured Thoughts For Improved Reasoning And Context Pruning
Structured try/outcome SFT improves math reasoning by up to 8% over standard SFT and enables pruning ~85% of context with ~9% accuracy drop.
-
ThinkDial: An Open Recipe for Controlling Reasoning Effort in Large Language Models
An open training recipe that teaches a 32B LLM to reason in three adjustable modes, cutting token use by half or three quarters with small accuracy losses.
-
OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
A 2.5M-example code reasoning dataset with critique traces enables Qwen2.5-based models to surpass prior open-weight distilled models on LiveCodeBench via test-time self-critique selection.
-
Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
The survey's L1/L2 taxonomy and benchmark show that current reasoning models waste compute on easy problems and underthink hard ones, motivating more adaptive inference.
-
Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty
Reward shaping with a powered length penalty makes LLMs answer easy questions with far fewer tokens while preserving or slightly improving accuracy on hard math benchmarks.
-
Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
ADAPT, a diversity-aware prefix fine-tuning method, improves best-of-N sampling efficiency for a 1.5B reasoning model, reaching 80% accuracy at N=32 versus N=256 for the baseline.
-
Don't Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models
DTO constructs compressed, ground-truth-curated reasoning trajectories from LRM outputs and uses them as preferred responses in SimPO, yielding up to 12% accuracy gains and roughly 40% token reduction on math benchmarks.
-
When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning
QwQ-32B under a save-thinking prompt exhibits three modes, no thinking, explicit thinking, and implicit thinking, with no-thinking saving tokens but losing accuracy while explicit and implicit thinking preserve accuracy.
-
Numerically Optimizing Shortcuts to Adiabaticity: A Hybrid Control Strategy
Combining analytical shortcuts to adiabaticity with numerical optimization yields up to 1000× better two-ion separation protocols without extra experimental cost.
-
AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up
The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.
-
SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
SPA-RL attributes a single final task reward to individual agent steps and uses the attributions plus an executability signal as dense rewards for PPO, yielding modest success-rate gains on WebShop, ALFWorld, and VirtualHome.
-
ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy
Skipping explicit chain-of-thought reasoning entirely, and prompting for a formatted answer, matches full CoT accuracy on several benchmarks while cutting latency and token counts.
-
Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models
A probabilistic saturation model for test-time scaling is proposed and fitted to reasoning benchmarks, but the plateau 'prediction' is computed from the same per-problem data used to measure it.
Discussion (0). Sign in to comment.