Pith. sign in

REVIEW 5 cited by

Optimizing Language Models for Inference Time Objectives using Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.19595 v2 pith:F2T337YL submitted 2025-03-25 cs.LG

Yunhao Tang , Kunhao Zheng , Gabriel Synnaeve , Rémi Munos This is my paper
classification cs.LG
keywords inferenceobjectivestimetrainingmodeloptimizingperformancelanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with $k$ samples, with a focus on pass@$k$ and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@$k$ objectives compared to the baseline method.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Outcome-based Exploration for LLM Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

  2. Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening

    cs.LG 2025-06 conditional novelty 6.0 of 10

    GRPO's rank bias reinforces likely answers and neglects rare correct proofs; an unlikeliness reward that down-weights likely correct samples improves pass@N in formal theorem proving.

  3. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Using Pass@k as an RLVR reward, with bootstrap sampling and an analytical advantage formula, improves exploration and later Pass@1 performance of reasoning LLMs.

  4. Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.

  5. Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them

    cs.LG 2025-07 conditional novelty 4.0 of 10

    GRPO produces modest math gains with small knowledge loss, while SFT gives larger math gains but degrades knowledge benchmarks more, with parameter analyses suggesting the loss comes from larger mid-layer MLP updates.

Pith tools