REVIEW 16 cited by
Critique-out-Loud Reward Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.
Forward citations
Cited by 16 Pith papers
-
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
LatentRM treats reasoning traces as discrete latent variables and trains a generator end-to-end so that the scalar reward model's likelihood of the true preference ranking is maximized, outperforming scalar, generativ...
-
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
Training a language model by distilling a coach's written experiential knowledge beats training on a scalar rubric score for open-ended tasks, with better out-of-distribution transfer.
-
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
RefCritic uses reinforcement learning with two rule-based rewards, judgment accuracy and refinement accuracy, to train long-chain-of-thought critic models that improve LLM math reasoning and detect erroneous steps.
-
Breaking the Myth: Can Small Models Infer Postconditions Too?
Fine-tuning Qwen2.5-Coder-7B on 1.5K reasoning examples yields postcondition generation close to GPT-4o and better than 32B open models on Defects4J.
-
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
An offline alignment pipeline using reward-filtered self-refinement data and long chain-of-thought SFT raises an 8B model's AlpacaEval 2 win rate from 25.0% to 51.0% with roughly 14k training examples.
-
RewardAnything: Generalizable Principle-Following Reward Models
RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...
-
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
DataRubrics introduces a structured ten-dimension rubric with an LLM-as-a-judge pipeline to automatically assess dataset quality, positioned as a more measurable alternative to descriptive datasheets for conference review.
-
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Socratic-PRMBench tests process reward models on deliberately flawed math reasoning across six reasoning patterns and reports that current PRMs score below LLM critics.
-
Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
A systematic benchmark shows multi-agent debate's value depends on task difficulty, model scale, and agent diversity: limited for math unless problems are hard or models weak, but useful for safety when agents are diverse.
-
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.
-
Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.
-
GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO
GFRIEND generates chain-of-thought preference judgments, scores them by perplexity, and uses weighted multi-level preference optimization so a reward model trained on 3,000 samples rivals models trained on much larger...
-
CyberV: Cybernetics for Test-time Scaling in Video Understanding
A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.
-
Generative RLHF-V: Learning Principles from Multi-modal Human Preference
A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Sign in to comment.