Pith. sign in

REVIEW 4 cited by

Self-Generated Critiques Boost Reward Modeling for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.16646 v3 pith:MNTG64K7 submitted 2024-11-25 cs.CL cs.AIcs.LG

Yue Yu , Zhengxing Chen , Aston Zhang , Liang Tan , Chenguang Zhu , Richard Yuanzhe Pang , Yundi Qian , Xuewei Wang
show 5 more authors
Suchin Gururangan Chao Zhang Melanie Kambadur Dhruv Mahajan Rui Hou
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords rewardcritiquesmodelsmodelingcritic-rmlanguageaccuracyhuman
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RewardAnything: Generalizable Principle-Following Reward Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...

  2. Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.

  3. MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.

  4. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools