Pith. sign in

REVIEW 5 cited by

Process Reward Model with Q-Value Rankings

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.11287 v2 pith:SRV56KWX submitted 2024-10-15 cs.CL cs.AI

Wendi Li , Yixuan Li This is my paper · ORCID
classification cs.CLcs.AI
keywords processmodellossq-valuerewardaddresscomparativefunction
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A generative multimodal process reward model that produces step-level critiques and corrections improves average math accuracy for six multimodal LLMs by 2.9 to 5.9 points under a refinement-based Best-of-N strategy.

  2. BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

    cs.CL 2025-07 conditional novelty 6.0 of 10

    BMMR provides a 110k-question bilingual, multimodal, college-level dataset across 300 subjects where state-of-the-art models score at most about 50%.

  3. FreePRM: Training Process Reward Models Without Ground Truth Process Labels

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A weakly supervised PRM training method using outcome-only pseudo-labels plus a buffer probability reaches 53.0% F1 on ProcessBench, beating supervised baselines in the paper's comparisons.

  4. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

  5. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling

    cs.CL 2025-02 reject novelty 4.0 of 10

    Small LLMs with compute-optimal test-time scaling can outperform much larger models on math benchmarks, but the reported strategy is selected on the same test sets used for evaluation.

Pith tools