REVIEW 5 major objections 5 minor 4 cited by
Generalist Reward Models: Found Inside Large Language Models
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Any language model trained by ordinary next-token prediction already contains a generalist reward model: its logits provably solve an offline inverse reinforcement learning objective, so a training-free reward can be elicited and used for…
desk verdict A clean algebraic identity yields standard imitation-learning bounds, but the 'any LLM' scope claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the inverse soft Bellman operator of inverse soft Q-learning, $r(s_h,a_h)=Q(s_h,a_h)-\alpha\log\sum_{a_{h+1}}\exp(Q(s_{h+1},a_{h+1})/\alpha)$, applied to the language model's logits treated as an estimated Q-function. The key identity is that the next-token prediction maximum-likelihood objective and the offline IRL objective are the same optimization problem, so the softmax of the logits is simultaneously the policy and the Q-function. Substituting it into the operator turns each token's log-probability into a dense token-level reward whose value-function terms are a potential-based reward shaping, which preserves optimal policies; at the trajectory level the reward collapses to the log-likelihood of the whole response plus a prompt-only term.
What would settle it
Construct a synthetic token-level MDP with a known true reward, compute the entropy-regularized optimal expert policy, sample demonstrations, train a tabular next-token policy by maximum likelihood, extract the endogenous reward, and re-train exactly to optimality; if the extracted reward is not the true reward up to the stated potential term, or if the re-trained policy's advantage over the base policy does not follow the predicted linear-versus-quadratic behavior in the horizon $H$, the paper's central identification fails.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that generation and evaluation are two uses of the same trained object. A next-token prediction model $\hat{\pi}$ with logits $\hat{f}$, written $\hat{\pi}(\cdot|s_h)=\mathrm{softmax}(\hat{f}(s_h,\cdot);\alpha)$, is the maximum-likelihood solution of the ordinary next-token objective; rewriting the offline inverse soft Q-learning objective shows it is the same maximization, so the logits are a Q-function for the data distribution. Feeding those logits into the inverse soft Bellman operator gives the endogenous reward $\hat{r}(s_h,a_h)=\alpha\log\hat{\pi}(a_h|s_h)+V_{\hat{Q}}(s_h)-V_{\hat{Q}}(s_{h+1})$, a potential-shaped version of the token log-probability reward. Over a full response this telescopes to $\alpha\log\hat{\pi}(\tau|s_1)$ plus a prompt-only constant, so the model's own probability of a response is its reward. The paper claims this is not a heuristic but the very reward an offline IRL algorithm would learn, and that RL on it corrects the compounding errors of imitation learning, giving the first theoretical proof it knows of that RL fine-tuning improves on the base model.
Load-bearing premise
The load-bearing premise is that the demonstration data came from a policy that is itself the entropy-regularized optimal policy for the true reward at the fixed temperature coefficient $\alpha$, and that the follow-up reinforcement learning is solved exactly rather than approximately.
Editorial extensions
If this is right
- The separate reward-model training stage of RLHF can be replaced by reading log-probabilities out of the base model, eliminating the need to collect preference labels or train a scorer.
- Because the reward is a shaped version of the base model's own log-probabilities, RL fine-tuning on it provably reduces the compounding-error penalty from $O(H^2)$ to $O(H)$; the math-reasoning results are the empirical form of that prediction.
- The endogenous reward is promptable: changing the system instruction changes the evaluation criterion of the same frozen model, which yields domain-specific judges without retraining.
- Existing generative reward models that score by the probability of a 'Yes' token or by pairwise 'Yes/No' comparisons are special cases of this formulation and inherit its theoretical grounding.
- Any autoregressive model, including image, video, or audio generation models, can in principle be aligned the same way, which extends RL-based alignment to modalities where human preference data is impractical to collect.
Reading between the lines
- An implication the paper does not draw out: because the reward is elicited rather than learned, its practical ceiling is exactly the quality of the base model's probability estimates; on preferences where those probabilities encode stylistic bias rather than correctness, RL on the endogenous reward could amplify the bias.
- The linear-in-$H$ advantage is proven for exact RL; under the approximate RL actually used in training, the margin over behavior cloning should narrow, and an ablation varying optimization steps or KL penalty would map where the theoretical benefit survives.
- The same identity suggests a cheap transfer test: any autoregressive multi-modal generator can be probed with this operator, and the method should transfer wherever token-level likelihood is well calibrated, and fail where it is not.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that any LLM trained by standard next-token prediction already contains a 'generalist reward model': its logits are shown to solve an offline inverse reinforcement learning (IRL) objective (Proposition 1), and the resulting 'endogenous reward' is used for reinforcement learning fine-tuning, with a claimed O(H) sub-optimality bound versus O(H^2) for behavior cloning (Theorem 2). The authors also provide experiments on RM-Bench, Multifaceted-Bench, DSP, and MATH-lighteval, reporting that the training-free endogenous reward outperforms LLM-as-a-judge baselines and is competitive with explicitly trained reward models, and that RL fine-tuning with it improves math reasoning. The theoretical core consists of Proposition 1, Theorem 1 (a total-variation bound on preference distributions), and Theorem 2 (a policy sub-optimality comparison), all built on the assumption that the expert policy is entropy-regularized optimal for the true reward.
Significance. If the central claims held, the paper would be significant: it would provide a theoretical justification for replacing the reward-modeling stage of RLHF with a training-free elicitation of log-probabilities from a base model, and it would prove a concrete advantage of RL over behavior cloning in the LLM setting. The paper also makes several useful secondary observations: Eq. (12) explicitly connects the outcome reward to α log π(τ|s1) plus a prompt-dependent potential; the reward-shaping perspective in §3.3 is correct; and the framing of existing generative verifiers as special cases of log-probability rewards is a useful unification. The proofs are transparent and the derivations in Appendix A are internally consistent under the stated idealizations. However, the significance is substantially reduced by the gap between the 'any LLM' scope claim and the actual assumptions: Eq. (13) assumes the training data are expert demonstrations for a coherent target reward, so the theorems apply to MLE policies trained on expert data, not to arbitrary web-scale pretrained models.
major comments (5)
- [§4.1, Eq. (13)] The 'any LLM' scope claim is contradicted by the assumption underlying both theorems. Eq. (13) asserts that the expert policy πE is the entropy-regularized optimal policy for the true reward r⋆; this is an assumption about the data-generating process, not a property of all next-token-prediction training. For web-scale pretraining data, no reward r⋆ is specified and no argument is given that the data are expert demonstrations for a coherent objective. Consequently, Proposition 1 shows only that the logits of any MLE policy solve the offline IRL objective for the data distribution actually used in training; Theorem 1 and Theorem 2 then measure how close bπ's log-probabilities are to πE's, not the discovery of a generalist reward in an arbitrary LLM. The paper should explicitly re-scope its claims to 'models trained on expert demonstrations for the target task' or provide a separate argument for the pretraining case.
- [§4.2, Theorem 2] The O(H) bound in Theorem 2 assumes exact RL: the theorem statement defines πRL = argmaxπ Vπ_br and the text says 'we ignore any optimization error in solving the optimal policy.' The experiments, however, use practical RL algorithms with KL penalties, sampling, and finite rollouts, so the stated bound does not apply to the reported RLFT results. To support the empirical claim of self-improvement, the paper needs either (i) a bound that includes optimization error, or (ii) an explicit argument that the experimental setup approximately satisfies the exact-RL idealization, ideally with a measurement of the optimization gap.
- [Appendix C, Eq. (10)-(12)] The empirical reward computation deviates from the theoretical outcome reward. Eq. (12) is derived by telescoping br(sh, ah) = α log bπ(ah|sh) + V(s_h) − V(s_{h+1}), which requires the undiscounted, unfloored sum. Appendix C instead uses max(γ^{h−1}, β) with γ = 0.95 or 0.93 and β = 0 or 0.03. This breaks the telescoping, so the reward used in the RM-Bench, Multifaceted-Bench, and RL experiments is not the reward analyzed in Theorem 1 and Theorem 2. Either the experiments should use the exact theoretical reward from Eq. (12), or the theory should be extended to discounted and floored rewards, including the corresponding value-function and error-bound analysis.
- [§3.2, Eq. (8) and Proposition 1] Proposition 1 is presented as the 'theoretical cornerstone' and as evidence that a reward model is 'latently present' in any LLM, but the argument is definitional: Eq. (8) is the next-token prediction objective from Eq. (1) rewritten with a softmax parameterization of the policy, so any softmax-parameterized MLE policy's logits solve the offline IRL objective by construction. The paper should state this plainly and build its case on the conditions under which the MLE policy is a good reward, rather than presenting the algebraic equivalence itself as the discovery. As written, the proposition gives no information about whether the endogenous reward is the true reward for any downstream task.
- [Introduction and §4.2] The abstract and introduction claim 'the first theoretical proof of the effectiveness of reinforcement learning for LLMs.' This is not supported by the manuscript: the quadratic-to-linear improvement in horizon dependence is a known result in imitation learning (e.g., Ross and Bagnell, 2010; Xu et al., 2021; Rajaraman et al., 2020), and Theorem 2 is a re-derivation of that style of bound in a token-level MDP under an additional entropy-regularization assumption. The claim should be softened and the prior bounds cited and differentiated, or the specific novelty (the reward-extraction mechanism) should be identified independently of the H-dependence comparison.
minor comments (5)
- [Throughout] There are several typos: 'This paper discover' should be 'This paper discovers' (§1); 'closed to' should be 'close to' (after Theorem 1); 'quadractic' should be 'quadratic' (§4.2); 'Multifacted-Bench' and 'Multifaceted-Bench' are used inconsistently; and the reference to 'prepring' should be 'preprint'.
- [§5.1 and Appendix C] The temperature settings are inconsistent: §5.1 says 'The temperatures for training and test are set to 1.0 and 0.6, respectively,' while Appendix C says 'All sampling temperatures are set to 1.0.' Please clarify which temperature is used for evaluation.
- [§5.4] The RL method is described as using the approach of Shao et al. [2024], but the cited reference is DeepSeekMath, which does not introduce GRPO in the form used here; please cite the actual algorithm paper or describe the RL procedure precisely, including the KL coefficient and the exact objective being optimized.
- [Appendix B.2] The proof of Theorem 1 contains a duplicated inequality line: the two identical lines bounding |log πE(τ) − log bπ(τ)| + |log πE(τ') − log bπ(τ')| appear consecutively; the duplicate should be removed.
- [§7.4] The discussion of extending the method to image, video, and audio generation claims that any autoregressive model can be prompted to evaluate its own output, but it does not address the absence of a well-defined 'token-level reward' or the fact that the theoretical results in §4 assume a token-level MDP with the same vocabulary for actions and states. Adding a caveat or a short argument for the multimodal extension would improve rigor.
Circularity Check
The 'logits are the IRL Q-function' result is the identity Eq. (6) ≡ Eq. (8) ≡ Eq. (1), and Theorems 1-2 connect the endogenous reward to the 'true reward' only through Eq. (13), which defines the target reward up to a potential as the expert's log-probability; the predicted reward and its claimed superiority therefore reduce to the base model's own fit.
-
self definitional
[Section 3.2 (Eqs. 6-8, Proposition 1) and Appendix B.1 (Eq. 16)]
"By applying a simple log-of-exponent transformation, the objective in Eq. (6) can be expressed as a maximum likelihood problem: ... The term inside the logarithm can be regarded as a policy, π_Q(·|sh) = softmax(Q(sh, ·); α). Therefore, the optimization problem seeks the Q-function whose corresponding policy, π_Q, maximizes the likelihood of the expert demonstrations in dataset D. This is precisely the goal of training a language model by next-token prediction. ... By comparing Eq.(16) and Eq.(6), we can conclude that bf is the optimal solution to the offline IRL objective of Eq.(6)."
By the paper's own Eq. (8), the offline IRL objective Eq. (6) is α times the softmax next-token log-likelihood: the summand log[exp(Q/α) / Σ exp(Q/α)] is exactly the MLE term of Eq. (1) with π = softmax(Q/α). The paper then states this 'is precisely the goal of training a language model by next-token prediction,' and the proof of Proposition 1 (B.1) concludes bf solves Eq. (6) by comparing Eq. (16) with Eq. (6). The two objectives share the same argmax by construction, because the inverse soft Bellman change of variable (Eqs. 7 and 15) was chosen exactly to turn MaxEnt IRL into this MLE objective.
-
fitted input called prediction
[Section 4.1 (Eq. 13), Section 3.3 (Eq. 12), and Appendix B.2 (proof of Theorem 1)]
"following maximum entropy inverse reinforcement learning [Ziebart et al., 2008], we consider that the expert policy πE is the entropy-regularized optimal policy regarding the unknown true reward r⋆. According to the theory on regularized MDPs [Geist et al., 2019], it holds that r⋆(sh, ah) = α log(πE(ah|sh)) + VQ⋆r⋆(sh) − VQ⋆r⋆(sh+1). ... According to Eq.(12), we have that r⋆(τ) − r⋆(τ′) = α(log πE(τ|s1) − log πE(τ′|s1)), br(τ) − br(τ′) = α(log bπ(τ|s1) − log bπ(τ′|s1))."
Theorems 1 and 2 claim the endogenous reward br, extracted from the fitted model bπ, is close to the unknown true reward r⋆ and that RL on br is provably superior. But the target r⋆ is fixed by Eq. (13) to be, up to a potential, α log πE — the log-probability of the data-generating expert — while Eq. (12) defines br, up to a potential, as α log bπ, the fitted model's own log-likelihood. The proof of Theorem 1 substitutes r⋆(τ) − r⋆(τ′) = α(log πE(τ|s1) − log πE(τ′|s1)) and br(τ) − br(τ′) = α(log bπ(τ|s1) − log bπ(τ′|s1)); Theorem 2's gap is αΣ|log πE − log bπ|. So the predicted reward quality and its superiority bound reduce to the statement that bπ fits the log-probabilities of the very data it was trained on (Eq. 1). No new reward is derived: under Eq.
1 more flagged steps
-
self definitional
[Section 4.3, 'Ineffectiveness of Iterative Improvement']
"By construction, the policy πRL is the optimal policy for the reward br. Consequently, the endogenous reward extracted from πRL is precisely the one for which πRL is already optimal. Applying another RL step would therefore yield the same policy, causing the process to converge immediately."
The paper itself concedes the fixed-point nature of the construction: πRL maximizes br, and the reward extracted from πRL is the one for which πRL is already optimal, so iteration stalls immediately. This admission, introduced with 'By construction', confirms that the endogenous reward is not an independently grounded evaluator but the policy's own softmax log-probability (Eq. 12): 'reward discovery' and 'reward optimization' operate on the same object. The self-referential identity is acknowledged in the manuscript itself and weighs against treating the elicited reward as an external or newly discovered quantity.
full rationale
The derivation chain has two theoretical pillars, and both partially reduce to their own inputs. (1) Proposition 1 is the algebraic identity Eq. (6) ≡ Eq. (8) ≡ Eq. (1): the offline IRL objective of Appendix A is obtained via the inverse soft Bellman change of variable, which turns MaxEnt IRL into the softmax MLE objective, so the statement that a model's logits solve the IRL objective restates that the model was trained to maximize next-token log-probability. The paper is transparent that Eq. (8) 'is precisely the goal of training a language model by next-token prediction', so the concern is not hidden algebra but the framing of a reparameterization as a discovery. (2) Theorems 1-2 connect the endogenous reward to the true reward only through Eq. (13), which defines r⋆ up to a potential as α log πE, while Eq. (12) defines br up to a potential as α log bπ; the bounds then measure |log πE − log bπ|, the model's own misfit on its training data. The assumption that a pretraining corpus was generated by an expert optimizing a coherent r⋆ is not argued, so the headline claim about 'any LLM trained via standard next-token prediction' is not supported by the theorems; the theory at most supports the conditional and known statement that if data are expert demonstrations for r⋆, MLE log-probability is a reward. Section 4.3's admission that iterative self-improvement fails 'by construction' corroborates the fixed-point character. Mitigating factors per rule 4: the empirical section is genuinely external — RM-Bench, Multifaceted-Bench, DSP, and the MATH suite are independent benchmarks, and EndoRM's reward is not fitted to those test sets — and the self-citations (Xu et al. 2020/2021/2022, including Lemma 4 used in Theorem 2's proof) are technical imitation-learning bounds with stated assumptions that do not include the target result, so they are real evidence and do not raise the score. Also weighed as in-scope limitations: Section 4.2 states 'we ignore any optimization error in solving the optimal policy', and Appendix C implements a discounted and floored reward (γ = 0.93–0.95, β ≥ 0), so the experiments do not directly test the O(H) bound; these caveats widen the gap between theory and the 'any LLM' claim but are correctness concerns, not additional circularity.
Assumptions & free parameters
free parameters (3)
- alpha (entropy regularization / softmax temperature) =
not reported in experiments
- discount gamma =
0.95 (Multifaceted, RL), 0.93 (RM-Bench)
- beta floor =
0 (Multifaceted, RL), 0.03 (RM-Bench)
assumptions (5)
- domain assumption Expert demonstrations πE are the entropy-regularized optimal policy for the unknown true reward r* (Eq. 13)
- domain assumption The base LLM bπ is the exact global maximizer of the next-token prediction likelihood (Proposition 1, proof in B.1)
- ad hoc to paper Reinforcement learning can be solved exactly, πRL = argmax V^π_br (Theorem 2, Section 4.2)
- domain assumption The environment is a deterministic token-level MDP with fixed horizon H and reward in [0,1] (Section 2.1)
- standard math Entropy-regularized MDP theory: softmax policies and inverse soft Bellman operator from Geist et al. 2019 and Garg et al. 2021 (Eq. 6-7)
Cite this review
Pith. "Pith review of Generalist Reward Models: Found Inside Large Language Models." pith.science (2026). https://pith.science/paper/XKJ7LBJL
@misc{pith2026250623235,
author = {Pith},
title = {Pith review of: Generalist Reward Models: Found Inside Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/XKJ7LBJL}},
note = {Machine review of arXiv:2506.23235}
}
read the original abstract
The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 4 Pith papers
-
CAST: Game Solvers as Turn-Level Teachers for LLM Agents
CAST converts a game solver's per-action cost-to-go changes into turn-level RL credits for LLM agents and reports gains over outcome-only RLVR on three games plus zero-shot transfer to ALFWorld and WebShop.
-
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
Learning prefix-conditioned success probabilities from outcome labels yields more reliable process rewards than sequence log-ratio PRMs and improves RL when used for candidate-token TD updates.
-
Stabilizing Policy Optimization via Logits Convexity
LCO replaces PPO-style policy gradients with regression toward the advantage-derived optimal logits/policy, restoring logits-level convexity and yielding more stable LLM RL training.
-
Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.
Reference graph
Works this paper leans on
-
[1]
**Adherence to Instruction:** The answer’s tone, style, and content must perfectly match the provided instruction
-
[2]
**Relevance to Query:** The answer must directly and comprehensively address the user’s query without any irrelevant information
-
[3]
**Factuality and Helpfulness:** The information in the answer should be accurate and useful. Please carefully review the following materials. ### Instruction {instruction} ### Query {query} ### Response {response} Based on your evaluation, is the answer a good response? Answer with only “YES” or “NO”. Figure 3: Prompt template of Generative Verifier. Syst...
-
[4]
**Adherence to Instruction:** Does the answer’s tone, style, and content align with the specified instruction?
-
[6]
**Factuality and Helpfulness:** Is the information accurate and useful? Based on these criteria, please review the following: ### Instruction {instruction} ### Query {query} ### Response 1 {response1} ### Response 2 {response2} **Your Task:** Is ‘Response 2’ better than ‘Response 1’? **Rules for Your Response:**
-
[7]
Respond with only “YES” or “NO”. Do not include explanations or any other text
-
[8]
“YES” means ‘Response 2’ is clearly better than ‘Response 1’
-
[9]
Figure 4: Prompt template of GenRM-Pairwise
“NO” means ‘Response 2’ is not better (it is either worse or of equal quality) than ‘Response 1’. Figure 4: Prompt template of GenRM-Pairwise. 19 Generalist Reward Models: Found Inside Large Language Models System: You are an AI evaluator. Your role is to assess AI-generated text for its quality and adherence to instructions. User: You need to evaluate th...
Show all 16 references
-
[10]
**Adherence to Instruction:** Does the answer’s tone, style, and content match the instruction?
-
[11]
**Relevance to Query:** Does the answer directly and comprehensively address the user’s query?
-
[12]
**Scoring Guide:** - **1-2:** Very Poor
**Factuality and Helpfulness:** Is the information accurate and useful? Please review the following: ### Instruction {instruction} ### Query {query} ### Response {response} Based on your assessment, provide a single integer score from 1 to 10. **Scoring Guide:** - **1-2:** Ver...
-
[13]
By Vieta’s formulas, we know: r1 + r2 + r3 = −a, r1r2 + r2r3 + r3r1 = a, r1r2r3 = −1
First, let’s denote the roots of the polynomial by r1, r2, r3. By Vieta’s formulas, we know: r1 + r2 + r3 = −a, r1r2 + r2r3 + r3r1 = a, r1r2r3 = −1
-
[14]
For a cubic polynomial x3 + px2 + qx + r = 0, the discriminant ∆ is given by: ∆ = 18abcd − 4b3d + b2c2 − 4ac3 − 27a2d2, where a = 1, b = a, c = a, and d = 1
To ensure that all roots are real, we need to analyze the polynomial and its discriminant. For a cubic polynomial x3 + px2 + qx + r = 0, the discriminant ∆ is given by: ∆ = 18abcd − 4b3d + b2c2 − 4ac3 − 27a2d2, where a = 1, b = a, c = a, and d = 1. Substituting these values in...
-
[15]
So, we need to solve the inequality: a4 − 8a3 + 18a2 − 27 ≥ 0
The discriminant ∆ must be non-negative for the polynomial to have three real roots. So, we need to solve the inequality: a4 − 8a3 + 18a2 − 27 ≥ 0
-
[16]
We can use numerical methods to find the roots of this polynomial
To find the smallest possible value of a, we will solve the equation a4 − 8a3 + 18a2 − 27 = 0. We can use numerical methods to find the roots of this polynomial. Let’s use Python to find the roots and determine the smallest positive real root. “‘python import numpy as np # Def...
-
[17]
a − 1 ≥ 2, which simplifies to a ≥ 3. 2. a − 1 ≤ −2, which simplifies to a ≤ −1. Since a is a positive real number, we discard the second case and keep the first case. Therefore, the smallest possible value of a is 3. Let’s verify this by checking if the polynomial x3 + 3x2 + ...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.