Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

ShiQ: Bringing back Bellman to LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM's logits can be trained by a Bellman residual loss, called ShiQ, to become Q-values whose softmax is the optimal KL-regularized policy, with no extra networks and no paired preference data.

desk verdict Genuinely new off-policy, token-level Q-learning loss with sound theory; deserves peer review, but the empirical claims need more rigor. read the letter →

arxiv 2505.11081 v1 pith:MR6TTSE6 submitted 2025-05-16 cs.LG

Pierre Clavier , Nathan Grinsztajn , Raphael Avalos , Yannis Flet-Berliac , Irem Ergun , Omar D. Domingues , Eugene Tarassov , Olivier Pietquin
show 3 more authors
Pierre H. Richemond Florian Strub Matthieu Geist
This is my paper · ORCID
classification cs.LG
keywords Q-learninglargelanguagemodelsBellmanequationsoff-policyreinforcementlearningKLregularizationtoken-wiselossmulti-turnRLrewardshaping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether Q-learning, rather than policy-gradient or direct-preference optimization, can be the workhorse for RL fine-tuning of LLMs. It answers yes by deriving a chain of Bellman consistency equations adapted to autoregressive language models and packaging the final one as a loss called ShiQ. Under the population-level support condition $\operatorname{supp}(D)=\operatorname{supp}(\rho\pi_{\mathrm{ref}})$, the unique minimizer of the ShiQ loss yields a logit function whose softmax policy is exactly the maximizer of the KL-regularized reward objective $J(\pi)=\mathbb{E}_{x\sim\rho}\mathbb{E}_{y\sim\pi(\cdot|x)}[R(x,y)-\beta\,\mathrm{KL}(\pi(\cdot|x)\|\pi_{\mathrm{ref}}(\cdot|x))]$. Because the loss is off-policy and token-level, training can reuse static datasets or replay buffers, and inference remains a plain softmax over the model's logits. If the paper is right, offline and multi-turn RL fine-tuning no longer needs paired preference data, importance sampling, or extra value networks.

What carries the argument

The central object is a regularized Bellman consistency equation rewritten as a relation between logits and log-partitions, shifted by the reference model. The paper defines the token-level log-partition $v_\ell(s)=\ln\sum_a\exp\ell(s,a)$ and proves that the policy $\pi_\ell(a|s)\propto\exp\ell(s,a)$ remains the unique optimal policy through a sequence of four transformations: the soft Bellman equation for a Q-function (Theorem 1), a reparameterization that lets the logits themselves be sampled at inference (Theorem 2), a potential-based reward shaping that makes the reference logits a natural initialization (Theorem 3), and a telescoping argument yielding the multi-step identity $\beta(v_\ell(s_t)-v_{\mathrm{ref}}(s_t))=\sum_{k=t}^{T}\gamma^{k-t}(r(s_k,a_k)-\beta\ln\frac{\pi_\ell(a_k|s_k)}{\pi_{\mathrm{ref}}(a_k|s_k)})$ (Theorem 4). ShiQ is the squared residual of this identity, summed over every token in the completion. This machinery does the work of preserving the optimal policy across each modification while making the loss off-policy, token-wise, and free of additional networks.

What would settle it

Run a tabular two-action bandit with a uniform reference policy, rewards 10 and 0 for the two completions, and a dataset containing only the zero-reward completion. The paper's theorem does not apply because the support condition fails, so record whether ShiQ's learned policy stays near the reference or shifts toward the missing high-reward arm; the magnitude of that shift quantifies the gap in the guarantee. To test the theorem itself, a tabular full-support experiment should reproduce zero residual and $\pi_{\ell^\star}=\pi^\star$ exactly, and any nonzero residual at the unique minimizer would contradict the corollary.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM's logits can be treated as Q-values, provided the Bellman equation is rewritten to respect three LLM-specific constraints: inference must sample directly from the trained logits; the reference model's logits should be a natural initialization, meaning no gradient update when the reward is identically zero; and sparse terminal rewards must propagate across many tokens. Theorems 1 through 4 show that each transformation, reparameterizing the Q-function into a softmax logit, applying potential-based reward shaping with the reference log-partition as potential, and telescoping the one-step equation into a multi-step consistency identity, preserves the unique optimal policy. The resulting loss is $L_{\mathrm{ShiQ}}(\ell)=\mathbb{E}_{x,y\in D}\sum_{t=1}^{|y|}\bigl(R(x,y)-\beta\ln\frac{\pi_\ell(y_{\ge t}|x,y_{<t})}{\pi_{\mathrm{ref}}(y_{\ge t}|x,y_{<t})}-\beta(v_\ell(x\oplus y_{<t})-v_{\mathrm{ref}}(x\oplus y_{<t}))\bigr)^2$. The direct corollary of Theorem 4 is that under $\operatorname{supp}(D)=\operatorname{supp}(\rho\pi_{\mathrm{ref}})$, the unique minimizer $\ell^\star$ satisfies $L_{\mathrm{ShiQ}}(\ell^\star)=0$ and $\pi_{\ell^\star}=\pi^\star$. The experiments then claim that on the multi-turn function-calling benchmark BFCL-V3, ShiQ outperforms the multi-turn DPO and CoPG baselines by exploiting full reward-position information, while on single-turn Anthropic-Harmless and UltraFeedback it matches CoPG using less information.

Load-bearing premise

The result collapses if the training data does not cover every prompt-completion pair the reference policy can generate, because the unique-minimizer-equals-optimal-policy proof relies on $\operatorname{supp}(D)=\operatorname{supp}(\rho\pi_{\mathrm{ref}})$, while real training uses finite data and a neural network outside that guarantee.

Editorial extensions

If this is right

  • If the population-level result is right, offline RL fine-tuning of an LLM can be done with a single policy network: the trained logits are already Q-values, and inference softmax needs no reference model or extra value head.
  • Because the loss is token-level and off-policy, any fixed dataset of prompt-completion-reward triples, including replay buffers from prior generations, can be reused without fresh on-policy rollouts or importance sampling.
  • The multi-step form propagates sparse terminal rewards to earlier tokens within a single update, which is what allows ShiQ to exploit intermediate reward information in multi-turn function calling.
  • On BFCL-V3 the paper reports that ShiQ outperforms multi-turn DPO and CoPG on the verifiable reward, and on the single-turn Anthropic-Harmless and UltraFeedback datasets ShiQ matches CoPG while using only unpaired trajectories.
  • The sequence-level ablation ShiQ/tk, which the paper shows is competitive when rewards arrive at turn boundaries, offers a cheaper alternative to learned value networks in off-policy LLM alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's support assumption is a natural place to extend: a pessimistic or data-coverage penalty added to Eq. (13) would give ShiQ a principled regime when the dataset under-covers the reference model, and the bandit toy from Sec. 3.1 could be rerun with increasingly skewed arm datasets to test such an extension.
  • Because the loss consumes unpaired prompt-completion-reward triples, any offline corpus with scalar quality scores, such as verifier accuracy for code, unit-test pass rates, or retriever scores, becomes usable training signal without constructing preference pairs; this is an application the paper motivates but does not explore.
  • The telescoping trick in Theorem 4 is a general consequence of KL regularization and exact token log-probabilities, so a version of ShiQ could plausibly apply to other structured sequence models; whether it remains stable under model misspecification is an open question the paper leaves implicit.
  • The paper's own ablation suggests ShiQ/tk is competitive when rewards arrive at turn boundaries, so a practical system could switch between the token-level and sequence-level losses depending on whether dense rewards are available.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces ShiQ (Shifted-Q), a loss function for fine-tuning LLMs that interprets logits as Q-values in a KL-regularized token-level MDP. The authors derive a sequence of Bellman consistency equations (Theorems 1–4) and a final loss (Eq. 13) that is off-policy, token-level, and requires only a single trajectory per prompt, together with an initialization trick based on potential-based reward shaping. They prove that, under a full-support dataset condition, the population loss has a unique minimizer whose softmax policy is the optimal KL-regularized policy, and they present a sequence-level variant ShiQ/tk. Experiments on a synthetic bandit, a grid-world MDP, HH-RLHF, UltraFeedback, and BFCL-V3 compare ShiQ with DPO, CoPG, and DRO.

Significance. The derivation is careful and self-contained: Appendix C re-proves regularized MDP, reparameterization, reward-shaping, and path-consistency results in the LLM setting, and the final loss is a genuinely new combination that avoids extra value networks and importance sampling. The multi-step and token-level formulation is a useful contribution, and ShiQ/tk offers a theoretically motivated alternative to DRO's separate value network. The main value, if the results hold, is an off-policy Bellman-style objective for LLMs that can exploit token-level or multi-turn rewards without paired preference data. The principal weaknesses are the gap between the population-level optimality theorem and finite neural-network training, and the absence of statistical evidence for the headline multi-turn empirical claim.

major comments (2)
  1. [Sec. 2.5, Eq. (13); corollary after Thm. 4] The uniqueness and optimality result does not transfer to the finite datasets used in the experiments. For an observed transition (s_t, a_t), the summand in Eq. (13) reduces to R(x,y) - beta (ell(s_t,a_t) - ell_ref(s_t,a_t)) - beta sum_{k>t} ln(pi_ell(a_k|s_k)/pi_ref(a_k|s_k)), because the -beta(v_ell(s_t) - v_ref(s_t)) term cancels the v_ell(s_t) - v_ref(s_t) contribution inside ln(pi_ell(y_{>=t}|s_t)/pi_ref(y_{>=t}|s_t)). The loss therefore contains no term involving the logits of unobserved actions at s_t. Since every finite dataset over a large vocabulary leaves the vast majority of next-token edges unobserved, the loss has massively non-unique minimizers; moreover, raising the logits of unobserved siblings at an observed state leaves the residual unchanged while diluting the softmax probability of the observed token. The support condition supp(D) = supp(rho pi_ref) cannot be satisfied by a finite dataset, so the claimed 'unique minimizer implies optimal policy' result is a population-level statement that does not justify the empirical results. The paper should either provide a finite-sample or identifiability analysis, add and analyze explicit regularization or coverage assumptions, or clearly scope the theoretical claims and explain why the non-identifiability does not invalidate the reported training behavior.
  2. [Sec. 3.4 and Sec. D.5] The multi-turn empirical claim is not yet robustly established. Appendix D.5 states that models were trained 'for one epoch while sweeping over the parameter beta in the set {0.001, 0.01, 0.1} and picking the best beta,' which selects the regularization temperature using the evaluation set (BFCL-V3); the paper does not report the chosen value per method, multiple seeds, or confidence intervals. Without this information, the headline statement that ShiQ 'outperforms these baselines' in Fig. 4 could reflect favorable beta selection rather than a systematic advantage. Please report the selection protocol, the selected beta values, and error bars or significance tests.
minor comments (5)
  1. [Sec. 2.2, after Thm. 2] The sentence 'the unique optimizer g* of Ltry1(g)' should refer to Ltry2(g), not Ltry1(g).
  2. [Sec. 2.4, Eq. (9) and Appx. C Eq. (25)] In the statement of Theorem 4, the summand writes r(s_t,a_t) and pi_ell(a_t|s_t) although the sum is over k = t..T; these should be r(s_k,a_k) and pi_ell(a_k|s_k), as used correctly in the proof.
  3. [Appx. C, proof of Thm. 4] The sentence 'Hence, f satisfies the Bellman equation of Thm. 1, and therefore f = q' should refer to Thm. 3 and conclude f = ell, not f = q.
  4. [Sec. 2.3, Sec. 3.3, Sec. 4] Several typos and broken sentences need correction: 'we first that it is not the case' in Sec. 2.3, 'ShiQ is capable of perform simularly' in Sec. 3.3, and the final paragraph of Sec. 4 has a grammatically incomplete clause involving 'represents or uses ShiQ for distillation'.
  5. [Sec. 3.1 and Appx. D.1] The dataset size is written '104 pairs' in Sec. 3.1 but '10^4 pairs' in Appx. D.1; please make the notation consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ShiQ derivation is re-proven in Appendix C, and the central minimizer-to-optimal-policy claim is derived rather than fitted or assumed.

full rationale

The paper's theoretical chain (Theorems 1-4) is not merely cited from prior work: full proofs are given in Appendix C, including the backward-induction argument for Theorem 1, the change-of-variable argument for Theorem 2, the reward-shaping argument for Theorem 3, and the telescoping/uniqueness argument for Theorem 4. The central claim that the unique minimizer of LShiQ has softmax equal to the optimal policy is a mathematical consequence of the Bellman equations and the support assumption supp(D) = supp(ρπref), not an equality inserted by definition. The loss is constructed so that the Bellman fixed point attains zero loss, and the proofs then show that any zero-loss point must satisfy the Bellman equation; this is a consistency argument, not a tautology. Self-citations to the authors' own prior work (regularized MDPs, Munchausen reparameterization, DRO, CoPG) are used for context and inspiration, but the load-bearing results are re-derived inside the paper, so the self-citations are not load-bearing. The finite-dataset identifiability gap and the reliance on a reliable reward model are genuine limitations, but they are correctness/transfer concerns rather than circularity: the paper does not fit a parameter and then rename it a prediction. No step in the derivation reduces to its own input by construction.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The derivation chains four published building blocks (regularized MDPs [15], Munchausen Q-reparametrization [43], potential-based shaping [29, 45], PCL [28]) into a new loss, with the theorems re-proven in Appx. C rather than assumed. The price of the combination is the support condition supp(D)=supp(ρπref) and reliance on a reliable reward signal; the main fitted quantity is β, tuned per method on the evaluation data. No new entities are postulated; the 'shifted Q' is a reparametrization of the logits.

free parameters (3)
  • β (KL regularization temperature, appears in all losses) = Per-method grid search: HH {0.001-1}; UF {0.0075-0.03} after narrowing; BFCL-V3 {0.001-0.1}
    The central hyperparameter of the loss; it is grid-searched per method and per dataset on the same evaluation prompts used for the reported figures (Appx. D.3-D.5), so it is a fitted quantity that affects every empirical comparison.
  • Learning rate = 1e-6 for all LLM runs
    Chosen once after checking insensitivity (Appx. D.3); not a major influence on the results.
  • γ (discount) and grid-world step penalty = γ=1 for LLM experiments; γ=0.99 and step penalty 0.05 in the grid world
    The main text motivates γ=1 for the finite-horizon LLM setting, but the toy MDP uses γ=0.99 with a per-step penalty (Appx. D.2); these are hand-set toy hyperparameters that shape the Fig. 2 claims.
assumptions (6)
  • domain assumption LLM generation is a finite-horizon MDP: state = (prompt, tokens so far), action = next token, deterministic transition by concatenation, discount γ(st,at)=0 at eos or Tmax.
    Sec. 2.1. This formalization is the substrate for all four theorems; it treats prompt sampling and token generation as a fully observed Markov process, ignoring any non-Markovian structure beyond the prefix.
  • domain assumption πref has full support over actions, and the data support satisfies supp(D) = supp(ρπref).
    Invoked in every corollary (Secs. 2.1-2.5) and in the proof of Thm. 1 ('we'll assume πref to have full support'). Any finite dataset violates it, so the 'unique minimizer implies optimal policy' results hold only for the population loss.
  • standard math Backward induction and the Legendre-Fenchel transform characterize the unique optimal policy of the KL-regularized finite-horizon objective.
    Used throughout the proof of Thm. 1 (Appx. C); the finite-horizon setting avoids contraction arguments and permits γ=1.
  • standard math Potential-based reward shaping preserves the optimal policy (Ng et al. 1999) and is equivalent to Q-function reinitialization (Wiewiora 2003).
    This is the basis of the Thm. 3 initialization trick, cited as [29] and [45].
  • domain assumption A reliable token-wise or sequence-level reward function r is available from a reward model or verifier.
    Sec. 2.1 assumes r; Sec. 4 concedes 'ShiQ assumes access to a reliable reward model, a condition rarely met in practice.' All experiments inherit this.
  • standard math Multi-step path-consistency equations hold without importance sampling thanks to the KL-regularized structure (Nachum et al. 2017, PCL).
    Basis of Thm. 4 and the LShiQ loss; the telescoping argument in Appx. C depends on the log-ratio terms exactly telescoping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ShiQ: Bringing back Bellman to LLMs." pith.science (2026). https://pith.science/paper/MR6TTSE6

@misc{pith2026250511081,
  author       = {Pith},
  title        = {Pith review of: ShiQ: Bringing back Bellman to LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MR6TTSE6}},
  note         = {Machine review of arXiv:2505.11081}
}
read the original abstract

The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM, seen as an initial policy. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness comes from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLMs. However, naively applying a Q-learning-style update to the model's logits is ineffective due to the specificity of LLMs. Our core contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we carefully adapt insights from the RL literature to account for LLM-specific characteristics, ensuring that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally, we evaluate ShiQ on both synthetic data and real-world benchmarks, e.g., UltraFeedback and BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings

Figures

Figures reproduced from arXiv: 2505.11081 by the authors.

Figure 1
Figure 1. Offline 3-arms bandit setting To empirically evaluate our method, we consider a syn￾thetic 3-armed bandit problem with associated rewards R = (2.5, 2, 1), arms sampled from two distributions: µ1 = (0.1, 0.2, 0.7) and µ2 = (0.05, 0.05, 0.9). Using these distributions, we construct a dataset comprising 104 pairs of rewarded arms. We define the reference pol￾icy as uniform: πref(y) = 1 3 for all y ∈ {1, 2, 3}. The opti… view at source ↗
Figure 2
Figure 2. Comparison of Regret and Pareto front using fine-grained and final rewards. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Reward optimisation and Pareto comparison for HH dataset [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Reward optimisation and Pareto comparison for BFCL-v3 dataset [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Regret and Pareto comparison with final reward on HH dataset [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Test loss on HH dataset 26 [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Regret and Pareto with UF dataset D.5 Experimental details for BFCL-V3 Note that the version BFCL-V3 used is the one before modification in 05/01/2025. Regarding BFCL training, models were trained for one epoch while sweeping over the parameter β in the set {0.001, 0.0…
Figure 8
Figure 8. Figure 8: Regret and Pareto comparison with multi-step rewards on BFLC-v3 dataset 28 [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

    cs.LG 2025-12 unverdicted novelty 6.0 of 10

    Under the chain rule of probability, autoregressive models and energy-based models are in exact bijection in function space, making the global optimum of teacher forcing equivalent to an energy-based model with implic...

Reference graph

Works this paper leans on

58 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    A learning algorithm for boltzmann machines.Cognitive science, 9(1):147–169, 1985

    David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines.Cognitive science, 9(1):147–169, 1985

    1985

  2. [2]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740, 2024

    Arash Ahmadian, Chris Cremer, Matthias Gall´ e, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet ¨Ust¨ un, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  3. [3]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. 10

    work page 2024

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [5]

    Residual algorithms: Reinforcement learning with function approximation

    Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. InMachine learning proceedings 1995, pages 30–37. Elsevier, 1995

    1995

  6. [6]

    Linear least-squares algorithms for temporal difference learning

    Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33–57, 1996

    work page 1996

  7. [7]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  8. [8]

    Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698, 2025

    Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Rapha¨ el Avalos, et al. Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698, 2025

    arXiv 2025

Show all 58 references
  1. [9]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023

    2023

  2. [10]

    Off-policy actor-critic

    Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. InInternational Conference on Machine Learning, 2012

    2012

  3. [11]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  4. [12]

    Contrastive policy gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion

    Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ah- madian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive policy gradient: Aligning LLMs on sequence-level scores in a supervised-friend...

    2024

  5. [13]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  6. [14]

    Is the bellman residual a bad proxy?Advances in Neural Information Processing Systems, 30, 2017

    Matthieu Geist, Bilal Piot, and Olivier Pietquin. Is the bellman residual a bad proxy?Advances in Neural Information Processing Systems, 30, 2017

    2017

  7. [15]

    A theory of regularized markov decision processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. InInternational Conference on Machine Learning, pages 2160–2169. PMLR, 2019

    2019

  8. [16]

    Averaging log-likelihoods in direct alignment.arXiv preprint arXiv:2406.19188, 2024

    Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, et al. Averaging log-likelihoods in direct alignment.arXiv preprint arXiv:2406.19188, 2024

    2024 arXiv

  9. [17]

    Efficient (soft) q-learning for text generation with limited good data

    Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Efficient (soft) q-learning for text generation with limited good data. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 6969–6991, 2022

    2022

  10. [18]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. InProceedings of the AAAI conference on artificial intelligence, ...

    2018

  11. [19]

    The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751, 2019. 11

    1904 arXiv

  12. [20]

    Q-sft: Q-learning for language models via supervised fine-tuning.arXiv preprint arXiv:2411.05193, 2024

    Joey Hong, Anca Dragan, and Sergey Levine. Q-sft: Q-learning for language models via supervised fine-tuning.arXiv preprint arXiv:2411.05193, 2024

    2024 arXiv

  13. [21]

    Enhancing multi-step reasoning abilities of language models through direct q-function optimization.arXiv preprint arXiv:2410.09302, 2024

    Kaixuan Ji, Guanlin Liu, Ning Dai, Qingping Yang, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing multi-step reasoning abilities of language models through direct q-function optimization.arXiv preprint arXiv:2410.09302, 2024

    arXiv

  14. [22]

    Buy 4 reinforce samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction (ICLR Workshop), 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction (ICLR Workshop), 2019

    2019

  15. [23]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=68n2s9ZJWF8

    2022

  16. [24]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning.Advances in Neural Information Processing Systems, 35:21314–21328, 2022

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning.Advances in Neural Information Processing Systems, 35:21314–21328, 2022

    2022

  17. [25]

    Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  18. [26]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  19. [27]

    Safe and efficient off-policy reinforcement learning.Advances in neural information processing systems, 29, 2016

    R´ emi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning.Advances in neural information processing systems, 29, 2016

    2016

  20. [28]

    Bridging the gap between value and policy based reinforcement learning.Advances in neural information processing systems, 30, 2017

    Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning.Advances in neural information processing systems, 30, 2017

    2017

  21. [29]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InICML, volume 99, pages 278–287, 1999

    1999

  22. [30]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  23. [31]

    Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems, 37:126544–126565, 2024

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems, 37:126544–126565, 2024

    2024

  24. [32]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2023

    2023

  25. [33]

    From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    2024 arXiv

  26. [34]

    Offline regularised reinforcement learning for large language models alignment.arXiv preprint arXiv:2405.19107, 2024

    Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, et al. Offline regularised reinforcement learning for large language models alignment.arXiv...

    2024 arXiv

  27. [35]

    Factually consistent summarization via reinforcement learning with textual entailment feedback

    Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Leonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos Garea, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietqu...

    2023

  28. [36]

    Approx- imate modified policy iteration and its application to the game of tetris.J

    Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approx- imate modified policy iteration and its application to the game of tetris.J. Mach. Learn. Res., 16(49): 1629–1676, 2015

    2015

  29. [37]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  30. [38]

    Multi-turn reinforcement learning from preference human feedback.arXiv preprint arXiv:2405.14655, 2024

    Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi-turn reinforcement learning from preference human feedback.arXiv preprint arXiv:2405.14655, 2024

    2024 arXiv

  31. [39]

    Offline RL for natural language generation with implicit language q learning

    Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline RL for natural language generation with implicit language q learning. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=aBH_DydEvoH

    2023

  32. [40]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Ruslan Salakhutdinov, Zico...

    2024

  33. [41]

    Rl-finetuning llms from on-and off-policy data with a single algorithm.arXiv preprint arXiv:2503.19612, 2025

    Yunhao Tang, Taco Cohen, David W Zhang, Michal Valko, and R´ emi Munos. Rl-finetuning llms from on-and off-policy data with a single algorithm.arXiv preprint arXiv:2503.19612, 2025

    2025 arXiv

  34. [42]

    Leverage the average: an analysis of kl regularization in reinforcement learning.Advances in Neural Information Processing Systems, 33:12163–12174, 2020

    Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, R´ emi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning.Advances in Neural Information Processing Systems, 33:12163–12174, 2020

    2020

  35. [43]

    Munchausen reinforcement learning.Advances in Neural Information Processing Systems, 33:4235–4246, 2020

    Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning.Advances in Neural Information Processing Systems, 33:4235–4246, 2020

    2020

  36. [44]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. InInternational conference on machine learning, pages 1995–2003. PMLR, 2016

    1995

  37. [45]

    Potential-based shaping and q-value initialization are equivalent.Journal of Artificial Intelligence Research, 19:205–208, 2003

    Eric Wiewiora. Potential-based shaping and q-value initialization are equivalent.Journal of Artificial Intelligence Research, 19:205–208, 2003

    2003

  38. [46]

    Function optimization using connectionist reinforcement learning algorithms.Connection Science, 3(3):241–268, 1991

    Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms.Connection Science, 3(3):241–268, 1991

    1991

  39. [47]

    Building math agents with multi-turn iterative preference learning.arXiv preprint arXiv:2409.02392, 2024

    Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning.arXiv preprint arXiv:2409.02392, 2024

    2024 arXiv

  40. [48]

    InThe Twelfth International Conference on Learning Representations, 2024

    Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang.$ \mathcal{B}$-coder: Value-based deep reinforcement learning for program synthesis. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=fLf589bx1f. 13

    2024

  41. [49]

    Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:2406.16793, 2024

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:2406.16793, 2024

    2024 arXiv

  42. [50]

    Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  43. [51]

    Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  44. [52]

    PhD thesis, Carnegie Mellon University, 2010

    Brian D Ziebart.Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, Carnegie Mellon University, 2010. 14 A Presentation of other variation ofShiQ ShiQ/init:We can skip the reward shaping used in Thm. 3, that aims at making the refere...

    2010

  45. [53]

    Initialization trick

  46. [54]

    Initialization trick

    Going Multi-Step 2. Initialization trick

  47. [55]

    This contrasts with other RL-finetuning approaches, such as DRO [ 34] or CoPG [ 12], that involve a square term per sequence of the batch

    Going Multi-StepLShiQ\ms(l) Ltry3′(l) = LShiQ\init(l) ShiQ/tk :The ShiQ loss is a token-level loss, in the sense that it involves a square term for each token of the batch. This contrasts with other RL-finetuning approaches, such as DRO [ 34] or CoPG [ 12], that involve a squa...

    2025

  48. [56]

    These tools use machine learning algorithms to generate realistic voices and faces

    Choose a Deepfake Software: There are several deepfake software tools available, such as DeepVoice, Deepfake, and FaceSwap. These tools use machine learning algorithms to generate realistic voices and faces

  49. [57]

    The more reference audio you have, the better the result

    Collect Reference Audio: You’ll need a high-quality audio recording of your own voice or the person whose voice you want to mimic. The more reference audio you have, the better the result

  50. [58]

    This process may take some time, depending on the complexity of the task and the 35

    Train the Model: Upload the reference audio to the deepfake software and train the model. This process may take some time, depending on the complexity of the task and the 35

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.