{"id":"41963c95-ab6a-4fe0-94f8-874c27bf2860","arxiv_id":"2505.11081","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ShiQ is a Bellman-derived loss that makes LLM logits behave as Q-values, enabling off-policy token-level and multi-turn reinforcement-learning fine-tuning with the softmax over logits as the policy.","lead":"The paper derives a Q-learning-style loss, ShiQ, that lets a language model's token probabilities act as estimates of future reward, enabling training from stored offline data instead of fresh generations. It reports that ShiQ matches or beats DPO, CoPG, and DRO on single-turn and multi-turn benchmarks, at the cost of assuming a reliable reward model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finite-data identifiability: the ShiQ loss constrains only observed next-token edges, so the population-level uniqueness result does not transfer to the finite datasets used in experiments.","rationale":"I read the paper as making two contributions: a population-level Bellman-derived loss with an optimality guarantee, and an empirical demonstration that ShiQ works on LLM benchmarks. The derivations in Secs. 2.1-2.5 and Appx. C are internally consistent: I checked the reward-shaping argument in Thm. 3, the telescoping argument in Thm. 4, and the cancellation that turns the Bellman residual into the ShiQ loss; no circular step or algebraic error appears. The theoretical result is sound under the stated full-support assumption. The empirical protocol, however, rests on transferring that population result to finite-data neural training, and the paper gives no justification for doing so. The strongest form of this gap is algebraic: because vℓ(prefix) cancels in Eq. (13), the residual at a state-action edge depends only on the logit of the observed next token and on future suffixes, not on the logits of alternative tokens at that state. With finite data, most edges are unobserved, the empirical loss has many zero-loss minimizers, and the recovered policy is not identified. This is not a generic 'finite-sample noise' worry; it is a structural property of the loss that makes the uniqueness theorem an exact artifact of the support assumption. The conclusion's admission that a reliable reward model is rarely available (Sec. 4) is a separate limitation, but it is orthogonal to this identifiability issue. My read does not move the reader's verdict: the theoretical contribution remains valuable, while the empirical claims remain conditional pending a finite-sample or coverage analysis. The proposed bandit and two-token checks would settle the concern directly and are cheap to run.","tokens_in":27662,"tokens_out":24764,"duration_ms":264850,"concrete_test":"Run the 3-armed bandit experiment of Sec. 3.1 with a dataset that omits one arm (e.g., only arms 1 and 2 are sampled) while keeping all other settings identical. Minimize LShiQ over unrestricted tabular logits initialized at ℓref; the residual equations fix ℓ for the observed arms but leave the third logit free. Sweeping that logit over a range should yield a continuum of minimizers with different policies πℓ, confirming that uniqueness is an artifact of supp(D)=supp(ρπref). Then repeat on a two-token MDP with a single observed completion, enumerating zero-residual logit assignments to show at least two induce different policies. If both checks pass, the finite-sample identifiability failure is confirmed and the practical claims require an explicit coverage or regularization analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central optimality guarantee (Sec. 2.5, corollary of Thm. 4) requires supp(D) = supp(ρπref), meaning the dataset must assign positive probability to every admissible state-action edge. Any finite dataset fails this condition, and the failure is particularly acute because of the algebraic structure of LShiQ. In Eq. (13), the term -β(vℓ(x⊕y<t) - vref(x⊕y<t)) exactly cancels the vℓ(x⊕y<t) contribution inside the suffix log-likelihood ln πℓ(y≥t|x,y<t). The residual at an observed transition (s,a) therefore reduces to R(x,y) - β(ℓ(s,a) - ℓref(s,a)) - β ln(πℓ(future|s⊕a)/πref(future|s⊕a)), which contains no dependence on the logits of unobserved actions at s. Consequently, with a finite dataset, all unobserved next-token edges are unconstrained by the loss, and the empirical minimizer is massively non-unique. Even the probability of an observed token is not pinned down, because raising the logits of its siblings leaves the residual unchanged while diluting its softmax probability. The paper provides no finite-sample or function-approximation analysis, and the LLM experiments train on datasets that cover only a tiny fraction of the vocabulary/state space, so the claimed 'minimizer implies optimal policy' result does not transfer to the reported results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ShiQ (Shifted-Q), a loss function for fine-tuning LLMs that interprets logits as Q-values in a KL-regularized token-level MDP. The authors derive a sequence of Bellman consistency equations (Theorems 1–4) and a final loss (Eq. 13) that is off-policy, token-level, and requires only a single trajectory per prompt, together with an initialization trick based on potential-based reward shaping. They prove that, under a full-support dataset condition, the population loss has a unique minimizer whose softmax policy is the optimal KL-regularized policy, and they present a sequence-level variant ShiQ/tk. Experiments on a synthetic bandit, a grid-world MDP, HH-RLHF, UltraFeedback, and BFCL-V3 compare ShiQ with DPO, CoPG, and DRO.","tokens_in":27895,"tokens_out":7941,"duration_ms":76505,"significance":"The derivation is careful and self-contained: Appendix C re-proves regularized MDP, reparameterization, reward-shaping, and path-consistency results in the LLM setting, and the final loss is a genuinely new combination that avoids extra value networks and importance sampling. The multi-step and token-level formulation is a useful contribution, and ShiQ/tk offers a theoretically motivated alternative to DRO's separate value network. The main value, if the results hold, is an off-policy Bellman-style objective for LLMs that can exploit token-level or multi-turn rewards without paired preference data. The principal weaknesses are the gap between the population-level optimality theorem and finite neural-network training, and the absence of statistical evidence for the headline multi-turn empirical claim.","major_comments":[{"comment":"The uniqueness and optimality result does not transfer to the finite datasets used in the experiments. For an observed transition (s_t, a_t), the summand in Eq. (13) reduces to R(x,y) - beta (ell(s_t,a_t) - ell_ref(s_t,a_t)) - beta sum_{k>t} ln(pi_ell(a_k|s_k)/pi_ref(a_k|s_k)), because the -beta(v_ell(s_t) - v_ref(s_t)) term cancels the v_ell(s_t) - v_ref(s_t) contribution inside ln(pi_ell(y_{>=t}|s_t)/pi_ref(y_{>=t}|s_t)). The loss therefore contains no term involving the logits of unobserved actions at s_t. Since every finite dataset over a large vocabulary leaves the vast majority of next-token edges unobserved, the loss has massively non-unique minimizers; moreover, raising the logits of unobserved siblings at an observed state leaves the residual unchanged while diluting the softmax probability of the observed token. The support condition supp(D) = supp(rho pi_ref) cannot be satisfied by a finite dataset, so the claimed 'unique minimizer implies optimal policy' result is a population-level statement that does not justify the empirical results. The paper should either provide a finite-sample or identifiability analysis, add and analyze explicit regularization or coverage assumptions, or clearly scope the theoretical claims and explain why the non-identifiability does not invalidate the reported training behavior.","section":"Sec. 2.5, Eq. (13); corollary after Thm. 4"},{"comment":"The multi-turn empirical claim is not yet robustly established. Appendix D.5 states that models were trained 'for one epoch while sweeping over the parameter beta in the set {0.001, 0.01, 0.1} and picking the best beta,' which selects the regularization temperature using the evaluation set (BFCL-V3); the paper does not report the chosen value per method, multiple seeds, or confidence intervals. Without this information, the headline statement that ShiQ 'outperforms these baselines' in Fig. 4 could reflect favorable beta selection rather than a systematic advantage. Please report the selection protocol, the selected beta values, and error bars or significance tests.","section":"Sec. 3.4 and Sec. D.5"}],"minor_comments":[{"comment":"The sentence 'the unique optimizer g* of Ltry1(g)' should refer to Ltry2(g), not Ltry1(g).","section":"Sec. 2.2, after Thm. 2"},{"comment":"In the statement of Theorem 4, the summand writes r(s_t,a_t) and pi_ell(a_t|s_t) although the sum is over k = t..T; these should be r(s_k,a_k) and pi_ell(a_k|s_k), as used correctly in the proof.","section":"Sec. 2.4, Eq. (9) and Appx. C Eq. (25)"},{"comment":"The sentence 'Hence, f satisfies the Bellman equation of Thm. 1, and therefore f = q' should refer to Thm. 3 and conclude f = ell, not f = q.","section":"Appx. C, proof of Thm. 4"},{"comment":"Several typos and broken sentences need correction: 'we first that it is not the case' in Sec. 2.3, 'ShiQ is capable of perform simularly' in Sec. 3.3, and the final paragraph of Sec. 4 has a grammatically incomplete clause involving 'represents or uses ShiQ for distillation'.","section":"Sec. 2.3, Sec. 3.3, Sec. 4"},{"comment":"The dataset size is written '104 pairs' in Sec. 3.1 but '10^4 pairs' in Appx. D.1; please make the notation consistent.","section":"Sec. 3.1 and Appx. D.1"}],"recommendation":"major_revision","confidential_remarks":"The theoretical results are sound under a support assumption that is impossible to satisfy with finite data, and the authors do not address the finite-sample identifiability gap. I would ask them to add a concrete discussion or analysis of this gap, and to strengthen the empirical claims with selection details and error bars. The derivation is a real contribution, so I do not recommend rejection, but the current manuscript overstates the transfer from population theory to the reported experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ShiQ is a genuinely new off-policy, token-level Q-learning loss for LLMs, and the theory behind it is sound. The shifted Bellman equation of Thm. 3 — using reference logits as a zero-residual initialization via reward shaping — is the real contribution, and the PCL-style multi-step extension in Thm. 4 is coherent. The appendix proofs of Thms. 1–5 check out under the stated support condition. The connection to DRO (Thm. 5/Rk. 2) is also a nice insight: ShiQ/tk can replace the extra value network in DRO with a tractable token-level value, which is both cheaper and theoretically cleaner.\n\nThe paper does some things well empirically: the grid-world toy with fine-grained rewards is a clean demonstration that ShiQ exploits intermediate rewards, and the multi-turn BFCL-V3 results are encouraging. The loss itself is simple to implement, which matters for adoption.\n\nThe soft spots are mostly experimental. β is tuned on the evaluation prompts, all reported numbers appear to be single runs without error bars, no code or data is released, and BFCL-V3 is only 200 samples. I would want the β-selection protocol clarified and some confidence intervals before believing specific performance claims. The theory section states the full-support assumption but does not analyze the finite-sample or function-approximation gap. The stress-test note claims unobserved next-token edges are completely unconstrained; that is overstated, because vℓ for a prefix appears in residuals at earlier positions, so unobserved actions are partially constrained through the multi-step structure. The broader caution stands: uniqueness does not transfer to finite data, and there are degenerate cases (e.g., states appearing only as final prefixes) where logits are underdetermined. That is a real limitation, but not a fatal one.\n\nThe citation pattern is honest: four building blocks come from the authors' own prior work, but the theorems are re-proven and the shifted parametrization is new. Self-reliance is justified.\n\nBottom line: this is a solid contribution for people working on off-policy RL fine-tuning of LLMs. It deserves a serious referee, but I would recommend revision: add multiple seeds or error bars, document the β sweep honestly, release code, and add a short discussion of finite-sample identifiability. My own verdict would be conditional acceptance, not rejection.","headline":"Genuinely new off-policy, token-level Q-learning loss with sound theory; deserves peer review, but the empirical claims need more rigor.","tokens_in":28595,"tokens_out":5799,"would_cite":true,"duration_ms":52041,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM's logits can be trained by a Bellman residual loss, called ShiQ, to become Q-values whose softmax is the optimal KL-regularized policy, with no extra networks and no paired preference data.","keywords":["Q-learning","large language models","Bellman equations","off-policy reinforcement learning","KL regularization","token-wise loss","multi-turn RL","reward shaping"],"falsifier":"Run a tabular two-action bandit with a uniform reference policy, rewards 10 and 0 for the two completions, and a dataset containing only the zero-reward completion. The paper's theorem does not apply because the support condition fails, so record whether ShiQ's learned policy stays near the reference or shifts toward the missing high-reward arm; the magnitude of that shift quantifies the gap in the guarantee. To test the theorem itself, a tabular full-support experiment should reproduce zero residual and $\\pi_{\\ell^\\star}=\\pi^\\star$ exactly, and any nonzero residual at the unique minimizer would contradict the corollary.","tokens_in":2082,"feed_emoji":"🎯","tokens_out":2685,"duration_ms":84969,"temperature":0.7,"pith_summary":"The paper asks whether Q-learning, rather than policy-gradient or direct-preference optimization, can be the workhorse for RL fine-tuning of LLMs. It answers yes by deriving a chain of Bellman consistency equations adapted to autoregressive language models and packaging the final one as a loss called ShiQ. Under the population-level support condition $\\operatorname{supp}(D)=\\operatorname{supp}(\\rho\\pi_{\\mathrm{ref}})$, the unique minimizer of the ShiQ loss yields a logit function whose softmax policy is exactly the maximizer of the KL-regularized reward objective $J(\\pi)=\\mathbb{E}_{x\\sim\\rho}\\mathbb{E}_{y\\sim\\pi(\\cdot|x)}[R(x,y)-\\beta\\,\\mathrm{KL}(\\pi(\\cdot|x)\\|\\pi_{\\mathrm{ref}}(\\cdot|x))]$. Because the loss is off-policy and token-level, training can reuse static datasets or replay buffers, and inference remains a plain softmax over the model's logits. If the paper is right, offline and multi-turn RL fine-tuning no longer needs paired preference data, importance sampling, or extra value networks.","feed_headline":"Bellman-based ShiQ loss provably recovers the optimal LLM policy","feed_subtitle":"Off-policy and token-level, it treats logits as Q-values and needs no paired preferences or extra networks.","key_machinery":"The central object is a regularized Bellman consistency equation rewritten as a relation between logits and log-partitions, shifted by the reference model. The paper defines the token-level log-partition $v_\\ell(s)=\\ln\\sum_a\\exp\\ell(s,a)$ and proves that the policy $\\pi_\\ell(a|s)\\propto\\exp\\ell(s,a)$ remains the unique optimal policy through a sequence of four transformations: the soft Bellman equation for a Q-function (Theorem 1), a reparameterization that lets the logits themselves be sampled at inference (Theorem 2), a potential-based reward shaping that makes the reference logits a natural initialization (Theorem 3), and a telescoping argument yielding the multi-step identity $\\beta(v_\\ell(s_t)-v_{\\mathrm{ref}}(s_t))=\\sum_{k=t}^{T}\\gamma^{k-t}(r(s_k,a_k)-\\beta\\ln\\frac{\\pi_\\ell(a_k|s_k)}{\\pi_{\\mathrm{ref}}(a_k|s_k)})$ (Theorem 4). ShiQ is the squared residual of this identity, summed over every token in the completion. This machinery does the work of preserving the optimal policy across each modification while making the loss off-policy, token-wise, and free of additional networks.","core_discovery":"The paper's central claim is that an LLM's logits can be treated as Q-values, provided the Bellman equation is rewritten to respect three LLM-specific constraints: inference must sample directly from the trained logits; the reference model's logits should be a natural initialization, meaning no gradient update when the reward is identically zero; and sparse terminal rewards must propagate across many tokens. Theorems 1 through 4 show that each transformation, reparameterizing the Q-function into a softmax logit, applying potential-based reward shaping with the reference log-partition as potential, and telescoping the one-step equation into a multi-step consistency identity, preserves the unique optimal policy. The resulting loss is $L_{\\mathrm{ShiQ}}(\\ell)=\\mathbb{E}_{x,y\\in D}\\sum_{t=1}^{|y|}\\bigl(R(x,y)-\\beta\\ln\\frac{\\pi_\\ell(y_{\\ge t}|x,y_{<t})}{\\pi_{\\mathrm{ref}}(y_{\\ge t}|x,y_{<t})}-\\beta(v_\\ell(x\\oplus y_{<t})-v_{\\mathrm{ref}}(x\\oplus y_{<t}))\\bigr)^2$. The direct corollary of Theorem 4 is that under $\\operatorname{supp}(D)=\\operatorname{supp}(\\rho\\pi_{\\mathrm{ref}})$, the unique minimizer $\\ell^\\star$ satisfies $L_{\\mathrm{ShiQ}}(\\ell^\\star)=0$ and $\\pi_{\\ell^\\star}=\\pi^\\star$. The experiments then claim that on the multi-turn function-calling benchmark BFCL-V3, ShiQ outperforms the multi-turn DPO and CoPG baselines by exploiting full reward-position information, while on single-turn Anthropic-Harmless and UltraFeedback it matches CoPG using less information.","pith_inferences":["The paper's support assumption is a natural place to extend: a pessimistic or data-coverage penalty added to Eq. (13) would give ShiQ a principled regime when the dataset under-covers the reference model, and the bandit toy from Sec. 3.1 could be rerun with increasingly skewed arm datasets to test such an extension.","Because the loss consumes unpaired prompt-completion-reward triples, any offline corpus with scalar quality scores, such as verifier accuracy for code, unit-test pass rates, or retriever scores, becomes usable training signal without constructing preference pairs; this is an application the paper motivates but does not explore.","The telescoping trick in Theorem 4 is a general consequence of KL regularization and exact token log-probabilities, so a version of ShiQ could plausibly apply to other structured sequence models; whether it remains stable under model misspecification is an open question the paper leaves implicit.","The paper's own ablation suggests ShiQ/tk is competitive when rewards arrive at turn boundaries, so a practical system could switch between the token-level and sequence-level losses depending on whether dense rewards are available."],"forward_implications":["If the population-level result is right, offline RL fine-tuning of an LLM can be done with a single policy network: the trained logits are already Q-values, and inference softmax needs no reference model or extra value head.","Because the loss is token-level and off-policy, any fixed dataset of prompt-completion-reward triples, including replay buffers from prior generations, can be reused without fresh on-policy rollouts or importance sampling.","The multi-step form propagates sparse terminal rewards to earlier tokens within a single update, which is what allows ShiQ to exploit intermediate reward information in multi-turn function calling.","On BFCL-V3 the paper reports that ShiQ outperforms multi-turn DPO and CoPG on the verifiable reward, and on the single-turn Anthropic-Harmless and UltraFeedback datasets ShiQ matches CoPG while using only unpaired trajectories.","The sequence-level ablation ShiQ/tk, which the paper shows is competitive when rewards arrive at turn boundaries, offers a cheaper alternative to learned value networks in off-policy LLM alignment."],"supporting_citations":[{"why":"Supplies the regularized-MDP formulation and Bellman optimality used by Theorem 1.","marker":"[15]"},{"why":"Provides the potential-based reward-shaping result behind the initialization trick in Theorem 3.","marker":"[29]"},{"why":"Path consistency learning, the multi-step consistency idea that Theorem 4 adapts to the KL-regularized LLM setting.","marker":"[28]"},{"why":"Contributes the logits-as-Q reparameterization idea used in Theorem 2.","marker":"[43]"},{"why":"The contrastive policy-gradient baseline compared empirically and the related off-policy bandit loss.","marker":"[12]"},{"why":"The single-trajectory baseline DRO and the value-function comparison discussed in the appendix.","marker":"[34]"},{"why":"Direct preference optimization, the paired-data baseline used in single-turn comparisons.","marker":"[32]"},{"why":"The multi-turn DPO baseline evaluated on BFCL-V3 in the multi-turn comparison.","marker":"[33]"}],"fun_headline_variants":["ShiQ: Bellman loss provably turns LLM logits into Q-values","ShiQ: Off-policy Bellman loss for LLMs, provably optimal","ShiQ: No extra networks, logits as Q-values, Bellman-derived loss","ShiQ: Bring Bellman to LLMs, provably recovers optimal policy","ShiQ: Q-learning for LLMs, provably optimal from logits"],"cache_read_input_tokens":30592,"weakest_assumption_plain":"The result collapses if the training data does not cover every prompt-completion pair the reference policy can generate, because the unique-minimizer-equals-optimal-policy proof relies on $\\operatorname{supp}(D)=\\operatorname{supp}(\\rho\\pi_{\\mathrm{ref}})$, while real training uses finite data and a neural network outside that guarantee.","fun_headline_variants_meta":{"raw":{"variants":["ShiQ: Bellman loss provably turns LLM logits into Q-values","ShiQ: Off-policy Bellman loss for LLMs, provably optimal","ShiQ: No extra networks, logits as Q-values, Bellman-derived loss","ShiQ: Bring Bellman to LLMs, provably recovers optimal policy","ShiQ: Q-learning for LLMs, provably optimal from logits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001528,"raw_usage":{"total_tokens":6222,"prompt_tokens":1151,"completion_tokens":5071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":767,"completion_tokens_details":{"reasoning_tokens":4960}},"tokens_in":767,"tokens_out":5071,"duration_ms":37223,"temperature":1.0,"reasoning_tokens":4960,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:01:42.642914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a tabular two-action bandit with a uniform reference policy, rewards 10 and 0 for the two completions, and a dataset containing only the zero-reward completion. The paper's theorem does not apply because the support condition fails, so record whether ShiQ's learned policy stays near the reference or shifts toward the missing high-reward arm; the magnitude of that shift quantifies the gap in the guarantee. To test the theorem itself, a tabular full-support experiment should reproduce zero residual and $\\pi_{\\ell^\\star}=\\pi^\\star$ exactly, and any nonzero residual at the unique minimizer would contradict the corollary.","supporting_citations":[{"cited_title":"A theory of regularized markov decision processes","cited_arxiv_id":null,"evidence_quote":"Supplies the regularized-MDP formulation and Bellman optimality used by Theorem 1."},{"cited_title":"Policy invariance under reward transformations: Theory and application to reward shaping","cited_arxiv_id":null,"evidence_quote":"Provides the potential-based reward-shaping result behind the initialization trick in Theorem 3."},{"cited_title":"Bridging the gap between value and policy based reinforcement learning.Advances in neural information processing systems, 30, 2017","cited_arxiv_id":null,"evidence_quote":"Path consistency learning, the multi-step consistency idea that Theorem 4 adapts to the KL-regularized LLM setting."},{"cited_title":"Munchausen reinforcement learning.Advances in Neural Information Processing Systems, 33:4235–4246, 2020","cited_arxiv_id":null,"evidence_quote":"Contributes the logits-as-Q reparameterization idea used in Theorem 2."},{"cited_title":"Contrastive policy gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion","cited_arxiv_id":null,"evidence_quote":"The contrastive policy-gradient baseline compared empirically and the related off-policy bandit loss."},{"cited_title":"Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2023","cited_arxiv_id":null,"evidence_quote":"Direct preference optimization, the paired-data baseline used in single-turn comparisons."}],"review_version":1}