{"id":"e8009d81-d1eb-484c-b8b3-16056f3ff42e","arxiv_id":"2505.06987","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"straQ* uses an LLM's token logits as Q-values to select emotional support strategies, improving long-term response quality and out-of-domain generalization in experiments.","lead":"This paper trains a language model to act as a strategy planner for emotional support conversations, using Q-learning to pick the helping strategy that leads to the best long-term outcome. The planner is a lightweight plug-in that can steer any conversational LLM, and tests on two dialogue datasets show it improves strategy selection and response quality over several baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (2) omits the max over next actions, so the offline update is SARSA-style evaluation of the logged policy, not Q-learning; the 'converges to optimum returns' claim is unsupported unless the code uses max over all 8 strategies.","rationale":"The paper's empirical results are plausible, and Tables 2-4 show consistent gains over several baselines for both straQ*-distill and straQ*-imit. The released code, the transition-matrix analysis, and the human-evaluation effort are all credit-worthy. However, the method's signature contribution is the conversion of an LLM into a Q-function that optimizes long-term returns, and that conversion hangs on the exact form of the Bellman target. Eq. 2, as printed, lacks the max operator that defines Q-learning; the target uses Q_phi(s',a'), and since training is offline on ESConv, the most natural implementation is to use the logged next strategy. That is a SARSA/off-policy-evaluation update, so the learned Q estimates the value of the demonstrated data policy rather than the optimal policy. The paper's own Table 6 shows only a 0.61 Acc and 1.14 Q margin over the auto-regressive supervised baseline, so the Bellman machinery is not obviously performing extra work unless the max target is actually present. This does not invalidate the empirical comparisons, but it does mean the 'converges to optimum returns' claim and the value interpretation in Table 5 need to be qualified or verified with the correct target. I agree with the reader's conditional verdict; the missing max should be stated as an explicit condition rather than dismissed as an implementation detail. My agreement is partial because the reader framed the issue as an ambiguity in the target, whereas I read Eq. 2 as omitting the max and therefore changing the learning objective from Q-learning to SARSA-style evaluation.","tokens_in":13382,"tokens_out":5619,"duration_ms":57496,"concrete_test":"Inspect the training loop in the released StraQ repository (github.com/suran662/StraQ) to determine which next-action value forms the target in Eq. 2: the logged a' from ESConv or max over all 8 strategy logits. Then run two controlled retrains of straQ*-imit with identical hyperparameters: (A) logged-next-action target (SARSA-style) and (B) max over all strategies at the next state (true Q-learning). Compare Acc, Q, B-2, and R-L on the ESConv test set used in Tables 2 and 6, with bootstrap confidence intervals over 1,000 resamples. If (A) and (B) differ by more than the auto-regressive margin (0.61 Acc), the reported numbers depend on the choice of target and the optimality claim is not established; if they are indistinguishable, Eq. 2 is effectively SARSA and the paper must reword the convergence claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning with the Bellman loss (Eq. 2) converts an LLM into a Q-function that selects optimal strategies. The load-bearing step is Eq. 2: L = |r(s,a) + Q_phi(s',a') - Q_theta(s,a)|^2. As written, and as described in Sections 3.3-3.4, training consumes offline (s,a,r,s') tuples from ESConv, so the natural implementation uses the logged next strategy a' as the target action. That target, r + gamma * Q_phi(s',a'), is a SARSA-style update evaluating the data-collection policy, not the Q-learning target r + gamma * max_{a'} Q_phi(s',a'). The Bellman optimality equation (Eq. 1) contains the max; the loss omits it. Without the max, greedy selection at inference (Eq. 6) is not justified as optimizing long-term return, and the 'converges to optimum returns' contribution claim, plus the interpretation of Table 5, goes beyond what the training objective guarantees. This is not cosmetic: if the target is the logged action, Q approximates the value of the behavior policy, and argmax_a Q(s,a) can be suboptimal when the behavior policy is biased or when unvisited actions have high true value. Conversely, if the implementation does use max over all K=8 strategies, the paper should say so explicitly, and offline max over actions raises distribution-shift and overestimation concerns that are not addressed. Either way, the method needs clarification. The empirical margin over the auto-regressive ablation (Table 6: Acc 46.83 vs 46.22; Q 43.15 vs 43.01) is small enough that the distinction between SARSA-style fitting and true Q-learning matters for interpreting the results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes straQ*, a framework that converts a pretrained LLM into a value-based strategic planner for emotional support conversation (ESC). It defines a strategy-level MDP, represents Q(s,a) as averaged action-token logits of an LLM, fine-tunes the LLM with a squared Bellman loss on offline ESConv data, and selects strategies by argmax over the resulting Q function. Two reward mechanisms are studied: imitation rewards (+1/-1 for dataset actions) and distillation rewards (GPT-4 scores). Experiments on ESConv and EmpatheticDialogues compare against direct inference, self-refine, chain-of-thought, fine-tuning, and FSM baselines, reporting improvements in strategy accuracy, response similarity, human judgments, and GPT-4-evaluated returns.","tokens_in":13719,"tokens_out":5573,"duration_ms":56566,"significance":"The empirical setup is attractive in several ways: the planner is lightweight and plug-and-play, the code is released, two reward formulations are compared, evaluation includes cross-domain and human judgments, and ablations and sensitivity analyses are provided. If the theoretical framing were correct, the paper would offer a useful new way to inject long-term strategic value into LLM dialogue. However, the central Q-learning claim is compromised by the offline update in Eq. (2), which uses the logged next action rather than the max over actions, and by the unvalidated use of averaged logits as Q-values. The empirical gains are plausible but the contribution as stated overclaims.","major_comments":[{"comment":"The loss is written as L = |r(s,a) + Q_phi(s',a') - Q_theta(s,a)|^2, with no max over next actions, while Eq. (1) is the Bellman optimality equation. On offline data, a' is the dataset's next strategy, so this is a SARSA-style policy-evaluation update for the logging policy, not Q-learning. Consequently, greedy selection in Eq. (6) is not justified as optimizing long-term return, and contribution (2)'s 'converges to optimum returns' is unsupported. If the implementation actually computes max over the K strategies, the paper must say so explicitly and discuss offline overestimation; either way the current text is misleading.","section":"Section 2, Eq. (2)"},{"comment":"Averaged action-token logits are asserted to be Q(s,a), but no evidence is given that logit magnitudes lie on the scale of the rewards (0-5 for distill, +/-1 for imitation). Without calibration or a stated link between logits and value, minimizing the TD loss can simply fit the reward statistics of the training corpus. The definition of 'averaged action logits' is also ambiguous: which tokens are averaged, and how does variable-length strategy phrasing affect the average? Please specify the exact computation and justify the scale.","section":"Section 3.3, Eq. (5)"},{"comment":"For straQ*-distill, the same GPT-4 judge supplies both the training reward and the reported average reward; the claim that straQ* 'statistically obtains higher returns' is therefore partly circular. Reporting an evaluation with a different judge, with human annotations, or on held-out reward labels would be needed to support the returns claim.","section":"Section 4.4, Table 5"},{"comment":"The only evidence for 'converges to optimum returns' is a decreasing training loss curve. A loss curve does not show convergence of Q to Q*; the paper provides no convergence analysis, no evaluation of Bellman error on a held-out set, and no comparison of the learned Q function to Monte Carlo returns. Please either add such an evaluation or remove or weaken the convergence claim.","section":"Section 4.3, Figure 4"},{"comment":"The advantage of straQ*-imit over the auto-regressive ablation is small (Acc 46.83 vs 46.22; Q 43.15 vs 43.01; B 0.80 vs 0.69, which is worse since lower B is better; B-2 7.63 vs 7.25; R-L 17.03 vs 16.48). No significance tests are reported, so the statement that straQ* 'can better align with the strategy semantics' is not established by this table.","section":"Table 6"}],"minor_comments":[{"comment":"The phrase 'might not define the diagram from the state model perspective' is unclear and should be rephrased, for example as 'might not formulate the problem from a state-model perspective'.","section":"Abstract"},{"comment":"The notation '↢' is nonstandard; please define precisely what averaging over action logits means and which tokens are included.","section":"Eq. (5)"},{"comment":"For the imitation reward, the paper states that a different action is randomly sampled and assigned r = -1; please specify whether the negative samples are drawn from the same state and whether this creates an unbalanced objective.","section":"Section 3.4"},{"comment":"The quantity <value> in Table 5 is not defined; please state the formula used to compute it, e.g., the discounted sum of GPT-4 rewards over subsequent turns.","section":"Section 4.4"},{"comment":"The human evaluation was conducted by the authors' interns; the limitation statement mentions potential bias, but reporting inter-annotator agreement would strengthen the reliability of the human results.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The practical recipe may be useful, but the Q-learning framing needs substantial repair. The authors should either implement the max variant of the Bellman target and discuss offline RL concerns, or reframe the method as SARSA-style policy evaluation and weaken the 'optimal returns' claims. The GPT-4 judge circularity in Table 5 should also be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the paper's central move—fine-tune an LLM to output averaged action-token logits and call that Q(s,a)—is a sensible, lightweight way to turn a language model into a strategy selector. The experiments are decent and the gains over direct inference, CoT, and FSM look real, especially for the distill variant on out-of-domain data. That part is worth taking seriously.\n\nWhat's new is the packaging: a strategy-level MDP, two reward schemes (imitation with +/-1 and distillation from a GPT-4 judge), and a plug-and-play planner that steers a separate response model. The authors also release code, which is more than many papers do. The ablation against an auto-regressive baseline shows straQ*-imit only slightly ahead (Acc 46.83 vs 46.22; Q 43.15 vs 43.01), but the B-2 and R-L improvements are a bit larger. That is a modest but nonzero signal.\n\nThe soft spot, and it is load-bearing, is Equation (2). As written, the loss uses Q_phi(s',a') with the dataset's next action a', not max over actions. That is a SARSA-style update evaluating the behavior policy, not Q-learning. The paper claims \"converges to optimum returns\" and says the planner picks \"optimal strategy based on long-term returns.\" If the code actually uses the max, fine, but the text needs to say so. If it does not, then argmax over Q at inference is not justified by the training objective, and the comparison to the auto-regressive baseline becomes the whole story: the TD bootstrapping adds very little over plain next-strategy prediction. Either way this needs a clear correction, not a tweak.\n\nOther issues are more standard: no error bars or significance tests, a small human evaluation that the authors themselves flag as possibly biased (Section 7), and a \"w/ value head\" ablation that is really a reward-model-style head, not a value head in the RL sense. The sensitivity analysis on gamma is fine but not deep.\n\nWho should read this? People working on RL-for-dialogue or emotional support systems will want the setup, and the code could be a useful starting point. The central idea is not paradigm-shifting, but it is practical and the empirical story is mostly consistent.\n\nMy recommendation: send it to peer review. It deserves referee time, but only with the requirement that the authors clarify the Bellman target, remove or qualify the \"converges to optimum returns\" claim, and report error bars. The math gap is fixable; the current ambiguity makes the paper's own strongest claims unverifiable as written.","headline":"A useful planning trick that is undercut by a Bellman target written as SARSA, not Q-learning; the claims outrun the objective as stated, but the idea and experiments deserve referee time.","tokens_in":14317,"tokens_out":1674,"would_cite":true,"duration_ms":19037,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a pretrained LLM can be fine-tuned with the Bellman equation to act as a value-based strategist that picks the optimal emotional-support strategy and improves responses.","keywords":["emotional support conversation","Q-learning","Bellman equation","strategy-level MDP","large language models","value-based planning","response generation"],"falsifier":"Take a held-out set of emotional-support conversations whose true downstream outcome (for example, the seeker's self-reported relief after the conversation) is measured, and compare the outcomes of conversations whose next strategy was chosen by the highest Q-value against conversations where a random, frequency-matched strategy was used; if the arg-max strategy does not lead to better outcomes, the Q-values are not measuring long-term value.","tokens_in":13101,"feed_emoji":"🎯","tokens_out":7372,"duration_ms":69586,"temperature":0.7,"pith_summary":"straQ* claims that a pretrained language model can be turned into a strategic planner for emotional-support conversation by replacing next-token prediction with the Bellman equation: the model reads the conversation state and a candidate strategy, and its averaged logits over the candidate's name act as a Q-value. The planner picks the strategy with the highest Q-value and passes it to a response generator. The authors report that this beats direct inference, self-refine, chain-of-thought, supervised fine-tuning, and finite-state-machine prompting on ESConv, and that it transfers zero-shot to EmpatheticDialogues. The attraction of the claim is that a small, plug-and-play planner can optimize long-term seeker satisfaction instead of just the immediate reply.","feed_headline":"LLM learns to pick support strategies that pay off long term","feed_subtitle":"Bellman-trained planner beats direct prompting, chain-of-thought, and refinement in emotional support.","key_machinery":"The load-bearing identity is $Q_\\theta(s,a)$ defined as the averaged action-token logits of $\\mathrm{LLM}_\\theta(I(s)\\oplus a)$, combined with the squared Bellman loss $L(\\theta)=|r(s,a)+Q_\\phi(s',a')-Q_\\theta(s,a)|^2$. Because the transformer's causal masking lets many turns be updated in parallel, the same architecture used for language modeling can be trained directly on this residual. The instruction template is formatted as a multiple-choice question over the $K$ strategies, which makes the action values comparable across candidates and lets inference be a single arg-max over option logits rather than free-form generation. The target network $\\phi$ is synchronized periodically, and the discount factor $\\gamma$ encodes how far ahead the planner looks; the paper's sensitivity analysis shows $\\gamma=0.9$ is best for strategy accuracy and $\\gamma=0.85$ for response-similarity metrics.","core_discovery":"On the paper's own terms, the central discovery is that the Bellman equation can serve as the training loss for a language model used as a value function. Emotional support is formalized as a strategy-level Markov decision process in which the eight annotated support strategies (question, restatement, reflection, self-disclosure, affirmation, suggestion, information, others) are the actions. The Q-value of a state-action pair is computed as the average logit of the action tokens when the instruction $I(s)$ is concatenated with the strategy answer $a$: $Q_\\theta(s,a) \\leftarrow \\mathrm{LLM}_\\theta(I(s)\\oplus a)$, and the optimal strategy is $a^\\star = \\arg\\max_a \\mathrm{LLM}(I(s)\\oplus a)$. Training minimizes the squared Bellman residual with a periodically synchronized target network, and two reward types are tested: +1 for the dataset's annotated strategy with −1 for a randomly sampled alternative, and a teacher-model judge score on a 0–5 scale. The authors claim this yields higher strategy accuracy, lower preference bias, improved response-similarity metrics, and better human-evaluated satisfaction than the baselines, with the distilled reward generalizing better.","pith_inferences":["The identity between averaged logits and Q-values is not forced by theory; a natural stress test is to compare it with a learned scalar head on the same hidden states, and the paper's own ablation shows a value head underperforms, hinting that keeping the value on the token-logit scale preserves semantic comparability.","The method should transfer to any dialogue or decision task with a small enumerable set of textual actions — tutoring strategies, medical interview moves, negotiation tactics — whenever a reward can be annotated or distilled, though the paper only tests emotional support.","Since training is offline, the Q-values can only be as good as the coverage of the corpus; a testable extension would be to interleave straQ* with online self-play or seeker simulation and check whether Bellman updates continue to improve beyond the expert transitions in the training set.","The reward definitions differ in an informative way: imitation rewards only distinguish the annotated action from a random one, while distillation rewards grade all actions, so the out-of-domain gap between the two variants suggests that reward granularity, not strategy taxonomy, drives generalization."],"forward_implications":["If the claim holds, a small value-based planner can be bolted onto an existing conversational LLM, steering it toward coherent long-horizon strategy progression without retraining the generator.","Strategy selection can be trained purely from an offline dialogue corpus using either expert-imitation rewards or distilled judge rewards, and the report that distillation transfers better to out-of-domain conversations suggests a practical recipe for data-lean settings.","Because the Q-network is an LLM, scaling the base model improves response quality, so the method tracks LLM capability improvements while still keeping the planner small.","The pattern of higher averaged rewards and values for chosen strategies than for direct inference indicates that the planner is shifting the distribution toward what the judge rewards, not merely mimicking frequency."],"supporting_citations":[{"why":"Supplies the ESConv dataset with the 8-strategy taxonomy and expert transitions that define the action space and TD targets.","marker":"Liu et al., 2021"},{"why":"Provides EmpatheticDialogues as the out-of-domain zero-shot test bed for response quality.","marker":"Rashkin et al., 2019"},{"why":"Defines the proficiency Q and preference-bias B metrics and the human annotation dimensions used to evaluate planning quality.","marker":"Kang et al., 2024"},{"why":"Identifies the pretrained instruction-tuned base model that is fine-tuned into the Q-function in the experiments.","marker":"AI@Meta, 2024"},{"why":"Defines the self-refine baseline against which straQ* is compared in both automatic and human evaluation.","marker":"Madaan et al., 2023"},{"why":"Defines the chain-of-thought prompting baseline that the method is claimed to outperform.","marker":"Wei et al., 2022"},{"why":"Defines the finite-state-machine prompting baseline used in the strategy-selection comparisons.","marker":"Wang et al., 2024b"}],"fun_headline_variants":["Bellman-trained LLM picks support strategies for long-term gains","LLM as value-based planner for emotional support conversations","Q-learning on LLMs: strategic planning for empathetic responses","Train LLM with Bellman loss to choose optimal support moves","Value-based LLM planner beats CoT and finetuning in ESC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the average score the model assigns to the words of a strategy name is a genuine measure of how valuable that strategy is, and that the next strategy written in the training dialogues is a faithful stand-in for the ideal next action; if either fails, the planner will fit the statistics of the corpus rather than improve the seeker's long-term state.","fun_headline_variants_meta":{"raw":{"variants":["Bellman-trained LLM picks support strategies for long-term gains","LLM as value-based planner for emotional support conversations","Q-learning on LLMs: strategic planning for empathetic responses","Train LLM with Bellman loss to choose optimal support moves","Value-based LLM planner beats CoT and finetuning in ESC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1311,"prompt_tokens":915,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":531,"tokens_out":396,"duration_ms":4209,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:28:19.649392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of emotional-support conversations whose true downstream outcome (for example, the seeker's self-reported relief after the conversation) is measured, and compare the outcomes of conversations whose next strategy was chosen by the highest Q-value against conversations where a random, frequency-matched strategy was used; if the arg-max strategy does not lead to better outcomes, the Q-values are not measuring long-term value.","supporting_citations":[],"review_version":1}