{"id":"5a11f117-eb82-48d0-8076-6201d7a5b1d2","arxiv_id":"2506.14574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TGDPO modifies DPO by weighting each token's log-ratio with an external token-level reward, and reports win-rate gains over DPO and SimPO on three instruction-following benchmarks.","lead":"This paper proposes TGDPO, a modified DPO training loss that weights each token's contribution by a token-level reward, typically derived from a pre-trained DPO model. It reports higher win rates than DPO and SimPO on MT-Bench, AlpacaEval 2, and Arena-Hard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Partition-function elimination in Theorem 4.4 is applied at dataset level, but per-instance monotonicity does not survive summation over instances, so Eq. (20) is not the MLE loss of Eq. (17).","rationale":"The reader's weakest assumption was the state-distribution relaxation s_t ~ D_t to s_t ~ D in Eq. (9). That is a real but standard approximation (also present in TDPO and related work), and it weakens the theoretical optimality claim without necessarily invalidating the loss. The partition-function elimination issue is more severe because it is a logical error in the derivation of the very loss that defines TGDPO: per-instance equivalence of the preference function is confused with equivalence of the expected loss. This directly undermines the abstract's claim that the work 'establishes a framework of computable loss functions' from the Bradley-Terry model. The empirical results, while promising, do not repair the derivation; they also lack error bars and show a regression on MT-Bench for Llama3-8B-Instruct ArmoRM (TGDPO score 7.9 and win rate 45.0 vs DPO 8.2 and 46.3), so the 'consistent' improvement claim is already fragile. Given that the central contribution is the derivation, the appropriate verdict is REJECT in the current form; a revised version that reframes TGDPO as a heuristic reweighted DPO, removes the claim of a derived framework, and adds statistical validation could be reconsidered. Credit is due for the code release and a clear experimental section, but those positive aspects do not compensate for the invalid derivation.","tokens_in":22954,"tokens_out":14376,"duration_ms":144779,"concrete_test":"Construct the two-instance counterexample with phi1=theta, phi2=-theta, delta1=5, delta2=-5. Minimize L_full(theta) = -log sigma(theta+5) - log sigma(-theta-5) and L_TGDPO(theta) = -log sigma(theta) - log sigma(-theta). The argmins are theta=-5 and theta=0 respectively, demonstrating that the dataset-level objectives are not equivalent. For an end-to-end check, on a small preference dataset with a tiny vocabulary where every prefix distribution is enumerable, compute Z([x,y_{ yl | x) as sigma(phi) and using -log sigma(phi) as the per-instance loss (Eq. 20). Theorem 4.4 is valid only per instance: for a fixed (x,yw,yl), sigma(phi+delta) and sigma(phi) have identical ascent directions and stationary points because sigma' > 0 and delta is theta-independent. However, the training loss is the expectation over the dataset. The gradient of -E[log sigma(phi+delta)] is E[(1-sigma(phi+delta)) grad phi], whereas the gradient of -E[log sigma(phi)] is E[(1-sigma(phi)) grad phi]; the instance-dependent factor (1-sigma(phi+delta))/(1-sigma(phi)) is not constant, so the two objectives have different stationary points and different global optima. A concrete counterexample: take two instances with phi1=theta, phi2=-theta, delta1=5, delta2=-5. The full negative log-likelihood -log sigma(theta+5)-log sigma(-theta-5) is minimized at theta=-5; the TGDPO loss -log sigma(theta)-log sigma(-theta) is minimized at theta=0. So dropping delta changes the optimizing policy. This is not a minor approximation: the partition-function terms are reward- and response-dependent, and the redefinition asserts an equivalence that holds per example but not for the summed objective. Consequently, Eq. (20) is not the maximum-likelihood loss of the Bradley-Terry model in Eq. (17), and the paper's central claim that TGDPO is a derived, computable DPO framework with token-level reward guidance is not supported. The method may still be a useful heuristic, but its theoretical foundation as presented is invalid.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TGDPO, a direct preference optimization variant that weights each token-level log-ratio term in the DPO loss by a function f(r_hat) of a token-level reward, with the practical choice f_w = 1 + alpha r_hat on win tokens and f_l = 1 - alpha r_hat on lose tokens. The derivation starts from a sequence-level PPO objective with token-level rewards, decomposes it into token-level PPO problems, incorporates the reward guidance, obtains a closed-form optimal token-level policy and an associated token-level reward, and then plugs this reward into the Bradley-Terry model. After eliminating the partition function via the new Theorem 4.4, the authors arrive at the loss in Eq. (20). Experiments on AlpacaEval 2, MT-Bench, and Arena-Hard across three base models report win-rate improvements over DPO and SimPO, with gains up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard, and additional analyses of convergence speed, robustness, and SFT initialization.","tokens_in":23383,"tokens_out":7839,"duration_ms":73520,"significance":"If the derivation were valid, the framework would provide a principled way to inject dense token-level reward signals into DPO while recovering several existing losses as special cases. The empirical direction is potentially useful: the token-level weighting is simple, the paper releases code, includes an additional TDPO baseline, and reports an SFT-initialization experiment, all of which are strengths. The convergence analysis in Section 5.3 is also a genuinely interesting observation. However, the central theoretical claim is not currently established: the partition-function elimination is applied at the wrong level, the relaxation in Eq. (9) changes the objective, and the single-policy representational assumption is asserted rather than proved. In addition, the experimental support is weaker than claimed because there are no error bars, one table cell actually shows a decrease relative to DPO, and hyperparameters are grid-searched per setting without a stated validation split. The core idea may be salvageable as a practical heuristic, but the paper in its present form overclaims its theoretical grounding.","major_comments":[{"comment":"The partition-function elimination is proved only for a fixed instance, but the loss in Eq. (20) is an expectation over the dataset. For a fixed (x, yw, yl), delta is independent of theta and the sigmoid is strictly increasing, so the per-instance maximizers of sigma(phi + delta) and sigma(phi) coincide. For the dataset objective, however, the gradient of -E[log sigma(phi + delta)] is E[(1 - sigma(phi + delta)) grad_theta phi], while the gradient of -E[log sigma(phi)] is E[(1 - sigma(phi)) grad_theta phi]; the instance-dependent ratio (1 - sigma(phi + delta))/(1 - sigma(phi)) is not constant, so the two objectives generally have different stationary points and different global optima. As a concrete counterexample, two instances with phi_1 = theta, phi_2 = -theta, delta_1 = 5, delta_2 = -5 give a full negative log-likelihood -log sigma(theta + 5) - log sigma(-theta - 5) minimized at theta = -5, while the TGDPO loss -log sigma(theta) - log sigma(-theta) is minimized at theta = 0. Thus Eq. (20) is not the maximum-likelihood loss of the Bradley-Terry model in Eq. (17), and the claim that TGDPO is a derived, computable DPO loss for that model is not established.","section":"Section 4.3, Theorem 4.4, Eq. (20)"},{"comment":"The relaxation of s_t ~ D_t to s_t ~ D replaces the policy-dependent token-state distribution with a fixed data distribution. The closed-form optimal policy in Theorem 4.3 and the reward representation in Eq. (14) solve only the relaxed problem, and the paper provides no bound or correction term relating the relaxed objective to the original sequence-level PPO objective in Eq. (2). Because the state distribution generated by the policy changes during training, the loss in Eq. (20) is not shown to optimize the RLHF objective that motivates it.","section":"Section 4.2, Eq. (9)"},{"comment":"The 'without loss of generality' claim that a single over-parameterized theta realizes the per-token optimal policies pi_{theta_t} for all positions simultaneously is a substantive representational assumption, not a consequence of over-parameterization. Training a single policy with the loss in Eq. (20) does not guarantee that each per-token optimal policy is individually realized, so the identification of the ground-truth reward with a single theta in Eq. (14) is not established.","section":"Section 4.2, paragraph before Eq. (14)"},{"comment":"The experimental evidence for the central empirical claim is weaker than the text suggests. Table 1 reports no error bars or significance tests, and in the Llama3-8B-Instruct ArmoRM row TGDPO scores 7.9 on MT-Bench versus DPO's 8.2 and a 45.0% win rate versus DPO's 46.3%, which contradicts the statement that TGDPO 'consistently outperforms baseline methods.' Additionally, hyperparameters are grid-searched per setting in Section D.1 with no stated validation split, so the reported gains may reflect selection over configurations.","section":"Section 5.2, Table 1"}],"minor_comments":[{"comment":"The item labeled 'Assumption 4.2' is later referred to as 'Theorem 4.2' in the text; the numbering and terminology should be made consistent.","section":"Section 4.2"},{"comment":"In the recovery of SimPO, the text defines f_l(r_hat) = 1/|y_w| but the following expression uses 1/|y_l|; one of these is a typo and should be corrected.","section":"Section C.2"},{"comment":"The tables are typeset too compactly, especially the header rows and the MT-Bench score/win-rate columns; please reformat for readability and use consistent spacing in numeric entries.","section":"Tables 1, 5-10"}],"recommendation":"major_revision","confidential_remarks":"The partition-function issue in Section 4.3 is fundamental to the paper's theoretical contribution. If the authors cannot prove a dataset-level equivalence or provide a valid surrogate objective, they should substantially reframe the paper as an empirical method with a heuristic motivation. The experimental section would also need error bars and a clear validation procedure before the empirical claims can be credited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes a token-level reward-weighted DPO loss and shows solid win-rate gains across three models and three benchmarks. The practical recipe – f_w = 1+αr̂, f_l = 1−αr̂ – is new and simple, and the framework does recover DPO, SimPO, R-DPO and D2PO as special cases. That part is worth reading.\n\nThe soft spot is the math underneath. The paper claims that because the partition-function term δ is θ-independent, maximizing σ(φ+δ) per instance is equivalent to maximizing σ(φ), and therefore the loss can be redefined as −log σ(φ). This is true for a single fixed instance. But the training objective is the sum of these log-likelihoods across the dataset. The δ terms vary per instance, so they change the relative weighting of gradients and the stationary points. A simple two-instance example shows the two objectives have different optima: φ1=θ, φ2=−θ, δ1=5, δ2=−5 gives full NLL minimized at θ=−5, while TGDPO's loss is minimized at θ=0. So Eq. (20) is not the MLE loss of the Bradley-Terry model in Eq. (17). The stress-test note is correct: the central theoretical claim is not supported.\n\nThat doesn't mean the method is useless. The token-weighted loss may still be a good heuristic, and the empirical results are encouraging. But the paper presents it as derived from a token-level PPO and a “new result for eliminating the partition function,” and that derivation is invalid as written.\n\nOther concerns are minor in comparison: no error bars, hyperparameters tuned per setting without a clear validation split, one MT-Bench regression (Llama3-8B ArmoRM score 7.9 vs 8.2 for DPO), and the relaxation in Eq. (9) from the policy-dependent state distribution to the data distribution is another unquantified step. The convergence analysis in Section 5.3 is interesting but informal.\n\nWho should read this: anyone working on DPO variants will find the loss formulation useful as a heuristic and the empirical comparisons informative. It deserves a serious referee, but the authors need to either fix the aggregation argument or reframe the contribution as a heuristic framework rather than a derivation. I would not cite it as a theoretical result in its current form.","headline":"A useful token-level reward-weighted DPO heuristic with real empirical gains, but the theoretical derivation is invalid because the partition-function cancellation does not survive dataset-level summation.","tokens_in":23956,"tokens_out":3499,"would_cite":false,"duration_ms":32825,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A token-level reward loss lifts DPO win rates by up to 7.5 points.","keywords":["direct preference optimization","token-level reward","RLHF","reward guidance","language model alignment","Bradley-Terry model","proximal policy optimization","TGDPO"],"falsifier":"Shuffle the token-level rewards across token positions while keeping each sequence-level total reward unchanged, then train TGDPO under the same settings; if token-level guidance is the operative mechanism, the gains over DPO on AlpacaEval 2 and Arena-Hard should largely disappear, and if the gains are actually sequence-level they should persist.","tokens_in":22797,"feed_emoji":"🎯","tokens_out":9119,"duration_ms":77339,"temperature":0.7,"pith_summary":"Direct Preference Optimization (DPO) aligns language models with human preferences at the level of whole responses, which leaves per-token quality signals unused. This paper sets out to bring dense, token-level rewards into DPO without abandoning DPO's reference-model form. It decomposes sequence-level proximal policy optimization into token-level problems, derives a closed-form optimal token-level policy, and from it builds a DPO-style loss in which the weight on each token is a function of that token's reward. With the concrete choice $f_w = 1 + \\alpha \\hat{r}$ and $f_l = 1 - \\alpha \\hat{r}$, the paper reports win-rate gains over DPO of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard. The central claim a sympathetic reader should take away is that sequence-level DPO can be improved by the same fine-grained reward signal that already helps PPO.","feed_headline":"Token-level rewards lift DPO win rates by up to 7.5 points","feed_subtitle":"A new token-weighted DPO loss, derived from a closed-form token-level policy, beats DPO on three instruction-following benchmarks.","key_machinery":"The load-bearing object is the modified token-level proximal policy optimization problem with reward guidance (Equation 11), obtained by relaxing $s_t \\sim D_t$ to $s_t \\sim D$ and dividing the reward by $\\beta f(\\hat{r}(s_t,a_t))$ before solving. Its closed-form optimal policy supplies the token-level reward representation that is inserted into the Bradley-Terry model. The second mechanism is Theorem 4.4: because the partition-function contributions are independent of $\\pi_{\\theta}$ and the sigmoid is strictly increasing, those contributions can be removed while preserving the maxima and ascent directions, yielding the computable loss in Equation (20). The practical choice $f_w = 1 + \\alpha \\hat{r}$, $f_l = 1 - \\alpha \\hat{r}$ turns the induced DPO token reward into per-token gradient weights that push the policy toward preferred tokens and away from dispreferred ones.","core_discovery":"Starting from the gap that DPO is a sequence-level bandit while PPO benefits from dense token rewards, the paper derives a token-level analogue of the DPO reparameterization. The derivation replaces the policy-dependent state distribution with the data distribution to make the problem tractable, injects a positive function $f(\\hat{r}(s_t,a_t))$ into the KL-regularized token-level objective, and obtains the closed-form optimal policy $\\pi_{\\theta}(a_t|s_t) = \\pi_{\\text{ref}}(a_t|s_t) \\exp(r_{\\phi}/(\\beta f(\\hat{r})))/Z(s_t)$. Inverting that policy expresses the unknown token reward as $\\beta f(\\hat{r}) \\log(\\pi_{\\theta}/\\pi_{\\text{ref}}) + \\beta f(\\hat{r}) \\log Z$, and substituting this into the Bradley-Terry model gives a per-token weighted preference loss. The partition-function terms are independent of the policy being optimized, so by the monotonicity of the sigmoid they can be discarded without changing the maximum or the ascent directions. The practical method sets $f_w = 1 + \\alpha \\hat{r}$ and $f_l = 1 - \\alpha \\hat{r}$, which upweights high-reward tokens of the chosen response and downweights low-reward tokens of the rejected response; experiments on three benchmarks and several base models report consistent gains over DPO and SimPO.","pith_inferences":["An untested extension is to feed the same loss with rewards from an external fine-grained reward model or process-supervision signal, since the derivation only requires a positive function of any token-level reward.","The relaxation that replaces the on-policy prefix distribution with the data distribution suggests TGDPO may be optimizing an upper-bound proxy of the sequence-level objective; comparing it against on-policy token-level PPO would measure how much the relaxation costs.","The assumption that one over-parameterized network realizes every per-token optimal policy simultaneously could be probed by running TGDPO on small models, where the benefit over DPO should shrink if that assumption carries the argument."],"forward_implications":["Each token is free to deviate from the reference policy by an amount tied to its own reward, instead of by one sequence-level constant.","The framework contains DPO as the special case $f_w = f_l = 1$, and it also recovers SimPO, R-DPO, and D2PO under other choices of $f$.","TGDPO retains good benchmark performance when trained to loss convergence, whereas DPO and SimPO degrade, which suggests less hyperparameter tuning is needed.","The value of $\\alpha$ controls convergence speed with comparable final performance, so training can be stopped early without much cost.","Using token rewards produced by DPO models trained with very different $\\beta$ values gives nearly the same final performance, indicating robustness to reward quality."],"supporting_citations":[{"why":"Supplies the sequence-level DPO loss and the induced reward reparameterization that TGDPO extends and compares against.","marker":"(Rafailov et al., 2023)"},{"why":"Provides the token-level interpretation of DPO and the implicit token reward form that TGDPO uses as its practical reward guidance.","marker":"(Rafailov et al., 2024)"},{"why":"Establishes the dense token-level reward guidance for PPO whose sequence-level objective TGDPO decomposes.","marker":"(Yang et al., 2023)"},{"why":"Supplies the token-level reward formulation and the sequence-level PPO objective with guidance that the paper starts from.","marker":"(Yin et al., 2025)"},{"why":"Gives the token-level DPO baseline and the state-distribution relaxation that TGDPO shares and builds on.","marker":"(Zeng et al., 2024)"},{"why":"Provides the paired-comparison model used to convert the derived token-level reward into a preference loss.","marker":"(Bradley & Terry, 1952)"},{"why":"Defines the PPO objective whose decomposition into token-level proximal problems is the starting point of the derivation.","marker":"(Schulman et al., 2017)"},{"why":"Provides the SimPO baseline, the experimental setup, and the evaluation protocol that the comparisons follow.","marker":"(Meng et al., 2024)"}],"fun_headline_variants":["Token-weighted DPO beats base DPO by 7.5 points","Reward-guided DPO: token-level boosts up to 7.5","Token rewards enhance DPO, win +7.5 on MT-Bench","New DPO loss uses token rewards for up to 7.5% gain","Dense token rewards improve DPO by up to 7.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation assumes the distribution over partial responses is fixed and independent of the policy being trained, when in reality the partial responses a policy sees are generated by that same policy; if that on-policy dependence matters, the derived loss optimizes a related but different objective.","fun_headline_variants_meta":{"raw":{"variants":["Token-weighted DPO beats base DPO by 7.5 points","Reward-guided DPO: token-level boosts up to 7.5","Token rewards enhance DPO, win +7.5 on MT-Bench","New DPO loss uses token rewards for up to 7.5% gain","Dense token rewards improve DPO by up to 7.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1768,"prompt_tokens":1094,"completion_tokens":674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":710,"tokens_out":674,"duration_ms":5275,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:01.091298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the token-level rewards across token positions while keeping each sequence-level total reward unchanged, then train TGDPO under the same settings; if token-level guidance is the operative mechanism, the gains over DPO on AlpacaEval 2 and Arena-Hard should largely disappear, and if the gains are actually sequence-level they should persist.","supporting_citations":[{"cited_title":"D., Ermon, S., and Finn, C","cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-level DPO loss and the induced reward reparameterization that TGDPO extends and compares against."},{"cited_title":"From r to q^* : Your language model is secretly a q-function","cited_arxiv_id":null,"evidence_quote":"Provides the token-level interpretation of DPO and the implicit token reward form that TGDPO uses as its practical reward guidance."},{"cited_title":"Preference-grounded token-level guidance for language model fine-tuning","cited_arxiv_id":null,"evidence_quote":"Establishes the dense token-level reward guidance for PPO whose sequence-level objective TGDPO decomposes."},{"cited_title":"Token-level direct preference optimization","cited_arxiv_id":null,"evidence_quote":"Gives the token-level DPO baseline and the state-distribution relaxation that TGDPO shares and builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the paired-comparison model used to convert the derived token-level reward into a preference loss."}],"review_version":1}