Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A token-level reward loss lifts DPO win rates by up to 7.5 points.

desk verdict A useful token-level reward-weighted DPO heuristic with real empirical gains, but the theoretical derivation is invalid because the partition-function cancellation does not survive dataset-level summation. read the letter →

arxiv 2506.14574 v1 pith:L5HEIOCS submitted 2025-06-17 cs.LG cs.AIcs.CL

Mingkang Zhu , Xi Chen , Zhongdao Wang , Bei Yu , Hengshuang Zhao , Jiaya Jia This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords directpreferenceoptimizationtoken-levelrewardRLHFguidancelanguagemodelalignmentBradley-TerryproximalpolicyTGDPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct Preference Optimization (DPO) aligns language models with human preferences at the level of whole responses, which leaves per-token quality signals unused. This paper sets out to bring dense, token-level rewards into DPO without abandoning DPO's reference-model form. It decomposes sequence-level proximal policy optimization into token-level problems, derives a closed-form optimal token-level policy, and from it builds a DPO-style loss in which the weight on each token is a function of that token's reward. With the concrete choice $f_w = 1 + \alpha \hat{r}$ and $f_l = 1 - \alpha \hat{r}$, the paper reports win-rate gains over DPO of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard. The central claim a sympathetic reader should take away is that sequence-level DPO can be improved by the same fine-grained reward signal that already helps PPO.

What carries the argument

The load-bearing object is the modified token-level proximal policy optimization problem with reward guidance (Equation 11), obtained by relaxing $s_t \sim D_t$ to $s_t \sim D$ and dividing the reward by $\beta f(\hat{r}(s_t,a_t))$ before solving. Its closed-form optimal policy supplies the token-level reward representation that is inserted into the Bradley-Terry model. The second mechanism is Theorem 4.4: because the partition-function contributions are independent of $\pi_{\theta}$ and the sigmoid is strictly increasing, those contributions can be removed while preserving the maxima and ascent directions, yielding the computable loss in Equation (20). The practical choice $f_w = 1 + \alpha \hat{r}$, $f_l = 1 - \alpha \hat{r}$ turns the induced DPO token reward into per-token gradient weights that push the policy toward preferred tokens and away from dispreferred ones.

What would settle it

Shuffle the token-level rewards across token positions while keeping each sequence-level total reward unchanged, then train TGDPO under the same settings; if token-level guidance is the operative mechanism, the gains over DPO on AlpacaEval 2 and Arena-Hard should largely disappear, and if the gains are actually sequence-level they should persist.

Watch

Extended reading notes

Core claim

Starting from the gap that DPO is a sequence-level bandit while PPO benefits from dense token rewards, the paper derives a token-level analogue of the DPO reparameterization. The derivation replaces the policy-dependent state distribution with the data distribution to make the problem tractable, injects a positive function $f(\hat{r}(s_t,a_t))$ into the KL-regularized token-level objective, and obtains the closed-form optimal policy $\pi_{\theta}(a_t|s_t) = \pi_{\text{ref}}(a_t|s_t) \exp(r_{\phi}/(\beta f(\hat{r})))/Z(s_t)$. Inverting that policy expresses the unknown token reward as $\beta f(\hat{r}) \log(\pi_{\theta}/\pi_{\text{ref}}) + \beta f(\hat{r}) \log Z$, and substituting this into the Bradley-Terry model gives a per-token weighted preference loss. The partition-function terms are independent of the policy being optimized, so by the monotonicity of the sigmoid they can be discarded without changing the maximum or the ascent directions. The practical method sets $f_w = 1 + \alpha \hat{r}$ and $f_l = 1 - \alpha \hat{r}$, which upweights high-reward tokens of the chosen response and downweights low-reward tokens of the rejected response; experiments on three benchmarks and several base models report consistent gains over DPO and SimPO.

Load-bearing premise

The derivation assumes the distribution over partial responses is fixed and independent of the policy being trained, when in reality the partial responses a policy sees are generated by that same policy; if that on-policy dependence matters, the derived loss optimizes a related but different objective.

Editorial extensions

If this is right

  • Each token is free to deviate from the reference policy by an amount tied to its own reward, instead of by one sequence-level constant.
  • The framework contains DPO as the special case $f_w = f_l = 1$, and it also recovers SimPO, R-DPO, and D2PO under other choices of $f$.
  • TGDPO retains good benchmark performance when trained to loss convergence, whereas DPO and SimPO degrade, which suggests less hyperparameter tuning is needed.
  • The value of $\alpha$ controls convergence speed with comparable final performance, so training can be stopped early without much cost.
  • Using token rewards produced by DPO models trained with very different $\beta$ values gives nearly the same final performance, indicating robustness to reward quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is to feed the same loss with rewards from an external fine-grained reward model or process-supervision signal, since the derivation only requires a positive function of any token-level reward.
  • The relaxation that replaces the on-policy prefix distribution with the data distribution suggests TGDPO may be optimizing an upper-bound proxy of the sequence-level objective; comparing it against on-policy token-level PPO would measure how much the relaxation costs.
  • The assumption that one over-parameterized network realizes every per-token optimal policy simultaneously could be probed by running TGDPO on small models, where the benefit over DPO should shrink if that assumption carries the argument.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes TGDPO, a direct preference optimization variant that weights each token-level log-ratio term in the DPO loss by a function f(r_hat) of a token-level reward, with the practical choice f_w = 1 + alpha r_hat on win tokens and f_l = 1 - alpha r_hat on lose tokens. The derivation starts from a sequence-level PPO objective with token-level rewards, decomposes it into token-level PPO problems, incorporates the reward guidance, obtains a closed-form optimal token-level policy and an associated token-level reward, and then plugs this reward into the Bradley-Terry model. After eliminating the partition function via the new Theorem 4.4, the authors arrive at the loss in Eq. (20). Experiments on AlpacaEval 2, MT-Bench, and Arena-Hard across three base models report win-rate improvements over DPO and SimPO, with gains up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard, and additional analyses of convergence speed, robustness, and SFT initialization.

Significance. If the derivation were valid, the framework would provide a principled way to inject dense token-level reward signals into DPO while recovering several existing losses as special cases. The empirical direction is potentially useful: the token-level weighting is simple, the paper releases code, includes an additional TDPO baseline, and reports an SFT-initialization experiment, all of which are strengths. The convergence analysis in Section 5.3 is also a genuinely interesting observation. However, the central theoretical claim is not currently established: the partition-function elimination is applied at the wrong level, the relaxation in Eq. (9) changes the objective, and the single-policy representational assumption is asserted rather than proved. In addition, the experimental support is weaker than claimed because there are no error bars, one table cell actually shows a decrease relative to DPO, and hyperparameters are grid-searched per setting without a stated validation split. The core idea may be salvageable as a practical heuristic, but the paper in its present form overclaims its theoretical grounding.

major comments (4)
  1. [Section 4.3, Theorem 4.4, Eq. (20)] The partition-function elimination is proved only for a fixed instance, but the loss in Eq. (20) is an expectation over the dataset. For a fixed (x, yw, yl), delta is independent of theta and the sigmoid is strictly increasing, so the per-instance maximizers of sigma(phi + delta) and sigma(phi) coincide. For the dataset objective, however, the gradient of -E[log sigma(phi + delta)] is E[(1 - sigma(phi + delta)) grad_theta phi], while the gradient of -E[log sigma(phi)] is E[(1 - sigma(phi)) grad_theta phi]; the instance-dependent ratio (1 - sigma(phi + delta))/(1 - sigma(phi)) is not constant, so the two objectives generally have different stationary points and different global optima. As a concrete counterexample, two instances with phi_1 = theta, phi_2 = -theta, delta_1 = 5, delta_2 = -5 give a full negative log-likelihood -log sigma(theta + 5) - log sigma(-theta - 5) minimized at theta = -5, while the TGDPO loss -log sigma(theta) - log sigma(-theta) is minimized at theta = 0. Thus Eq. (20) is not the maximum-likelihood loss of the Bradley-Terry model in Eq. (17), and the claim that TGDPO is a derived, computable DPO loss for that model is not established.
  2. [Section 4.2, Eq. (9)] The relaxation of s_t ~ D_t to s_t ~ D replaces the policy-dependent token-state distribution with a fixed data distribution. The closed-form optimal policy in Theorem 4.3 and the reward representation in Eq. (14) solve only the relaxed problem, and the paper provides no bound or correction term relating the relaxed objective to the original sequence-level PPO objective in Eq. (2). Because the state distribution generated by the policy changes during training, the loss in Eq. (20) is not shown to optimize the RLHF objective that motivates it.
  3. [Section 4.2, paragraph before Eq. (14)] The 'without loss of generality' claim that a single over-parameterized theta realizes the per-token optimal policies pi_{theta_t} for all positions simultaneously is a substantive representational assumption, not a consequence of over-parameterization. Training a single policy with the loss in Eq. (20) does not guarantee that each per-token optimal policy is individually realized, so the identification of the ground-truth reward with a single theta in Eq. (14) is not established.
  4. [Section 5.2, Table 1] The experimental evidence for the central empirical claim is weaker than the text suggests. Table 1 reports no error bars or significance tests, and in the Llama3-8B-Instruct ArmoRM row TGDPO scores 7.9 on MT-Bench versus DPO's 8.2 and a 45.0% win rate versus DPO's 46.3%, which contradicts the statement that TGDPO 'consistently outperforms baseline methods.' Additionally, hyperparameters are grid-searched per setting in Section D.1 with no stated validation split, so the reported gains may reflect selection over configurations.
minor comments (3)
  1. [Section 4.2] The item labeled 'Assumption 4.2' is later referred to as 'Theorem 4.2' in the text; the numbering and terminology should be made consistent.
  2. [Section C.2] In the recovery of SimPO, the text defines f_l(r_hat) = 1/|y_w| but the following expression uses 1/|y_l|; one of these is a typo and should be corrected.
  3. [Tables 1, 5-10] The tables are typeset too compactly, especially the header rows and the MT-Bench score/win-rate columns; please reformat for readability and use consistent spacing in numeric entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TGDPO's loss is constructed from an explicitly modified token-level PPO with externally trained reward inputs, and the reported benchmark gains are empirical outcomes rather than consequences of the construction.

full rationale

The derivation chain is self-contained rather than circular. TGDPO's loss in Eq. (20) is built from an explicit modified token-level PPO (Eq. 11), whose closed-form optimal policy yields a token-level reward representation (Eqs. 12--15); the Bradley-Terry model then gives Eq. (17), and Theorem 4.4 is used to drop the partition terms in constructing a computable loss. The practical reward weights f_w = 1 + alpha*r_hat and f_l = 1 - alpha*r_hat (Eq. 21) come from an externally trained DPO model's implicit token-level reward, not from the MT-Bench, AlpacaEval 2, or Arena-Hard scores, and Table 4 ablates the source of r_hat; therefore the win-rate gains of 7.5, 6.2, and 4.3 points are external empirical outcomes, not outputs forced by the construction. The framework's recovery of DPO, SimPO, R-DPO, and D2PO is explicitly presented as a demonstration of reasonableness, not as a derivation of those methods. There are no load-bearing self-citations by this paper's authors; the cited DPO and token-level reward results are external prior work. The paper does make openly acknowledged assumptions, including the relaxation s_t ~ D_t to s_t ~ D in Eq. (9), the 'without loss of generality' over-parameterization in Section 4.2, and the redefinition of the preference function after Theorem 4.4. The dataset-level application of Theorem 4.4 is a mathematical validity concern, because per-instance monotonicity in a theta-independent shift need not preserve the optimum of an expected log-likelihood, but this is a derivation flaw rather than circularity: Eq. (20) is not identical to Eq. (17) by construction, and no fitted parameter is renamed as a prediction. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central derivation rests on a small set of standard RLHF assumptions plus three paper-specific choices: the state-distribution relaxation, the single-parameterization realizability assumption, and the linear shaping functions for reward guidance. No new physical entities are introduced; the method introduces a reward-weighting function rather than a new force, particle, or conserved quantity.

free parameters (4)
  • alpha (guidance strength) = 0.2, 0.5, 1.2, 2.0 depending on setting (Table 10)
    Controls how strongly token-level reward weights deviate from 1; grid-searched per model and data setting.
  • beta (loss temperature) = 0.01 or 0.1 in TGDPO runs (Table 10)
    Standard DPO temperature, grid-searched; separate from the beta used to compute r_hat.
  • beta for r_hat = 0.01, 0.1, or 2.5 (Table 10)
    Temperature of the pre-trained DPO model that generates the token-level reward r_hat; affects the magnitude of r_hat.
  • shaping function form = f_w = 1 + alpha r_hat, f_l = 1 - alpha r_hat
    The linear choice for f is an ad hoc design decision; other positive functions are possible within the framework.
assumptions (6)
  • domain assumption State distribution relaxation: s_t ~ D_t is replaced by s_t ~ D (Equation 9)
    Replaces the on-policy token state distribution with a fixed data distribution, making the per-token optimization solvable but introducing an unquantified error.
  • domain assumption Single policy parameterization realizes per-token optimal policies
    Section 4.2 assumes 'without loss of generality' a single theta with pi_theta(a_t|s_t) = pi_theta_t(a_t|s_t) for all t, relying on over-parameterization.
  • domain assumption Existence of a dense token-level reward model r_hat with positive shaping f
    Assumption 4.2 requires a token-level reward and positive f; the practical f may violate positivity for large alpha.
  • standard math Bradley-Terry preference model
    Assumed human preference model, standard in DPO and RLHF.
  • domain assumption LLM text generation modeled as a token-level MDP
    Standard RLHF formulation; used to decompose sequence reward into token rewards.
  • domain assumption KL-constrained RL objective (Equation 1) as the alignment target
    Standard RLHF setup inherited from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization." pith.science (2026). https://pith.science/paper/L5HEIOCS

@misc{pith2026250614574,
  author       = {Pith},
  title        = {Pith review of: TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L5HEIOCS}},
  note         = {Machine review of arXiv:2506.14574}
}
read the original abstract

Recent advancements in reinforcement learning from human feedback have shown that utilizing fine-grained token-level reward models can substantially enhance the performance of Proximal Policy Optimization (PPO) in aligning large language models. However, it is challenging to leverage such token-level reward as guidance for Direct Preference Optimization (DPO), since DPO is formulated as a sequence-level bandit problem. To address this challenge, this work decomposes the sequence-level PPO into a sequence of token-level proximal policy optimization problems and then frames the problem of token-level PPO with token-level reward guidance, from which closed-form optimal token-level policy and the corresponding token-level reward can be derived. Using the obtained reward and Bradley-Terry model, this work establishes a framework of computable loss functions with token-level reward guidance for DPO, and proposes a practical reward guidance based on the induced DPO reward. This formulation enables different tokens to exhibit varying degrees of deviation from reference policy based on their respective rewards. Experiment results demonstrate that our method achieves substantial performance improvements over DPO, with win rate gains of up to 7.5 points on MT-Bench, 6.2 points on AlpacaEval 2, and 4.3 points on Arena-Hard. Code is available at https://github.com/dvlab-research/TGDPO.

Figures

Figures reproduced from arXiv: 2506.14574 by the authors.

Figure 1
Figure 1. Training loss curve for DPO and our TGDPO with dif￾ferent values of α. Changing the value of α leads to different convergence speeds for our method. imizing the loss for many preference optimization methods often results in degenerate policies. This issue necessitates extensive hyperparameter tuning to identify a sweet spot be￾tween the initialization and convergence points, significantly limiting the practicality a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

    cs.CL 2026-08 reject novelty 5.0 of 10

    Se-DPO assigns evolving per-token credits from implicit reward magnitude and reference entropy during DPO training, improving instruction-following win rates but with an internally inconsistent derivation.

Reference graph

Works this paper leans on

38 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hindsight experience replay

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30, 2017

    work page 2017

  2. [2]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324 -- 345, 1952

    work page 1952

  3. [3]

    On the weaknesses of reinforcement learning for neural machine translation

    Choshen, L., Fox, L., Aizenbud, Z., and Abend, O. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations, 2020

    work page 2020

  4. [4]

    Ultrafeedback: Boosting language models with scaled AI feedback

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 9722 -- 9744, 2024

    work page 2024

  5. [5]

    Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimization. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research,...

    work page 2024

  6. [6]

    The L lama 3 herd of models, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., and et al. The L lama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  7. [7]

    Direct language model alignment from online AI feedback, 2024

    Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online AI feedback, 2024. URL https://arxiv.org/abs/2402.04792

    arXiv 2024

  8. [8]

    Open RLHF : An easy-to-use, scalable and high-performance RLHF framework, 2024

    Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. Open RLHF : An easy-to-use, scalable and high-performance RLHF framework, 2024. URL https://arxiv.org/abs/2405.11143

    arXiv 2024

Show all 38 references
  1. [9]

    A., Choi, Y., and Hajishirzi, H

    Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking DPO and PPO : Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  2. [10]

    Jiang, D., Ren, X., and Lin, B. Y. LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023

    2023

  3. [11]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  4. [12]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. AlpacaEval : An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  5. [13]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  6. [14]

    Sim PO : Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [15]

    Aligning code LLM s with direct preference optimization, 2024

    Miao, Y., Gao, B., Quan, S., Lin, J., Zan, D., Liu, J., Yang, J., Liu, T., and Deng, Z. Aligning code LLM s with direct preference optimization, 2024. URL https://arxiv.org/abs/2410.18585

    2024 arXiv

  8. [16]

    GPT -4 technical report, 2024

    OpenAI, Achiam, J., Adler, S., and et al. GPT -4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  9. [17]

    F., Leike, J., and Lowe, R

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to f...

    2022

  10. [18]

    Token-level proximal policy optimization for query generation, 2024

    Ouyang, Y., Wang, L., Yang, F., Zhao, P., Huang, C., Liu, J., Pang, B., Yang, Y., Zhan, Y., Sun, H., Lin, Q., Rajmohan, S., Deng, W., Zhang, D., Sun, F., and Zhang, Q. Token-level proximal policy optimization for query generation, 2024. URL https://arxiv.org/abs/2411.00722

    2024 arXiv

  11. [19]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics: ACL, pp.\ 4998--5017, 2024

    2024

  12. [20]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pp.\ 53728--53741, 2023

    2023

  13. [21]

    From r to q^* : Your language model is secretly a q-function

    Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q^* : Your language model is secretly a q-function. In First Conference on Language Modeling, 2024

    2024

  14. [22]

    Proximal policy optimization algorithms, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  15. [23]

    Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective

    Shao, R., Li, B., Liu, G., Chen, Y., ZhouXiang, Wang, J., Cai, X., and Li, P. Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    V., Kostrikov, I., Su, Y., Yang, S., and Levine, S

    Snell, C. V., Kostrikov, I., Su, Y., Yang, S., and Levine, S. Offline RL for natural language generation with implicit language q learning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  17. [25]

    Gemini: A family of highly capable multimodal models, 2024 a

    Team, G., Anil, R., Borgeaud, S., and et al. Gemini: A family of highly capable multimodal models, 2024 a . URL https://arxiv.org/abs/2312.11805

    2024 arXiv

  18. [26]

    Gemma 2: Improving open language models at a practical size, 2024 b

    Team, G., Riviere, M., Pathak, S., and et al. Gemma 2: Improving open language models at a practical size, 2024 b . URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  19. [27]

    D., and Finn, C

    Tian, K., Mitchell, E., Yao, H., Manning, C. D., and Finn, C. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024

    2024

  20. [28]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of EMNLP, 2024

    2024

  21. [29]

    A., Ostendorf, M., and Hajishirzi, H

    Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine-grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  22. [30]

    Inverse- Q *: Token level reinforcement learning for aligning large language models without preference data

    Xia, H., Gao, S., Ge, Q., Xi, Z., Zhang, Q., and Huang, X. Inverse- Q *: Token level reinforcement learning for aligning large language models without preference data. In Findings of the Association for Computational Linguistics: EMNLP, pp.\ 8178--8188, 2024

    2024

  23. [31]

    Preference-grounded token-level guidance for language model fine-tuning

    Yang, S., Zhang, S., Xia, C., Feng, Y., Xiong, C., and Zhou, M. Preference-grounded token-level guidance for language model fine-tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  24. [32]

    A dense reward view on aligning text-to-image diffusion with preference

    Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-to-image diffusion with preference. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 55998--56032, 2024

    2024

  25. [33]

    Segmenting text and learning their rewards for improved RLHF in language model, 2025

    Yin, Y., Yang, S., Xie, Y., Yang, Z., Sun, Y., Awadalla, H., Chen, W., and Zhou, M. Segmenting text and learning their rewards for improved RLHF in language model, 2025. URL https://arxiv.org/abs/2501.02790

    2025 arXiv

  26. [34]

    Token-level direct preference optimization

    Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  27. [35]

    Judging LLM -as-a-judge with MT-Bench and Chatbot Arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM -as-a-judge with MT-Bench and Chatbot Arena . In NeurIPS Datasets and Benchmarks Track, 2023

    2023

  28. [36]

    DPO meets PPO : Reinforced token optimization for RLHF

    Zhong, H., Feng, G., Xiong, W., Cheng, X., Zhao, L., He, D., Bian, J., and Wang, L. DPO meets PPO : Reinforced token optimization for RLHF . In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  29. [37]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

    2020 arXiv

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.