Pith. sign in

REVIEW 3 major objections 4 minor 43 references

A history-aggregated objective that collapses each rollout to a cumulative log-likelihood and normalizes it with a Plackett-Luce softmax is claimed to make policy optimization faster, more exploratory, and more memory-efficient than PPO, GR

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HAEPO weights each trajectory by its softmax-normalized cumulative log-likelihood, adds entropy and KL penalties, and matches or slightly surpasses PPO, GRPO, and DPO on small RL and summarization tasks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The implemented HAEPO update omits the sampling-distribution term the paper's own appendix proves necessary, so the unbiasedness claim doesn't hold for the algorithm actually run; the paper is still a clear, honest, and partially useful contribution with fixable flaws. the 3 major comments →

arxiv 2508.18884 v1 pith:DUXT2OPL submitted 2025-08-26 cs.LG cs.AI

HAEPO: History-Aggregated Exploratory Policy Optimization

Gaurish Trivedi , Alakh Sharma , Kartikey Singh Bhandari , Dhruv Kumar , Pratik Narang , Jagat Sesh Challa This is my paper
classification cs.LG cs.AI
keywords reinforcement learningpolicy optimizationPlackett-Luce weightingtrajectory-level credit assignmentexplorationLLM alignmententropy regularizationKL trust region
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HAEPO is a policy-gradient method that treats each full rollout as one ranked item: it sums the log-probabilities of all actions in an episode, runs a Plackett-Luce softmax over the batch to get a normalized weight per trajectory, and multiplies those weights by normalized returns to form the update. The paper argues that this trajectory-level listwise weighting preserves distinctions among sequences that per-token aggregation dilutes, and that adding an entropy bonus plus a soft KL penalty to a frozen reference policy keeps the update from collapsing into premature exploitation. Empirically, the authors report faster convergence and lower variance on bandit and random-walk tasks, competitive sample efficiency with smoother curves on CartPole, and higher human preference ratings than GRPO or DPO on TL;DR summarization while using 26.4 percent less peak GPU memory than GRPO. If these results hold, HAEPO offers a simpler, critic-free and memory-frugal alternative to PPO-style and group-relative methods for long-horizon RL and LLM alignment.

Core claim

The central claim is that an objective of the form L(θ) = −Σ_k w_k R̃_k + β Σ_k w_k log w_k + λ Σ_k w_k (log w_k − log w_k^ref), with w_k the Plackett-Luce weight over cumulative log-likelihoods, is a stable, low-variance estimator of expected return that yields better exploration than per-token or pairwise losses. The paper derives the gradient as an expectation over PL weights of a centered score function, and reports that on four benchmarks — stochastic bandits, a sparse random walk, CartPole, and TL;DR human-feedback fine-tuning — HAEPO converges quickly, explores broadly, aligns with human preference better or on par with PPO, GRPO, and DPO, and cuts GPU memory relative to GRPO.

What carries the argument

The load-bearing object is the Plackett-Luce weight w_k = exp(L_k)/Σ_j exp(L_j), where L_k = Σ_t log π_θ(a_t|s_t) is the cumulative log-likelihood of rollout k. Because ∇_θ log w_k = ∇_θ L_k − Σ_j w_j ∇_θ L_j, the reward-weighted gradient automatically subtracts a batch-averaged score, giving a centered estimator without a learned baseline. The entropy term β Σ w log w and the KL penalty λ Σ w (log w − log w_ref) are applied to the weight distribution, not to individual actions, so they govern how the whole batch of trajectories spreads or shifts.

Load-bearing premise

The load-bearing premise is that ignoring the way the policy's own sampling distribution changes during an update does not bias learning; if that ignored effect matters, HAEPO is not optimizing the objective it claims to optimize.

What would settle it

On the random-walk or CartPole setup, compare training with the implemented HAEPO gradient (Eq. 6/9) against training with the full unbiased estimator from the appendix (Eq. 36), which appends −(Σ_k R_k w_k)∇_θ log p(τ;θ) to the update. If including this term changes the learning curves materially—or if its gradient norm is not small relative to the implemented gradient—the claim that HAEPO is an unbiased, low-variance estimator of expected return is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • HAEPO can be implemented as a simple loss on per-rollout log-probability sums, with no critic network and no per-token clipping; that is what keeps its per-update cost and memory footprint small.
  • On sparse long-horizon tasks such as the 500-step random walk, trajectory-level PL weighting should keep rare successful walks visible in the gradient, improving credit assignment relative to per-step policy gradients.
  • For LLM alignment, HAEPO's listwise reward weighting is a drop-in alternative to GRPO's group-relative baseline, with the reported 26.4% peak-VRAM savings on a 1.5B model.
  • The two regularization coefficients β_ent and β_kl control the exploration-stability trade-off; the paper notes these require tuning per task and reward scale.
  • Because HAEPO stores one log-probability sum per trajectory and runs a softmax over the batch, its overhead grows linearly with batch size M; the paper lists this as a computational limitation for very large batches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The implemented gradient (Eq. 6) drops the sampling-distribution term that the appendix's unbiased estimator (Eq. 36) includes; a head-to-head comparison of the two updates on a long-horizon task would reveal whether the omitted term is negligible or whether HAEPO is optimizing a different objective.
  • Editorial inference: HAEPO's return-weighted PL normalization is a natural fit for listwise preference datasets with graded or full ranking feedback; the paper only demonstrates it on scalar-return environments, not on explicit preference rankings.
  • Editorial inference: The reported VRAM reduction suggests HAEPO could enable longer contexts or larger batches on a fixed GPU than GRPO, but that extrapolates beyond the tested 1B/1.5B models and 1,000-update schedule.
  • Editorial inference: The paper's single-agent limitation points to an open question of whether PL weighting over joint trajectories can be stabilized in multi-agent settings, since the reference-policy KL would then need to account for opponents' changing policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes HAEPO, a trajectory-level policy optimization method. Each rollout is compressed to the sum of its action log-probabilities, a Plackett-Luce softmax over a batch converts these into normalized weights, and the loss is the negative of the return-weighted sum of these weights, augmented by an entropy bonus and a KL penalty to a frozen reference policy. The method is evaluated on multi-armed bandits, a 500-step random walk, CartPole, and TL;DR summarization with LLMs, comparing against PPO, DPO, and GRPO. The main text derives the gradient of the sample loss, claims unbiasedness and variance reduction, and presents empirical evidence of faster or comparable convergence and lower GPU memory use.

Significance. If the theoretical claims were correct, HAEPO would be a simple, low-memory trajectory-level alternative to GRPO/PPO, with a clear listwise interpretation. The paper includes several strengths: the method is simple and interpretable, the appendix contains an unusually explicit derivation and a finite-difference check, and the experiments span both small RL environments and an LLM alignment task with human evaluation. However, the central theoretical claim — that the implemented update is an unbiased, low-variance estimator of the expected return objective — is contradicted by the paper's own appendix. The implemented gradient (Eq. 6) omits the sampling-distribution correction that the unbiasedness proof (Eq. 36) shows is necessary. Consequently the contribution, as presented, is a heuristic trajectory-level loss on a fixed batch, not an unbiased policy-gradient method. The empirical results may still be useful, but the paper's stated theoretical foundation and 'provably lowers variance' claims need substantial revision or removal.

major comments (3)
  1. [§3.4, Eq. (6) and Appendix, Eq. (35)–(36)] The gradient implemented in Eq. (6) is the gradient of the sample loss -Σ_k R_k w_k with respect to θ while holding the sampled trajectories fixed. The appendix correctly derives the gradient of the expectation E_{τ~p(τ;θ)}[-Σ_k R_k w_k] and obtains Eq. (35), whose Monte Carlo estimator in Eq. (36) includes the additional term - (Σ_k R_k w_k) ∇_θ log p(τ;θ). This term is generally nonzero. Therefore the update actually run in the experiments is not an unbiased estimator of the gradient of the objective defined in Eq. (34). The unbiasedness proof and the low-variance claims in §3.5 do not apply to the algorithm as implemented. Please either include the missing sampling-distribution term and rerun the experiments, or explicitly reframe HAEPO as a deterministic sample-loss minimization heuristic and provide a different justification for why it improves expected return.
  2. [§3.5, Z-Score Normalization] The claim that z-score normalization 'provably lowers variance' is not supported by any proof in the main text or appendix. Moreover, the statement that the transformation 'leaves the expectation unchanged so the gradient remains unbiased' is incorrect for the full z-score: replacing R_k by (R_k - μ)/σ multiplies the entire gradient by 1/σ, which changes the scale of the expected gradient and is not equivalent to subtracting a constant baseline. Subtracting a constant b from all returns in a batch does leave the gradient unchanged because Σ_k w_k ∇_θ log w_k = 0, but dividing by the sample standard deviation does not. Please provide a precise statement with a proof, or remove the 'provably' and 'unbiased' claims for this normalization.
  3. [§3.8, Eqs. (10)–(11) vs. Appendix Eq. (30)] The gradient decomposition in Eq. (10) writes the entropy contribution as β E_{k~w}[∇_θ log w_k]. Differentiating the entropy term β Σ_k w_k log w_k from Eq. (7), however, gives β Σ_k w_k (1 + log w_k) ∇_θ log w_k, which is the form correctly used in Eq. (30). Eq. (11)'s collapsed form -E[(R̃_k - β - λD_k)∇_θ log w_k] also omits the (1 + log w_k) factor. The main-text derivation and the appendix are therefore inconsistent. Please correct the main-text equations so that the reported gradient matches the defined loss.
minor comments (4)
  1. [§4.4, TL;DR setup and Table 1] The setup says each method is run with three random seeds, but Table 1 and the text report results over five seeds. Please align these numbers. Also, 'a 50-50 split of 1,000 supervised examples and 1,000 RL updates' is ambiguous; clarify the dataset size and update count.
  2. [§3.7, Eq. (8)–(9)] The KL penalty is applied to the Plackett-Luce weight distributions w and w_ref, not to the policy distributions. This is a batch-level, trajectory-weight trust region, not a policy trust region. Please state this explicitly to avoid overclaiming the connection to PPO-style KL penalties.
  3. [§3.5, Sum-Normalization] For sum normalization, if the batch sum of returns is negative, the normalized returns ˜R_k can be negative and do not sum to +1 in the usual sense. The claim that the rescaled rewards satisfy Σ_k ˜R_k = 1 assumes a positive sum. Please clarify the handling of negative or zero-sum batches.
  4. [Appendix, Code Sanity Check] The finite-difference check verifies gradients of the loss with respect to the inputs newlp, oldlp, and returns, not with respect to policy parameters θ. This does not validate the unbiasedness of the estimator in Eq. (36). Please clarify what the check actually demonstrates.

Circularity Check

0 steps flagged

No circularity: HAEPO's loss and gradients are derived from the defined objective; the omitted sampling-distribution term is an internal consistency gap, not a circular dependency.

full rationale

The derivation chain is self-contained and non-circular. Eq. (1)-(3) define the trajectory log-likelihood L_k, the Plackett-Luce weight w_k, and the original loss L_orig = -Σ R_k w_k. Eq. (4)-(6) then differentiate this loss via the chain rule, producing the centered score-function gradient. Eq. (9) explicitly adds entropy and KL-penalty terms, and Eq. (10)-(11) differentiate those terms. No parameter is fitted to a target result, no prediction is statistically forced by an earlier fit, and no load-bearing self-citation or imported uniqueness theorem supplies the method. The paper does not redefine a known result under new coordinates; it proposes a new loss and computes its gradient. Flagged for correctness rather than circularity: the main-text gradient in Eq. (6), ∇θL = -Σ R_k w_k [∇θL_k - Σ_j w_j ∇θL_j], is the gradient of the sample loss holding trajectories fixed. The appendix's unbiasedness proof (Eqs. 33-37) correctly shows that the unbiased estimator of the expected loss must include the additional sampling-distribution term - (Σ R_k w_k) ∇θ log p(τ;θ), as in Eq. (36). The implemented algorithm appears to omit this term, so the unbiasedness proof in the appendix does not cover the exact gradient used in the experiments. This is a substantive internal-consistency concern about the 'unbiased, low-variance' claims, but it is not an example of circularity: the derivation does not reduce to its own inputs by construction. It belongs in a correctness or reproducibility pass rather than raising the circularity score. Since the paper is otherwise self-contained and makes no fitted-parameter-renamed-as-prediction move, the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method introduces one discrete modeling choice (reward normalization scheme) and two continuous hyperparameters (β_ent, β_kl). The math relies on standard probability and calculus identities. The main additional assumption is that the fixed-batch gradient (without the sampling-distribution correction) is a valid update, which the appendix does not actually prove.

free parameters (4)
  • β_ent (entropy weight) = 5e-2 (bandit), 5e-5 (random walk), 1e-1 (CartPole), 1e-2 (TL;DR)
    Hand-tuned per task; the authors acknowledge in §5 that optimal settings vary substantially across tasks and reward scales.
  • β_kl (KL penalty weight) = Same values as β_ent
    Hand-tuned per task, paired with β_ent; the paper provides no automated schedule.
  • Reward normalization scheme = Sum-norm vs z-score, selected per task
    The choice is made per task based on the authors' ablations, effectively a discrete free parameter selected after seeing results.
  • Gradient-norm clip = 0.5 only in CartPole experiments
    Applied only in CartPole, not used in other experiments.
axioms (4)
  • standard math Score-function identity: ∇θ p(τ) = p(τ) ∇θ log p(τ; θ)
    Used in the appendix's unbiasedness derivation; standard result cited to Glynn 1990 / Williams 1992.
  • standard math Softmax weights w_k sum to 1 and Σ w_k ∇ log w_k = 0
    Used to simplify the entropy and KL gradients in Appendix Eq. (29); standard property.
  • ad hoc to paper The implemented loss is differentiated treating the batch of M trajectories as fixed data
    The experiments use the gradient of Eq. (6), which omits the ∇θ log p correction term; the unbiasedness proof in the appendix requires that term. This is the load-bearing assumption flagged in weakest_assumption.
  • domain assumption Trajectories in a batch are independent and identically distributed from πθ
    Required by the joint-density factorization in Appendix Eq. (33); standard in RL but not true when using a replay buffer or when episodes are truncated.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HAEPO: History-Aggregated Exploratory Policy Optimization." pith.science (2026). https://pith.science/paper/DUXT2OPL

@misc{pith2026250818884,
  author       = {Pith},
  title        = {Pith review of: HAEPO: History-Aggregated Exploratory Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUXT2OPL}},
  note         = {Machine review of arXiv:2508.18884}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Exploration is essential in modern learning, from reinforcement learning environments with small neural policies to large language models (LLMs). Existing work, such as DPO, leverages full sequence log-likelihoods to capture an entire trajectory of the model's decisions, while methods like GRPO aggregate per-token ratios into a trajectory-level update. However, both often limit exploration on long-horizon tasks. We introduce History-Aggregated Exploratory Policy Optimization (HAEPO), a history-aware exploratory loss to combat these shortcomings. HAEPO compresses each trajectory into the sum of its logarithmic probabilities (a cumulative logarithmic likelihood), and applies a Plackett-Luce softmax across trajectories to obtain normalized weights proportional to their returns, thus encouraging broader exploration. We add entropy regularization to stabilize the aggressive updates to prevent premature collapse and a soft KL penalty relative to a frozen copy of the previous (reference) policy. Empirically, HAEPO converges fast, explores thoroughly, aligns closely with true rewards, and demonstrates robust learning behavior better or at par with PPO, GRPO, and DPO across diverse tasks. Thus, HAEPO provides a stable and interpretable framework by explicitly leveraging full-trajectory history while balancing exploration and stability.

Figures

Figures reproduced from arXiv: 2508.18884 by Alakh Sharma, Dhruv Kumar, Gaurish Trivedi, Jagat Sesh Challa, Kartikey Singh Bhandari, Pratik Narang.

Figure 1
Figure 1. Figure 1: Loss-function workflow for History-Aggregated Exploratory Policy Optimization (HAEPO) (§3). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Multi-armed Bandit Environment: Mean per-step regret over [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Random Walk Environment: Mean return over 100 updates for state sizes 10 (left) and 20 (right). Legend [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Wall-clock learning curves on the CartPole-v1 benchmark, showing the average return (mean ± 1 std across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: HAEPO on the Deterministic Chain MDP: sum-norm vs. z-score. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: HAEPO on the Single-Period Newsvendor: sum-norm vs. z–score. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Human preference questionnaire for one seed, showing the original Reddit posts and the six anonymized [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 25 canonical work pages · 3 internal anchors

  1. [1]

    Tuning continual exploration in reinforcement learning: An optimality property of the boltzmann strategy

    Youssef Achbany, François Fouss, Luh Yen, Alain Pirotte, and Marco Saerens. Tuning continual exploration in reinforcement learning: An optimality property of the boltzmann strategy. Neurocomputing, 71: 0 2507--2520, 2008. URL https://api.semanticscholar.org/CorpusID:10215165

    work page 2008

  2. [2]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22--31. PMLR, 2017

    work page 2017

  3. [3]

    Understanding the impact of entropy on policy optimization

    Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pages 151--160. PMLR, 2019

    work page 2019

  4. [4]

    Arrow, Theodore Edward Harris, and Jacob Marschak

    Kenneth J. Arrow, Theodore Edward Harris, and Jacob Marschak. Optimal inventory policy. Econometrica, 19: 0 133, 1951. URL https://api.semanticscholar.org/CorpusID:51766626

    work page 1951

  5. [5]

    Finite-time analysis of the multiarmed bandit problem

    Peter Auer, Nicol \`o Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47: 0 235--256, 2002. URL https://api.semanticscholar.org/CorpusID:207609497

    work page 2002

  6. [6]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Barto, Richard S

    Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13 0 (5): 0 834--846, 1983. doi:10.1109/TSMC.1983.6313077

    work page arXiv 1983

  8. [8]

    Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos

    Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count‐based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems 29, pages 1471--1479, 2016

    work page 2016

  9. [9]

    Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Gr'egoire Del'etang, Markus Kunesch, Shane Legg, and Pedro A. Ortega. Your policy regularizer is secretly an adversary. ArXiv, abs/2203.12592, 2022. URL https://api.semanticscholar.org/CorpusID:247618673

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  10. [10]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    2016

  11. [11]

    A survey on policy search for robotics

    Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends in Robotics , 2 0 (1--2): 0 1--142, 2013

    work page 2013

  12. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv--2407, 2024

    2024

  13. [13]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. URL https://arxiv.org/abs/2402.01306

    Pith/arXiv arXiv 2024

  14. [14]

    Peter W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Commun. ACM, 33: 0 75--84, 1990. URL https://api.semanticscholar.org/CorpusID:8473536

    work page 1990

  15. [15]

    Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration

    Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International Conference on Machine Learning, 2020. URL https://api.semanticscholar.org/CorpusID:219179219

    work page 2020

  16. [16]

    Exploration in deep reinforcement learning: From single-agent to multiagent domain

    Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 35 0 (7): 0 8762--8782, 2024. doi:10.1109/TNNLS.2023.3236361

    work page arXiv 2024

  17. [17]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024. URL https://arxiv.org/abs/2403.07691

    Pith/arXiv arXiv 2024

  18. [18]

    Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

    Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Learning dynamics model in reinforcement learning by incorporating the long term future. ArXiv, abs/1903.01599, 2019. URL https://api.semanticscholar.org/CorpusID:67877018

    work page internal anchor Pith review Pith/arXiv arXiv 1903

  19. [19]

    Variational policy search via trajectory optimization

    Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. In Neural Information Processing Systems, 2013. URL https://api.semanticscholar.org/CorpusID:5941161

    work page 2013

  20. [20]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  21. [21]

    Liu, and Xuanhui Wang

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J. Liu, and Xuanhui Wang. Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878, 2024. URL https://arxiv.org/abs/2402.01878

    Pith/arXiv arXiv 2024

  22. [22]

    Improved exploration through latent trajectory optimization in deep deterministic policy gradient

    Kevin Sebastian Luck, Mel Vecer \'i k, Simon Stepputtis, Heni Ben Amor, and Jonathan Scholz. Improved exploration through latent trajectory optimization in deep deterministic policy gradient. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3704--3711, 2019. URL https://api.semanticscholar.org/CorpusID:208139347

    work page 2019

  23. [23]

    Exploration via Planning for Information about the Optimal Trajectory

    Viraj Mehta, Ian Char, Joseph Abbate, Rory Conlin, Mark D. Boyer, Stefano Ermon, Jeff G. Schneider, and Willie Neiswanger. Exploration via planning for information about the optimal trajectory. Advances in Neural Information Processing Systems, abs/2210.04642, 2022. URL https://api.semanticscholar.org/CorpusID:252780176

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  24. [24]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta AI . Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, September 2024. Accessed: 2025-07-31

    work page 2024

  25. [25]

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292--6299, 2017. URL https://api.semanticscholar.org/CorpusID:3543784

    work page 2018

  26. [26]

    Projection-based constrained policy optimization

    Karthik Narasimhan. Projection-based constrained policy optimization. ArXiv, abs/2010.03152, 2020. URL https://api.semanticscholar.org/CorpusID:210903109

    Pith/arXiv arXiv 2010

  27. [27]

    G \'o mez

    Gergely Neu, Anders Jonsson, and V. G \'o mez. A unified view of entropy-regularized markov decision processes. ArXiv, abs/1705.07798, 2017. URL https://api.semanticscholar.org/CorpusID:19318639

    Pith/arXiv arXiv 2017

  28. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv.org/abs/2203.02155

    Pith/arXiv arXiv 2022

  29. [29]

    Maximum entropy gain exploration for long horizon multi-goal reinforcement learning

    Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International conference on machine learning, pages 7750--7761. PMLR, 2020

    work page 2020

  30. [30]

    Plackett

    Robin L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 24 0 (2): 0 193--202, 1975

    work page 1975

  31. [31]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023 a . URL https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2023

  32. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023 b

    work page 2023

  33. [33]

    Some aspects of the sequential design of experiments

    Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58 0 (5): 0 527--535, 1952

    work page 1952

  34. [34]

    Jordan, and Pieter Abbeel

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. URL https://arxiv.org/abs/1506.02438

    Pith/arXiv arXiv 2015

  35. [35]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017. URL https://api.semanticscholar.org/CorpusID:28695052

    Pith/arXiv arXiv 2017

  36. [36]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  37. [37]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.17492, 2024. URL https://arxiv.org/abs/2306.17492

    Pith/arXiv arXiv 2024

  38. [38]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018. ISBN 9780262039246. URL http://incompleteideas.net/book/the-book-2nd.html

    work page 2018

  39. [39]

    Deep reinforcement learning for robotics: A survey of real-world successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart'in-Mart'in, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. In AAAI Conference on Artificial Intelligence, 2024. URL https://api.semanticscholar.org/CorpusID:271745346

    work page 2024

  40. [40]

    TL ; DR : Mining R eddit to learn automatic summarization

    Michael V \"o lske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL ; DR : Mining R eddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization, pages 59--63, Copenhagen, Denmark, September 2017. Association for Computational Linguistic...

    doi:10.18653/v1/w17-4508 2017

  41. [41]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8 0 (3-4): 0 229--256, 1992

    1992

  42. [42]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Penalized proximal policy optimization for safe reinforcement learning

    Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In International Joint Conference on Artificial Intelligence, 2022. URL https://api.semanticscholar.org/CorpusID:249017615

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.