Pith. sign in

REVIEW 3 major objections 4 minor 13 references

Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that two simple, RL-native objectives—advantage-weighted regression and policy gradient—make generative policies match or beat prior state-of-the-art results on offline reinforcement learning benchmarks.

desk verdict Worth sending to review for the GMPO recipe and framework, but the SOTA claim is undercut by under-tuned re-implemented baselines and the GMPO derivation has a gap. read the letter →

arxiv 2412.01245 v1 pith:53VKQEIZ submitted 2024-12-02 cs.LG cs.AI

Jinouwen Zhang , Rongkun Xue , Yazhe Niu , Yun Chen , Jing Yang , Hongsheng Li , Yu Liu This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearninggenerativepoliciesdiffusionmodelsflowmatchingadvantage-weightedregressionpolicygradientimplicitQ-learningD4RL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that prior generative-policy algorithms in offline reinforcement learning have accumulated unnecessary complexity—behavior-policy pretraining, resampling, and model-specific tricks—and that two simple, RL-native objectives suffice. The first, GMPO, is an advantage-weighted regression objective that uses the generative model's matching loss (score matching or flow matching) in place of a log-likelihood term, and it works even when the policy is initialized from scratch with no behavior-policy pretraining and no resampling. The second, GMPG, is a numerically stable, native policy-gradient objective for continuous-time generative models built on Neural ODE gradients and Hutchinson trace estimation of log-likelihood. On D4RL and RL Unplugged benchmarks, the authors report state-of-the-art or comparable performance for both schemes across diffusion and flow models, which would make a single unified recipe for training generative policies if the claims hold.

What carries the argument

The machinery is the closed-form optimal policy of constrained offline RL, π*(a|s) ∝ $e^{{β(Q(s,a)−V(s))}}$ μ(a|s), and its two KL divergence training objectives. Forward-KL (advantage-weighted regression) yields GMPO after replacing the log-likelihood log π_θ(a|s) with the generative model's matching loss L_Matching(θ) (denoising score matching or conditional flow matching), so any generative model with a matching objective can be trained as a policy. Reverse-KL (policy gradient) yields GMPG, which computes the policy log-likelihood through the instantaneous change-of-variables formula integrated by a Neural ODE, with the Jacobian trace estimated by Hutchinson's estimator. The crucial load-bearing identity is the equivalence between maximum-likelihood training and score matching for diffusion models, cited from Song et al. (2021a), which the paper extends—without proof—to the advantage-weighted variant.

What would settle it

On a task like halfcheetah-medium-v2, train one policy with GMPO (advance-weighted matching loss) and another with the same architecture but an explicit advantage-weighted log-likelihood objective computed through a Neural ODE, keeping the Q-function, β, and step budget identical; if GMPO's returns cannot be matched or exceeded by the explicit-likelihood version, the substitution in Eq. 6 is not preserving the intended optimum.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that the advantage-weighted regression objective used by prior forward-KL methods can be reduced to a clean matching-loss objective—GMPO—and that this objective extracts near-optimal policies directly from an IQL-trained Q-function without needing a pretrained behavior policy or inference-time resampling. The paper states explicitly that a generative policy trained with advantage-weighted regression from a scratch initialization gains performance comparable to IDQL's resampling-based scheme, overturning the earlier claim that advantage-weighted regression does not help for expressive models. The companion objective GMPG implements the reverse-KL policy gradient by simulating the generative ODE forward and back-propagating through it, giving a stable policy-gradient baseline that outperforms GMPO on several medium and medium-replay locomotion tasks. Together the two objectives are presented as a unified, model-agnostic training scheme valid for both diffusion and flow models.

Load-bearing premise

The argument leans on the assumption that the well-known equivalence between score matching and maximum likelihood survives when the matching loss is multiplied by an exponential advantage weight and evaluated only on offline dataset actions, so that GMPO's optimum equals the true advantage-weighted regression optimum.

Editorial extensions

If this is right

  • Behavior-policy pretraining and inference-time resampling are unnecessary for competitive generative policies: GMPO matches IDQL-level returns while training the policy from a scratch initialization.
  • The same objectives transfer directly across generative model families, as shown by comparable GMPO/GMPG scores with VP-SDE diffusion, GVP diffusion, and I-CFM flow models.
  • A reverse-KL policy gradient implemented with ODE simulation and trace estimation is stable enough for training (T=1000), removing the low-step-size constraint highlighted in Diffusion-QL.
  • IQL-trained Q-functions suffice for explicit generative policy extraction, decoupling critic training from policy optimization and simplifying the pipeline.
  • GMPG's aggressive Q-guidance helps in medium and medium-replay locomotion tasks but hurts in AntMaze and expert-data tasks, indicating a systematic trade-off between the two objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If GMPO's substitution is only known to hold for unweighted maximum likelihood, the advantage-weighted matching loss may select a different policy optimum; a controlled comparison against explicit advantage-weighted log-likelihood on the same Q-function would settle the gap.
  • The same template could extend to other generative families (e.g., Schrödinger bridges or normalizing flows) since only a matching loss and a sampling procedure are required, not diffusion-specific machinery.
  • In high-dimensional action spaces, the toy-example distinction between GMPO's data-filtering and GMPG's manifold-preserving paths could become empirically decisive, favoring GMPG where manifold structure matters.
  • The paper's importance-sampled GMPG variant (Eq. 42) is a natural hybrid of forward and reverse KL that the experiments do not compare; testing it against GMPO and GMPG would map the stability-performance trade-off.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper revisits generative policies for offline reinforcement learning, classifies prior methods into forward-KL and reverse-KL families, and proposes two training schemes: GMPO, an advantage-weighted regression objective using the generative matching loss, and GMPG, a policy-gradient objective with likelihoods computed through Neural ODEs. The authors introduce a framework called GenerativeRL and report experiments on D4RL locomotion, D4RL AntMaze, and RL Unplugged DeepMind Control Suite, including ablations over generative model type, sampling steps, temperature, and solver schemes. The central claims are that GMPO reaches IDQL-level performance without behavior-policy pretraining and that GMPO/GMPG offer unified, simple, and state-of-the-art or comparable generative policy training.

Significance. If the claims are established, the paper would be a useful consolidation: it identifies a minimal advantage-weighted regression formulation, provides a tractable policy-gradient implementation for continuous-time generative models, and contributes a modular framework that decouples generative models from RL components. The multi-seed evaluations, the comparison of diffusion and flow backbones, and the ablation of solver and sampling choices are commendable and go beyond many prior works. However, the theoretical grounding of GMPO is currently incomplete, and the empirical comparison is weakened by the reported gap between the paper's reimplemented baselines and the published scores of those baselines. The contribution is valuable as a heuristic and engineering study, but the paper's stronger theoretical and comparative claims need revision before publication.

major comments (3)
  1. [Section 4.3, Eq. (6), and Appendix C.6] The derivation of GMPO is not complete. Eq. (6) replaces log pi_theta(a|s) in the forward-KL AWR objective (Eq. 4) with the matching loss L_Matching(theta), and the text says this is justified by Song et al. (2021a) and that it 'can be derived independently.' Appendix C.6, Eq. (38), only rearranges the expectation over a ~ pi* into an expectation over a ~ mu with the exponential importance weight; it does not prove that the argmin of the weighted matching-loss objective equals the argmin of the weighted log-likelihood objective. The cited unweighted equivalence between maximum likelihood and score matching does not automatically survive multiplication by e^{beta(Q-V)}/Z, nor does it address restriction to the offline dataset's action support. Moreover, Appendix A.2.1, Eq. (14), states an ELBO-type upper bound, not an equality. The authors should either provide a proof that the optimum is preserved under this per-sample weighting or explicitly characterize GMPO as an empirically motivated heuristic and soften the 'native AWR' and theoretical claims accordingly.
  2. [Section 6.1, Table 3, and Table 9] The headline comparison is biased by the reimplemented baselines. Table 9 reports published averages of 86.6 for QGPO, 82.1 for IDQL, and 87.1 for SRPO, while the GenerativeRL reimplementations achieve 81.8, 78.3, and 82.8 respectively, i.e., 3-5 points lower. Table 3 uses these reimplemented values as the comparison points. Against the published baselines, GMPO's 82.7 average is only 0.6 points above IDQL and 3.9 points below QGPO, and GMPG's 87.3 average is 0.2 points above the published SRPO and 0.7 points below the published Diffusion-QL average of 88.0. The gap may indicate under-tuned baselines rather than an advantage of the proposed methods. The paper should either re-tune the reimplemented baselines to their published performance, provide the full hyperparameter settings and code for those baselines, or restrict the comparative claims to 'our reimplementation' and avoid the abstract's state-of-the-art wording.
  3. [Section 6.2] The claimed overturning of the IDQL finding about advantage-weighted regression is not supported as stated. The text says GMPO achieves 'comparably equivalent performance to IDQL using resampling tricks,' but Table 3 shows GMPO at 82.7 against the reimplemented IDQL at 78.3, while the published IDQL is 82.1. This difference is within the reported standard deviations, and the comparison also involves different generative model types (GVP for GMPO versus VPSDE for IDQL in the reimplementation). The authors should either use the published IDQL score as the reference, explain the reimplementation discrepancy, or weaken the claim to a statement about comparable performance under careful tuning.
minor comments (4)
  1. [Section 4.2] The sentence 'As shown in Table 1, all methods require pretraining a behavior policy' contradicts Table 1 and Algorithm 1, where GMPO explicitly does not require a behavior policy; the sentence should say 'all previous methods'.
  2. [Appendix C.6, Eqs. (39) and (40)] The sign convention is inconsistent: Eq. (6) defines GMPO as an expectation of the nonnegative matching loss L_Matching, but Eqs. (39) and (40) introduce a leading minus sign before the expectation. The authors should clarify whether these expressions are meant as losses to be minimized or as negative objectives, and make the notation consistent.
  3. [Section 6.1 and Appendix D.1] The paper states that baselines were reimplemented 'under the same experimental settings for fair comparisons,' but it provides no code repository link and no per-baseline hyperparameter table. Given that the proposed methods use task-specific temperature beta tuning (Table 8), the fairness argument would be materially strengthened by releasing the benchmark code and the baseline configurations.
  4. [Section 5] The claim that GenerativeRL provides a 'standardized experimental framework' is only supported by usage examples in the appendix; the paper should include a clear statement of which parts of the framework are released, with a public link, so that the claimed standardization is verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed objectives are derived from external KL/AWR formulations and independently benchmarked; the cited score-matching/MLE equivalence is external and not self-referential.

full rationale

The paper's central derivations are self-contained reductions from independent published objectives. GMPO starts from Peng et al.'s analytical optimal policy and minimizes forward KL (Eq. 4). Replacing the log-likelihood term with L_Matching cites Song et al. (2021a), an external mathematical result, and GMPO's empirical evaluation uses an IQL-trained critic from the dataset; no target result is assumed inside the loss. GMPG is the exact reverse-KL objective (Eq. 8) with the log-likelihood computed via Neural ODE and Hutchinson trace estimation, both external numerical tools. The 'scratch initialization overturns IDQL' claim is an empirical comparison, not a construction. The main weakness—the unweighted score-matching/MLE equivalence is invoked under an advantage-weighted expectation without proof—is a soundness gap, not circularity, because Song et al.'s result is logically independent of the present paper's claims. Similarly, the lower re-implemented baseline scores in Table 9 affect whether the 'comparable/state-of-the-art' claim is supported, but that is a benchmarking correctness concern, not a self-referential loop. No fitted parameter is renamed as a prediction, and no load-bearing argument reduces to the authors' own prior work.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The methods rely on standard RL optimal-policy form and IQL, plus one paper-specific assumption: that matching loss is a valid drop-in surrogate for log-likelihood inside the advantage-weighted objective. Beta per task is the main free parameter; no new entities are posited.

free parameters (2)
  • temperature coefficient beta = per-task values 1.0-20.0 for GMPO, 0.25-8.0 for GMPG (Table 8)
    Tuned per task per algorithm; controls strength of Q-guidance in both GMPO and GMPG and directly determines the advantage weights.
  • IQL expectile tau = 0.7 default, 0.9 for AntMaze
    Chosen per dataset family, affects the Q and V functions used in the advantage weights.
assumptions (4)
  • domain assumption The optimal policy has the form pi*(a|s) proportional to exp(beta Q(s,a)) mu(a|s) (Peng et al. 2021)
    Used in Section 4.1 to derive both forward and reverse KL objectives.
  • ad hoc to paper Replacing log pi_theta(a|s) with L_Matching(theta) in the weighted objective Eq. 6 preserves the optimum of the AWR objective
    The paper cites Song et al. (2021a) for the unweighted equivalence but does not prove it holds under the exponential importance weights; Appendix C.6 only rearranges the expectation.
  • domain assumption IQL provides a good enough estimate of the optimal Q and V for policy extraction
    Both algorithms use IQL-trained critics for the advantage weights and Q guidance (Section 4.3, Algorithm 1/3).
  • domain assumption The Hutchinson trace estimator gives sufficiently accurate log-likelihood gradients for GMPG training
    GMPG uses Eq. 45 for the trace of the Jacobian; the bias/variance properties under policy-gradient training are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective." pith.science (2026). https://pith.science/paper/53VKQEIZ

@misc{pith2026241201245,
  author       = {Pith},
  title        = {Pith review of: Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53VKQEIZ}},
  note         = {Machine review of arXiv:2412.01245}
}
read the original abstract

Generative models, particularly diffusion models, have achieved remarkable success in density estimation for multimodal data, drawing significant interest from the reinforcement learning (RL) community, especially in policy modeling in continuous action spaces. However, existing works exhibit significant variations in training schemes and RL optimization objectives, and some methods are only applicable to diffusion models. In this study, we compare and analyze various generative policy training and deployment techniques, identifying and validating effective designs for generative policy algorithms. Specifically, we revisit existing training objectives and classify them into two categories, each linked to a simpler approach. The first approach, Generative Model Policy Optimization (GMPO), employs a native advantage-weighted regression formulation as the training objective, which is significantly simpler than previous methods. The second approach, Generative Model Policy Gradient (GMPG), offers a numerically stable implementation of the native policy gradient method. We introduce a standardized experimental framework named GenerativeRL. Our experiments demonstrate that the proposed methods achieve state-of-the-art performance on various offline-RL datasets, offering a unified and practical guideline for training and deploying generative policies.

Figures

Figures reproduced from arXiv: 2412.01245 by the authors.

Figure 1
Figure 1. Log-Likelihood of D4RL datasets evaluated by GMPO/GMPG-GVP policies during [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. 2D toy Swiss Roll dataset with assigned value function. Values range from [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 3
Figure 3. Generation trajectories of models trained by GMPO and GMPG on the 2D toy Swiss Roll [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An example of using GenerativeRL for defining models, training, and sampling. All experiment configurations are orgnized in a nested dictionary and can be recorded for reproductions. Configuration of every component is modular and can be easily switched. Diverse genera…
Figure 5
Figure 5. Figure 5: Framework structure of GenerativeRL. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [4]

    URL https://doi.org/10.48550/arXiv

    doi: 10.48550/ARXIV .2304.10573. URL https://doi.org/10.48550/arXiv. 2304.10573. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (eds.), Advances in Neu- ral Information Processing Systems , volume 33, pp. 6840–6851. Curran Associates, Inc.,

    arXiv doi:10.48550/arxiv.2304.10573

  2. [8]

    Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu

    URL https://openreview.net/forum?id=XVjTT1nw5z. Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pp. 14429–14460. PMLR, 2022. Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li,...

    arXiv 2022

  3. [10]

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann

    URL https://proceedings.mlr.press/v202/pooladian23a.html. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http://jmlr.org/papers/v22/ 20-1364.html. Allen Z Ren, Justin Lidard, L...

    arXiv 2021

  4. [13]

    cuda:0") if torch.cuda.is_available() else torch.device(

    URL https://openreview.net/forum?id=BJg9hTNKPH. Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp. 42363–42389. PMLR, 2023a. Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen. Guided flows for gene...

    arXiv 2020

  5. [2015]

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon

    URL https://proceedings.mlr.press/v37/sohl-dickstein15.html. Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. In M Ranzato, A Beygelzimer, Y Dauphin, P S Liang, and J Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1415–1428. Curran Associates, ...

    work page 2021

  6. [2018]

    [Online; accessed 25-June- 2019]

    URL https://github.com/tensorflow/agents. [Online; accessed 25-June- 2019]. Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio G ´omez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. Rl unplugged: A suite of benchmarks for offline reinforcement learning.Advances in Neural Information Processing Sy...

    arXiv 2019

  7. [2019]

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E

    URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/c2073ffa77b5357a498057413bb09d3a-Paper.pdf. Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (IC...

    work page 2019

  8. [2020]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet

    URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems ...

    work page 2020

Show all 13 references
  1. [2022]

    Matthew W

    URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/39235c56aef13fb05a6adc95eb9d8d66-Paper-Conference.pdf. Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Sta´nczyk, Sabela Ramos, Anton Raichuk, D...

    2022 arXiv

  2. [2023]

    OpenDILab Contributors

    URL https://api.semanticscholar.org/CorpusID:257378658. OpenDILab Contributors. Treetensor: a generalized tree-based tensor structure.https://github. com/opendilab/DI-treetensor, 2021. Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radf...

    2021 arXiv

  3. [2024]

    11 Ricky T

    URL https://openreview.net/forum?id=xCRr9DrolJ. 11 Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural...

    2018 arXiv

  4. [7727]

    Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, and Stefano Ermon

    URL https://ojs.aaai.org/index.php/AAAI/article/view/7727. Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, and Stefano Ermon. Torchdyn: implicit models and neural numerical methods in pytorch. InNeural Information Processing Systems, Workshop on...

    2021

  5. [8856]

    Expert Certification

    URL https://openreview.net/forum?id=CD9Snc73AW. Expert Certification. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 23(7):1661–1674, 2011. doi: 10.1162/NECO a 00142. Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro C...

    2011 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.