{"id":"da2adc94-55ec-4286-a9ef-e595c7893178","arxiv_id":"2412.01245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Two simple training schemes, GMPO and GMPG, let diffusion and flow models act as offline-RL policies with competitive or better scores on standard benchmarks, and a new framework, GenerativeRL, standardizes such experiments.","lead":"This paper proposes two simplified ways to train diffusion and flow models as reinforcement learning policies, one based on regression and one on policy gradients, and tests them on standard offline-RL benchmarks. A smart generalist reader might care because it offers a cleaner recipe for using generative models as controllers, plus a unified framework for fair comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline re-implementations in Table 9 are 3–5 points below published scores, biasing the Table 3 comparison and leaving the 'comparable/state-of-the-art' claim unestablished.","rationale":"I disagree with the reader's choice of weakest assumption. The substitution of the matching loss for the log-likelihood in GMPO is not a fundamental flaw: the exponential advantage weight simply reweights the target distribution from the behavior policy μ to the optimal policy π*, and denoising score matching for π* is a consistent surrogate for maximum likelihood under that target. The paper lacks a proof, but the step is standard and unlikely to fail. The more concrete threat is the baseline under-performance documented in Table 9. Since the paper's central claim is empirical ('comparable or better', 'state-of-the-art'), a comparison against weakened baselines directly undermines it. This is testable by re-running the baselines with their original setups; if they recover published scores, the proposed methods' margins shrink and the SOTA claim is contradicted by Diffusion-QL's published average alone. Thus the verdict stays CONDITIONAL pending baseline verification and claim tempering.","tokens_in":29044,"tokens_out":15641,"duration_ms":132397,"concrete_test":"Run QGPO, IDQL, and SRPO on the 9 D4RL locomotion tasks using the original authors' released code (or their published hyperparameters, including model type DDPM for IDQL and task-tuned β), and recompute the averages. If the re-implementations fall below the published averages by more than one standard error, the Table 3 comparison is unfair. Also compute the overall average including the published Diffusion-QL score (88.0) and report whether any GMPO/GMPG variant exceeds it; if not, revise the 'state-of-the-art' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that GMPO/GMPG match or exceed prior generative policy methods. For the three baselines the authors re-implemented (QGPO, IDQL, SRPO), Table 9 shows their framework versions score substantially below the published numbers: QGPO 81.8 vs 86.6, IDQL 78.3 vs 82.1, SRPO 82.8 vs 87.1. Table 3 uses these re-implemented scores, so GMPO's 82.7 is compared against an IDQL that is 3.8 points weaker than published, and GMPG's 87.3 is compared against an SRPO that is 4.3 points weaker. This inflates the apparent advantage. Against the published baselines, GMPO is only 0.6 points above IDQL and GMPG is 0.2 points above SRPO, while both fall below Diffusion-QL's published 88.0 average. The abstract's 'state-of-the-art' claim is therefore not supported by the paper's own table, and the 'comparable' claim rests on baseline implementations that may be under-tuned. The authors state that re-implementations were run 'under the same experimental settings for fair comparisons,' but no baseline hyperparameters or code are provided to verify that the baselines were given their best configuration, whereas the proposed methods have task-specific temperature β tuning (Table 8). This is a load-bearing concern because if the baselines recover their published scores with proper tuning, the relative performance of GMPO/GMPG weakens and may no longer be 'comparable' in the strong sense claimed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits generative policies for offline reinforcement learning, classifies prior methods into forward-KL and reverse-KL families, and proposes two training schemes: GMPO, an advantage-weighted regression objective using the generative matching loss, and GMPG, a policy-gradient objective with likelihoods computed through Neural ODEs. The authors introduce a framework called GenerativeRL and report experiments on D4RL locomotion, D4RL AntMaze, and RL Unplugged DeepMind Control Suite, including ablations over generative model type, sampling steps, temperature, and solver schemes. The central claims are that GMPO reaches IDQL-level performance without behavior-policy pretraining and that GMPO/GMPG offer unified, simple, and state-of-the-art or comparable generative policy training.","tokens_in":29405,"tokens_out":5321,"duration_ms":52809,"significance":"If the claims are established, the paper would be a useful consolidation: it identifies a minimal advantage-weighted regression formulation, provides a tractable policy-gradient implementation for continuous-time generative models, and contributes a modular framework that decouples generative models from RL components. The multi-seed evaluations, the comparison of diffusion and flow backbones, and the ablation of solver and sampling choices are commendable and go beyond many prior works. However, the theoretical grounding of GMPO is currently incomplete, and the empirical comparison is weakened by the reported gap between the paper's reimplemented baselines and the published scores of those baselines. The contribution is valuable as a heuristic and engineering study, but the paper's stronger theoretical and comparative claims need revision before publication.","major_comments":[{"comment":"The derivation of GMPO is not complete. Eq. (6) replaces log pi_theta(a|s) in the forward-KL AWR objective (Eq. 4) with the matching loss L_Matching(theta), and the text says this is justified by Song et al. (2021a) and that it 'can be derived independently.' Appendix C.6, Eq. (38), only rearranges the expectation over a ~ pi* into an expectation over a ~ mu with the exponential importance weight; it does not prove that the argmin of the weighted matching-loss objective equals the argmin of the weighted log-likelihood objective. The cited unweighted equivalence between maximum likelihood and score matching does not automatically survive multiplication by e^{beta(Q-V)}/Z, nor does it address restriction to the offline dataset's action support. Moreover, Appendix A.2.1, Eq. (14), states an ELBO-type upper bound, not an equality. The authors should either provide a proof that the optimum is preserved under this per-sample weighting or explicitly characterize GMPO as an empirically motivated heuristic and soften the 'native AWR' and theoretical claims accordingly.","section":"Section 4.3, Eq. (6), and Appendix C.6"},{"comment":"The headline comparison is biased by the reimplemented baselines. Table 9 reports published averages of 86.6 for QGPO, 82.1 for IDQL, and 87.1 for SRPO, while the GenerativeRL reimplementations achieve 81.8, 78.3, and 82.8 respectively, i.e., 3-5 points lower. Table 3 uses these reimplemented values as the comparison points. Against the published baselines, GMPO's 82.7 average is only 0.6 points above IDQL and 3.9 points below QGPO, and GMPG's 87.3 average is 0.2 points above the published SRPO and 0.7 points below the published Diffusion-QL average of 88.0. The gap may indicate under-tuned baselines rather than an advantage of the proposed methods. The paper should either re-tune the reimplemented baselines to their published performance, provide the full hyperparameter settings and code for those baselines, or restrict the comparative claims to 'our reimplementation' and avoid the abstract's state-of-the-art wording.","section":"Section 6.1, Table 3, and Table 9"},{"comment":"The claimed overturning of the IDQL finding about advantage-weighted regression is not supported as stated. The text says GMPO achieves 'comparably equivalent performance to IDQL using resampling tricks,' but Table 3 shows GMPO at 82.7 against the reimplemented IDQL at 78.3, while the published IDQL is 82.1. This difference is within the reported standard deviations, and the comparison also involves different generative model types (GVP for GMPO versus VPSDE for IDQL in the reimplementation). The authors should either use the published IDQL score as the reference, explain the reimplementation discrepancy, or weaken the claim to a statement about comparable performance under careful tuning.","section":"Section 6.2"}],"minor_comments":[{"comment":"The sentence 'As shown in Table 1, all methods require pretraining a behavior policy' contradicts Table 1 and Algorithm 1, where GMPO explicitly does not require a behavior policy; the sentence should say 'all previous methods'.","section":"Section 4.2"},{"comment":"The sign convention is inconsistent: Eq. (6) defines GMPO as an expectation of the nonnegative matching loss L_Matching, but Eqs. (39) and (40) introduce a leading minus sign before the expectation. The authors should clarify whether these expressions are meant as losses to be minimized or as negative objectives, and make the notation consistent.","section":"Appendix C.6, Eqs. (39) and (40)"},{"comment":"The paper states that baselines were reimplemented 'under the same experimental settings for fair comparisons,' but it provides no code repository link and no per-baseline hyperparameter table. Given that the proposed methods use task-specific temperature beta tuning (Table 8), the fairness argument would be materially strengthened by releasing the benchmark code and the baseline configurations.","section":"Section 6.1 and Appendix D.1"},{"comment":"The claim that GenerativeRL provides a 'standardized experimental framework' is only supported by usage examples in the appendix; the paper should include a clear statement of which parts of the framework are released, with a public link, so that the claimed standardization is verifiable.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands on both main pillars of the paper. The theoretical derivation in Section 4.3 does not establish the claimed equivalence between the weighted matching-loss objective and the weighted log-likelihood objective, and the baseline reimplementation gap in Table 9 materially weakens the empirical comparison. Both issues are fixable: the authors can either prove the equivalence or reframe GMPO as a heuristic, and they can rerun or transparently document the baselines. I do not see a load-bearing error that forces rejection, but the current version's abstract and Section 6.2 overstate what the paper demonstrates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth reading for the GMPO idea, but the headline empirical claim does not survive contact with Table 9.\n\nWhat's actually new: GMPO combines an IQL critic with an advantage-weighted matching loss on the offline dataset, no behavior-policy pretraining, and applies to both diffusion and flow models. That combination looks new, and the demonstration that scratch-initialized GMPO roughly matches IDQL's resampling-based performance is a legitimate counter to Hansen-Estruch et al.'s claim that AWR doesn't help expressive models. GMPG is a more straightforward contribution: a policy gradient for continuous-time generative policies using exact log-likelihood via Neural ODEs and Hutchinson trace. The paper also does useful housekeeping by classifying existing methods and providing ablations over model type, solver, and sampling steps. Five seeds is fine for this setting.\n\nThe soft spots are real but fixable. First, the derivation of GMPO is not fully on solid ground. The paper replaces the log-likelihood in the forward-KL AWR objective with the matching loss and cites Song et al. for an unweighted equivalence. Appendix C.6 only rearranges the expectation; it does not show the equivalence survives the exponential advantage weight. That may be true in practice—Diffusion-QL and others make similar substitutions—but it means GMPO is a heuristic with an appealing derivation, not a 'native' AWR method, until someone supplies the argument.\n\nSecond, the empirical comparison is biased by the re-implemented baselines. QGPO, IDQL, and SRPO all score 3-5 points lower in the authors' framework than in their original papers (Table 9), and Table 3 uses those lower numbers. Against published scores, GMPO is 0.6 points above IDQL and GMPG is 0.2 points above SRPO, with both below Diffusion-QL's 88.0. The 'state-of-the-art' claim in the abstract is not supported by the paper's own tables. This is the biggest issue.\n\nThird, beta is tuned per task for GMPO/GMPG, and no baseline hyperparameters or code are given, so it's hard to tell if the baselines got a fair shake. No code release also hurts reproducibility, though the framework description helps.\n\nBottom line: the GMPO recipe is a useful contribution and the framework could be handy, but the empirical claims need to be re-run against published baseline numbers, and the theory gap needs to be acknowledged or closed. I'd send it to a serious referee, with the request that the authors address those two things.","headline":"Worth sending to review for the GMPO recipe and framework, but the SOTA claim is undercut by under-tuned re-implemented baselines and the GMPO derivation has a gap.","tokens_in":29919,"tokens_out":2389,"would_cite":true,"duration_ms":20949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two simple, RL-native objectives—advantage-weighted regression and policy gradient—make generative policies match or beat prior state-of-the-art results on offline reinforcement learning benchmarks.","keywords":["offline reinforcement learning","generative policies","diffusion models","flow matching","advantage-weighted regression","policy gradient","implicit Q-learning","D4RL"],"falsifier":"On a task like halfcheetah-medium-v2, train one policy with GMPO (advance-weighted matching loss) and another with the same architecture but an explicit advantage-weighted log-likelihood objective computed through a Neural ODE, keeping the Q-function, β, and step budget identical; if GMPO's returns cannot be matched or exceeded by the explicit-likelihood version, the substitution in Eq. 6 is not preserving the intended optimum.","tokens_in":28854,"feed_emoji":"🤖","tokens_out":7012,"duration_ms":55137,"temperature":0.7,"pith_summary":"This paper argues that prior generative-policy algorithms in offline reinforcement learning have accumulated unnecessary complexity—behavior-policy pretraining, resampling, and model-specific tricks—and that two simple, RL-native objectives suffice. The first, GMPO, is an advantage-weighted regression objective that uses the generative model's matching loss (score matching or flow matching) in place of a log-likelihood term, and it works even when the policy is initialized from scratch with no behavior-policy pretraining and no resampling. The second, GMPG, is a numerically stable, native policy-gradient objective for continuous-time generative models built on Neural ODE gradients and Hutchinson trace estimation of log-likelihood. On D4RL and RL Unplugged benchmarks, the authors report state-of-the-art or comparable performance for both schemes across diffusion and flow models, which would make a single unified recipe for training generative policies if the claims hold.","feed_headline":"Two simpler objectives make generative policies competitive","feed_subtitle":"GMPO and GMPG unify diffusion and flow policies, reaching state-of-the-art offline-RL scores without extra tricks.","key_machinery":"The machinery is the closed-form optimal policy of constrained offline RL, π*(a|s) ∝ $e^{{β(Q(s,a)−V(s))}}$ μ(a|s), and its two KL divergence training objectives. Forward-KL (advantage-weighted regression) yields GMPO after replacing the log-likelihood log π_θ(a|s) with the generative model's matching loss L_Matching(θ) (denoising score matching or conditional flow matching), so any generative model with a matching objective can be trained as a policy. Reverse-KL (policy gradient) yields GMPG, which computes the policy log-likelihood through the instantaneous change-of-variables formula integrated by a Neural ODE, with the Jacobian trace estimated by Hutchinson's estimator. The crucial load-bearing identity is the equivalence between maximum-likelihood training and score matching for diffusion models, cited from Song et al. (2021a), which the paper extends—without proof—to the advantage-weighted variant.","core_discovery":"On the paper's terms, the central discovery is that the advantage-weighted regression objective used by prior forward-KL methods can be reduced to a clean matching-loss objective—GMPO—and that this objective extracts near-optimal policies directly from an IQL-trained Q-function without needing a pretrained behavior policy or inference-time resampling. The paper states explicitly that a generative policy trained with advantage-weighted regression from a scratch initialization gains performance comparable to IDQL's resampling-based scheme, overturning the earlier claim that advantage-weighted regression does not help for expressive models. The companion objective GMPG implements the reverse-KL policy gradient by simulating the generative ODE forward and back-propagating through it, giving a stable policy-gradient baseline that outperforms GMPO on several medium and medium-replay locomotion tasks. Together the two objectives are presented as a unified, model-agnostic training scheme valid for both diffusion and flow models.","pith_inferences":["If GMPO's substitution is only known to hold for unweighted maximum likelihood, the advantage-weighted matching loss may select a different policy optimum; a controlled comparison against explicit advantage-weighted log-likelihood on the same Q-function would settle the gap.","The same template could extend to other generative families (e.g., Schrödinger bridges or normalizing flows) since only a matching loss and a sampling procedure are required, not diffusion-specific machinery.","In high-dimensional action spaces, the toy-example distinction between GMPO's data-filtering and GMPG's manifold-preserving paths could become empirically decisive, favoring GMPG where manifold structure matters.","The paper's importance-sampled GMPG variant (Eq. 42) is a natural hybrid of forward and reverse KL that the experiments do not compare; testing it against GMPO and GMPG would map the stability-performance trade-off."],"forward_implications":["Behavior-policy pretraining and inference-time resampling are unnecessary for competitive generative policies: GMPO matches IDQL-level returns while training the policy from a scratch initialization.","The same objectives transfer directly across generative model families, as shown by comparable GMPO/GMPG scores with VP-SDE diffusion, GVP diffusion, and I-CFM flow models.","A reverse-KL policy gradient implemented with ODE simulation and trace estimation is stable enough for training (T=1000), removing the low-step-size constraint highlighted in Diffusion-QL.","IQL-trained Q-functions suffice for explicit generative policy extraction, decoupling critic training from policy optimization and simplifying the pipeline.","GMPG's aggressive Q-guidance helps in medium and medium-replay locomotion tasks but hurts in AntMaze and expert-data tasks, indicating a systematic trade-off between the two objectives."],"supporting_citations":[{"why":"Supplies the equivalence between score matching and maximum likelihood training for diffusion models, which GMPO relies on to replace the log-likelihood term with the matching loss.","marker":"Song et al. (2021a)"},{"why":"Introduces IDQL and IQL, the critic method GMPO/GMPG share, the resampling baseline GMPO is compared against, and the claim that AWR does not help expressive models, which the paper overturns.","marker":"Hansen-Estruch et al. (2023)"},{"why":"Provides QGPO, the advantage-weighted regression baseline, the U-Net architecture, and the experimental settings reused in this paper's comparisons.","marker":"Lu et al. (2023)"},{"why":"Diffusion-QL, the reverse-KL policy gradient with score-matching loss and T=5 limitation that GMPG directly addresses with full ODE simulation.","marker":"Wang et al. (2023)"},{"why":"SRPO, a reverse-KL regularized policy optimization baseline using the behavior policy's score function, compared against GMPO and GMPG.","marker":"Chen et al. (2024)"},{"why":"AWR; gives the closed-form optimal policy expression e^{β(Q−V)}μ/Z that the forward-KL and reverse-KL objectives are derived from.","marker":"Peng et al. (2021)"},{"why":"Neural ODEs and the adjoint method, the gradient machinery GMPG uses for backpropagation through the generative sampling ODE.","marker":"Chen et al. (2018)"},{"why":"Provides Hutchinson trace estimation and likelihood computation for continuous-time generative models, used in GMPG's log-likelihood term.","marker":"Grathwohl et al. (2019)"},{"why":"Defines the D4RL benchmark and normalized score metric on which the main comparisons in Table 3 are reported.","marker":"Fu et al. (2021)"},{"why":"Defines the conditional flow matching objective and I-CFM model used to demonstrate that GMPO and GMPG apply to flow models.","marker":"Tong et al. (2024)"}],"fun_headline_variants":["Simplicity wins: two clean objectives for generative policies","GMPO and GMPG: simpler generative RL achieves SOTA","Forget complex tricks: simple objectives fix generative policies","Generative policies made simple: matching loss beats resampling","A unified scheme for diffusion and flow policies, minus the extras"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument leans on the assumption that the well-known equivalence between score matching and maximum likelihood survives when the matching loss is multiplied by an exponential advantage weight and evaluated only on offline dataset actions, so that GMPO's optimum equals the true advantage-weighted regression optimum.","fun_headline_variants_meta":{"raw":{"variants":["Simplicity wins: two clean objectives for generative policies","GMPO and GMPG: simpler generative RL achieves SOTA","Forget complex tricks: simple objectives fix generative policies","Generative policies made simple: matching loss beats resampling","A unified scheme for diffusion and flow policies, minus the extras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1436,"prompt_tokens":907,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":523,"tokens_out":529,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:31:49.994293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a task like halfcheetah-medium-v2, train one policy with GMPO (advance-weighted matching loss) and another with the same architecture but an explicit advantage-weighted log-likelihood objective computed through a Neural ODE, keeping the Q-function, β, and step budget identical; if GMPO's returns cannot be matched or exceeded by the explicit-likelihood version, the substitution in Eq. 6 is not preserving the intended optimum.","supporting_citations":[],"review_version":1}