REVIEW 3 major objections 6 minor 1 cited by
Policy-gradient reinforcement learning can tune velocity-only flow-based generative models of crystals without ever computing a score, by treating a noise-perturbed ODE as the sampling policy.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 06:01 UTC pith:7LSA565U
load-bearing objection The empirical method is real and useful; the theory is hand-wavy, the energy evaluation is partly circular, and the diversity claim is unmeasured — but the paper deserves a serious referee. the 3 major comments →
Open Materials Generation with Inference-Time Reinforcement Learning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that effective policy-gradient RL is possible even in velocity-only flow-based models without an explicit score. The authors introduce a surrogate stochastic differential equation that adds a noise schedule to the learned velocity ODE; because the omitted score correction is argued to be subdominant, of order sigma_ref squared, the surrogate is treated as a valid stochastic policy for GRPO/PPO updates, with KL regularization keeping the reinforced policy close to the frozen pretrained model. In crystal structure prediction, reinforcing relative energy per atom yields an improvement of roughly 0.5 eV per atom at 50 integration steps, matching the performance of a score-ba
What carries the argument
The key object is the surrogate SDE that augments the velocity ODE with a noise schedule sigma_ref(t) sqrt(Delta t) xi, providing stochasticity for exploration and likelihood-ratio estimation without requiring the score. Group-relative policy optimization (GRPO) compares terminal rewards across rollouts of the same composition to compute advantages, and KL regularization to a frozen reference policy preserves the pretrained model's inductive biases. A second mechanism, velocity-annealing OMatG-IRL, learns a time-dependent scaling s_theta(t) applied to the frozen velocity, effectively learning a residual policy that replaces handcrafted annealing schedules.
Load-bearing premise
The load-bearing premise is that dropping the score correction from the true SDE leaves the sampling distribution essentially unchanged, with deviation bounded by O(sigma_ref^2), even though the correction term can become singular as t approaches 1 because gamma(t) vanishes there; the practical guarantee rests on empirical stability rather than a proved bound.
What would settle it
Run the surrogate SDE and the full score-based SDE on a model that provides both velocity and denoiser (e.g., the Trig-SDE-Gamma model), then measure the distance between the endpoint distributions or structure metrics as a function of noise scale and time near t = 1; if the deviation grows faster than O(sigma^2) or visibly degrades match rate at the noise scales used during RL, the policy-gradient estimates are biased by samples that are not representative of the pretrained model's distribution.
If this is right
- RL can now be applied to flow-matching and stochastic-interpolant models that learn only a velocity field, broadening the class of generative models that can be aligned with downstream objectives.
- Energy-based reinforcement improves CSP quality: relative energy per atom drops by about 0.5 eV, RMSE decreases, and match rate is maintained while using 50 integration steps instead of 740.
- The velocity-only approach matches the score-based approach in reinforcement performance, so the score-free route is not a performance compromise.
- A learned velocity-annealing schedule recovers baseline METRe/cRMSE quality at N_t = 100 and stays accurate down to N_t = 10, where hand-tuned annealing schedules fail.
- In the CSP setting, diversity emerges naturally from composition conditioning, so explicit diversity rewards are unnecessary, in contrast to de novo generation tasks.
Where Pith is reading between the lines
- Editorial inference: if the surrogate-SDE shortcut is as benign as it appears, similar inference-time RL could be applied to other velocity-only flow models beyond crystals, such as protein backbones or small molecules, without needing score heads.
- Editorial inference: the learned annealing schedule's qualitative difference from handcrafted schedules suggests discretization error and stochasticity interact in ways that tuned scalar annealings miss; testing whether the learned schedule transfers across models, datasets, or step counts would clarify whether it captures a general correction.
- Editorial inference: the fragile O(sigma_ref^2) Girsanov argument is the piece to stress-test; a direct comparison of endpoint densities from the surrogate and the true SDE near t = 1 could turn the practical heuristic into a rigorous validity criterion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces OMatG-IRL, a policy-gradient RL framework for continuous-time generative models of crystal structures built on stochastic interpolants. The key methodological contribution is a velocity-only variant that augments the deterministic ODE dynamics with an additive noise schedule, avoiding any explicit score computation, and uses GRPO/PPO-style updates with KL regularization. The paper demonstrates the approach on crystal structure prediction (CSP): an energy-based reward is reinforced with Nt=50 integration steps, matching the performance of a score-based variant, and a learned velocity-annealing schedule recovers the performance of a pretrained OMatG model while reducing integration steps from Nt=950 to as few as Nt=10.
Significance. If the empirical results hold, the paper makes a useful contribution: it extends policy-gradient RL to velocity-only flow/SI models, reports what appears to be the first RL application to CSP, and shows that a learned time-dependent annealing schedule can replace handcrafted ones. The release of the OMatG-IRL code is a concrete strength, as are the controlled comparisons between score-based and velocity-based variants and the extensive hyperparameter documentation. However, the theoretical justification of the core surrogate-SDE construction is not rigorous, and the energy-based evaluation partly overlaps with the reward, which tempers the strength of the headline claims. The central empirical claim is nevertheless plausible and worth publishing after revision.
major comments (3)
- [§3.2, Eq. (13)] The claim that the omitted score correction is O(σ_ref²) and that Girsanov gives an O(σ_ref²) bound on the deviation from the true marginal is not substantiated. For the square-root schedule σ_ref(t)=a√((1−t)/t) used in the RL experiments, and for the OMatG trig interpolants where γ(t) vanishes at the endpoints, the Girsanov integrand |σ_ref(t) z_θref/(2γ(t))|² is not square-integrable in t (near t=0 it behaves as 1/t³ for typical trig γ). The path-measure KL is therefore not finite in general, and the stated bound does not follow. Moreover, Girsanov applies to continuous-time SDEs, while Eq. (13) is a discrete Euler–Maruyama update; no bound for the discretized process is given. Figure 2 reports only final-time scalar metrics, which do not control the pathwise deviation relevant to the likelihood ratios in Eqs. (10)–(11). Since the KL regularizer in Eq. (11) uses this surrogate as the r
- [§4.2 / §2.4, Fig. 3] The energy-based reward (negative clipped MACE-MPA-0 energy per atom) and the headline evaluation metric ('relative energy per atom') use the same MACE-MPA-0 model. The observed ~0.5 eV/atom reduction is therefore in part a tautology: the policy is directly optimizing the metric being reported. The manuscript should state this explicitly and place the emphasis on the non-reward metrics (match rate, RMSE, invalid-energy rate), which do show that optimization does not destroy overall generation quality. As written, the energy improvement is over-interpreted.
- [§3.3, Eqs. (14)–(15)] The velocity-annealing surrogate scales the exploration noise by the pretrained velocity b_θref(t, x_t), giving state-dependent multiplicative noise. No justification is provided for this choice, and the noise vanishes wherever b_θref is small, potentially inhibiting exploration exactly where the velocity changes sign or is near zero. The learned-annealing results rest on this surrogate; please clarify the rationale and discuss the behavior, or consider a noise schedule independent of the velocity.
minor comments (6)
- [Eq. (14)] The notation b_θref is used without arguments; define b_θref(t, x_t) explicitly.
- [§3.3] The expression 'st' in the handcrafted annealing rule is ambiguous; write s·t or define it explicitly.
- [Fig. 2 / Appendix D] The values of a_s, a_m, a_l are not given in the text. Report the actual numerical values for reproducibility.
- [Appendix G] The chosen reference noise scales are denoted a^pos_ref twice; the second should be a^cell_ref.
- [Appendix H] The hyperparameter sweep only varies scalar coefficients of a handcrafted (presumably linear-in-time) annealing schedule. The conclusion that 'no configuration recovers' should be limited to that restricted family; a more general sweep over nonlinear schedules would be needed to fully substantiate the claim that the improvement arises from the learned policy.
- [Abstract / §1.2] The claim of 'the first application of RL to CSP' should be hedged as 'to our knowledge' to avoid overclaiming.
Circularity Check
Energy and cRMSE headline results are the RL rewards by construction; core score-free RL claim is independent.
specific steps
-
fitted input called prediction
[Section 4.2 / Section 2.4 / Appendix E]
"As the reward, we use a clipped relative energy per atom to prevent outliers from dominating the group-normalized advantages and to keep policy updates stable (see Appendix E). ... The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards."
The RL reward is the clipped MACE-MPA-0 energy per atom (Appendix E), and the reported evaluation metric is the same MACE-MPA-0 relative energy per atom (Section 2.4). The headline result that RL lowers relative energy by ~0.5 eV/atom is therefore the training objective itself evaluated at convergence, not an independent prediction: the improvement is forced by maximizing the reward. The paper does add independent evidence (RMSE, match rate, held-out test evaluation), so this is partial circularity, not the whole claim.
-
fitted input called prediction
[Section 4.3 / Appendix G]
"Because the learned annealing schedule depends only on time (and not on composition), and because velocity annealing is known to directly improve structural metrics, we can reinforce the model using a cRMSE-like objective in this setting (see Appendix G for details). ... The reward is defined as r(x^i_{t=1}) = 0.5 − cRMSE^i, where cRMSE^i is the normalized root-mean-square displacement to this best match, with missing matches penalized in the mean using stol=0.5."
The policy is trained to minimize the cRMSE-like reward r = 0.5 − cRMSE computed against training-set polymorphs, and the paper then reports cRMSE as a key test-set evaluation metric. The cRMSE improvements/recovery reported in Fig. 4 are thus directly aligned with the fitted objective rather than being an independent check. Independent content comes from the METRe metric, the explicit handcrafted-annealing sweep at N_t=10, and robustness across integration step counts.
full rationale
Core methodological claim—policy-gradient RL on velocity-only SI models via the surrogate SDE (Eq. 13)—is not circular: the surrogate policy, GRPO objective, and KL regularization are defined independently of the reward and of the reported metrics. The Girsanov/O(σ²) justification for Eq. (13) is a mathematical-rigor concern (the bound is asserted, not proven, and may fail for the square-root schedule), but it is a correctness issue, not a circularity, and was not scored here. The circularity is limited to the evaluation of the trained policies: two headline 'predictions' (relative energy reduction in Sec. 4.2; cRMSE recovery in Sec. 4.3) use evaluation metrics that are the same as (energy) or directly derived from (cRMSE-like) the RL rewards. Because the paper also reports independent metrics (match rate, RMSE, METRe) and compares against handcrafted baselines, the central derivation retains independent content. Self-citations to OMatG/Höllmer et al. supply pretrained models and the SI framework as external resources, not as a load-bearing uniqueness argument. Overall, partial circularity in the headline metrics: score 6.
Axiom & Free-Parameter Ledger
free parameters (5)
- square-root noise scale a =
a_s, a_m, a_l selected via validation sweep; exact values not reported
- reference noise scale a_ref =
a_s or a_m
- reward clipping band and invalid-energy penalty =
3 eV/atom penalty; clipping to ±3 std around group mean
- RL hyperparameters (G, PPO epochs, ε, KL weight, denoiser-distillation weight, learning rate) =
Tuned via Optuna/Ray Tune; final values not listed
- number of integration steps Nt =
50 for energy RL, 100 for annealing RL
axioms (5)
- standard math The SI velocity field trained by Eq. (2) produces correct marginals when integrated by the ODE Eq. (3).
- domain assumption The omitted score correction in Eq. (13) leads to a negligible deviation from the true marginal, O(σ²_ref).
- domain assumption MACE-MPA-0 energies are a faithful proxy for chemical stability.
- domain assumption Composition conditioning preserves diversity in CSP without explicit diversity rewards.
- domain assumption A cRMSE-like reward computed against training-set polymorphs transfers to test-set improvements.
read the original abstract
Continuous-time generative models for crystalline materials enable inverse materials design by learning to predict stable crystal structures, but incorporating explicit target properties into the generative process remains challenging. Policy-gradient reinforcement learning (RL) provides a principled mechanism for aligning generative models with downstream objectives but typically requires access to the score, which has prevented its application to flow-based models that learn only velocity fields. We introduce Open Materials Generation with Inference-time Reinforcement Learning (OMatG-IRL), a policy-gradient RL framework that operates directly on the learned velocity fields and eliminates the need for the explicit computation of the score. OMatG-IRL leverages stochastic perturbations of the underlying generation dynamics preserving the baseline performance of the pretrained generative model while enabling exploration and policy-gradient estimation at inference time. Using OMatG-IRL, we present the first application of RL to crystal structure prediction (CSP). Our method enables effective reinforcement of an energy-based objective while preserving diversity through composition conditioning, and it achieves performance competitive with score-based RL approaches. Finally, we show that OMatG-IRL can learn time-dependent velocity-annealing schedules, enabling accurate CSP with order-of-magnitude improvements in sampling efficiency and, correspondingly, reduction in generation time. The OMatG-IRL code is included in a new release of the Open Materials Generation (OMatG) framework available at https://github.com/FERMat-ML/OMatG.
Figures
Forward citations
Cited by 1 Pith paper
-
VASP Plugins: Linking the Vienna ab-initio Simulation Package with Python
A C++/pybind11 shared-memory plugin layer exposes VASP SCF and ionic data as NumPy arrays so Python can modify structure, forces, local potential, and occupancies in place.
Reference graph
Works this paper leans on
-
[1]
The unit-cell volume is smaller than0.1 ˚A 3
-
[2]
Any pairwise atomic distance (accounting for periodic boundary conditions) is smaller than0.5 ˚A
-
[3]
The polar sine of the lattice vectors is smaller than10 −3, indicating a nearly degenerate unit cell. C. Sensitivity of OMatG to Integration Steps and V elocity Annealing In Table 1, we show that the performance of the pretrained Trig-SDE-Gamma model in OMatG is largely unchanged when switching between SDE integration via Eq. (7) and ODE integration via E...
2024
-
[4]
Øksendal, B.Stochastic Differential Equations
URL https://www.doi.org/10.1039/ D0SC00594K. Øksendal, B.Stochastic Differential Equations. Springer, sixth edition, 2013. Ong, S. P., Richards, W. D., Jain, A., Hautier, G., Kocher, M., Cholia, S., Gunter, D., Chevrier, V . L., Persson, K. A., and Ceder, G. Python Materials Genomics (py- matgen): A robust, open-source python library for mate- rials analy...
arXiv 2013
-
[7]
11 Open Materials Generation with Inference-Time Reinforcement Learning Sriram, A., Miller, B
URL https://openreview.net/forum? id=PxTIG12RRHS. 11 Open Materials Generation with Inference-Time Reinforcement Learning Sriram, A., Miller, B. K., Chen, R. T. Q., and Wood, B. M. FlowLLM: Flow Matching for Material Gener- ation with Large Language Models as Base Distribu- tions. InThe Thirty-eighth Annual Conference on Neural Information Processing Syst...
Pith/arXiv arXiv 2024
-
[12]
The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards
and clip the resulting values to a band of three standard deviations around the group mean to prevent outliers from dominating the advantages. The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards. Since all structures within a GRPO group share the same composition, we do not nee...
2018
-
[2018]
doi: https://www.doi.org/10.1126/science.aat2663. Satorras, V . G., Hoogeboom, E., and Welling, M. E(n) Equivariant Graph Neural Networks. InProceedings of the 38th International Conference on Machine Learning,
-
[2020]
URL https://dl.acm.org/doi/abs/10. 5555/3495724.3496298. H¨ollmer, P., Egg, T., Martirossyan, M., Fuemmeler, E., Shui, Z., Gupta, A., Prakash, P., Roitberg, A., Liu, M., Karypis, G., Transtrum, M., Hennig, R., Tadmor, E. B., and Martiniani, S. Open Materials Generation with Stochastic Interpolants. InForty-second International Conference on Machine Learni...
arXiv 2025
-
[2021]
URL https://proceedings.mlr.press/ v139/satorras21a.html. Schmidt, J., Marques, M. R. G., Botti, S., and Marques, M. A. L. Recent advances and applications of machine learn- ing in solid-state materials science.npj Computational Materials, 5:83, 2019. URL https://doi.org/10. 1038/s41524-019-0221-0. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and ...
Pith/arXiv arXiv 2019
-
[2023]
Batatia, I., Benner, P., Chiang, Y ., Elena, A
URL https://openreview.net/forum? id=li7qeBbCR1t. Batatia, I., Benner, P., Chiang, Y ., Elena, A. M., Kov´acs, D. P., Riebesell, J., Advincula, X. R., Asta, M., Avaylon, M., Baldwin, W. J., Berger, F., Bernstein, N., Bhowmik, A., Bigi, F., Blau, S. M., C˘arare, V ., Ceriotti, M., Chong, S., Darby, J. P., De, S., Della Pia, F., Deringer, V . L., Elijoˇsius...
arXiv 2025
-
[2024]
Yim, J., Campbell, A., Foong, A
URL https://openreview.net/forum? id=wm4WlHoXpC. Yim, J., Campbell, A., Foong, A. Y . K., Gastegger, M., Jim´enez-Luna, J., Lewis, S., Satorras, V . G., Veeling, B. S., Barzilay, R., Jaakkola, T., and No´e, F. Fast protein backbone generation with SE(3) flow matching, 2023. URLhttps://arxiv.org/abs/2310.05297. Zeni, C., Pinsler, R., Z ¨ugner, D., Fowler, ...
Pith/arXiv arXiv 2023
-
[2025]
De Breuck, P.-P., Wang, H.-C., Rignanese, G.-M., Botti, S., and Marques, M
URL https://openreview.net/forum? id=7J1kwZY72h. De Breuck, P.-P., Wang, H.-C., Rignanese, G.-M., Botti, S., and Marques, M. A. L. Generative AI for crys- tal structures: a review.npj Computational Materials, 11:370, 2025. URL https://doi.org/10.1038/ s41524-025-01881-2. Fan, Y . and Lee, K. Optimizing DDPM sampling with shortcut fine-tuning. InProceeding...
arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.