Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Policy-gradient reinforcement learning can tune velocity-only flow-based generative models of crystals without ever computing a score, by treating a noise-perturbed ODE as the sampling policy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 06:01 UTC pith:7LSA565U

load-bearing objection The empirical method is real and useful; the theory is hand-wavy, the energy evaluation is partly circular, and the diversity claim is unmeasured — but the paper deserves a serious referee. the 3 major comments →

arxiv 2602.00424 v2 pith:7LSA565U submitted 2026-01-31 cs.LG cond-mat.mtrl-sci

Open Materials Generation with Inference-Time Reinforcement Learning

classification cs.LG cond-mat.mtrl-sci
keywords reinforcement learningflow-based generative modelsstochastic interpolantscrystal structure predictionvelocity annealingGRPOmaterials generationscore-free RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to let reinforcement learning steer continuous-time generative models of crystals that learn only a velocity field, with no access to a score. It claims that policy-gradient RL works in this setting if the deterministic ODE is augmented with a carefully scaled noise schedule, turning the perturbed ODE into a stochastic policy while leaving baseline generation quality nearly unchanged. Applied to crystal structure prediction for the first time, the method reinforces an energy-based reward, cutting relative energy per atom by about 0.5 eV while reducing the number of integration steps from 740 to 50. A learned, time-dependent velocity-annealing schedule further makes generation accurate with an order-of-magnitude fewer steps than the hand-tuned baseline.

Core claim

The central claim is that effective policy-gradient RL is possible even in velocity-only flow-based models without an explicit score. The authors introduce a surrogate stochastic differential equation that adds a noise schedule to the learned velocity ODE; because the omitted score correction is argued to be subdominant, of order sigma_ref squared, the surrogate is treated as a valid stochastic policy for GRPO/PPO updates, with KL regularization keeping the reinforced policy close to the frozen pretrained model. In crystal structure prediction, reinforcing relative energy per atom yields an improvement of roughly 0.5 eV per atom at 50 integration steps, matching the performance of a score-ba

What carries the argument

The key object is the surrogate SDE that augments the velocity ODE with a noise schedule sigma_ref(t) sqrt(Delta t) xi, providing stochasticity for exploration and likelihood-ratio estimation without requiring the score. Group-relative policy optimization (GRPO) compares terminal rewards across rollouts of the same composition to compute advantages, and KL regularization to a frozen reference policy preserves the pretrained model's inductive biases. A second mechanism, velocity-annealing OMatG-IRL, learns a time-dependent scaling s_theta(t) applied to the frozen velocity, effectively learning a residual policy that replaces handcrafted annealing schedules.

Load-bearing premise

The load-bearing premise is that dropping the score correction from the true SDE leaves the sampling distribution essentially unchanged, with deviation bounded by O(sigma_ref^2), even though the correction term can become singular as t approaches 1 because gamma(t) vanishes there; the practical guarantee rests on empirical stability rather than a proved bound.

What would settle it

Run the surrogate SDE and the full score-based SDE on a model that provides both velocity and denoiser (e.g., the Trig-SDE-Gamma model), then measure the distance between the endpoint distributions or structure metrics as a function of noise scale and time near t = 1; if the deviation grows faster than O(sigma^2) or visibly degrades match rate at the noise scales used during RL, the policy-gradient estimates are biased by samples that are not representative of the pretrained model's distribution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RL can now be applied to flow-matching and stochastic-interpolant models that learn only a velocity field, broadening the class of generative models that can be aligned with downstream objectives.
  • Energy-based reinforcement improves CSP quality: relative energy per atom drops by about 0.5 eV, RMSE decreases, and match rate is maintained while using 50 integration steps instead of 740.
  • The velocity-only approach matches the score-based approach in reinforcement performance, so the score-free route is not a performance compromise.
  • A learned velocity-annealing schedule recovers baseline METRe/cRMSE quality at N_t = 100 and stays accurate down to N_t = 10, where hand-tuned annealing schedules fail.
  • In the CSP setting, diversity emerges naturally from composition conditioning, so explicit diversity rewards are unnecessary, in contrast to de novo generation tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the surrogate-SDE shortcut is as benign as it appears, similar inference-time RL could be applied to other velocity-only flow models beyond crystals, such as protein backbones or small molecules, without needing score heads.
  • Editorial inference: the learned annealing schedule's qualitative difference from handcrafted schedules suggests discretization error and stochasticity interact in ways that tuned scalar annealings miss; testing whether the learned schedule transfers across models, datasets, or step counts would clarify whether it captures a general correction.
  • Editorial inference: the fragile O(sigma_ref^2) Girsanov argument is the piece to stress-test; a direct comparison of endpoint densities from the surrogate and the true SDE near t = 1 could turn the practical heuristic into a rigorous validity criterion.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces OMatG-IRL, a policy-gradient RL framework for continuous-time generative models of crystal structures built on stochastic interpolants. The key methodological contribution is a velocity-only variant that augments the deterministic ODE dynamics with an additive noise schedule, avoiding any explicit score computation, and uses GRPO/PPO-style updates with KL regularization. The paper demonstrates the approach on crystal structure prediction (CSP): an energy-based reward is reinforced with Nt=50 integration steps, matching the performance of a score-based variant, and a learned velocity-annealing schedule recovers the performance of a pretrained OMatG model while reducing integration steps from Nt=950 to as few as Nt=10.

Significance. If the empirical results hold, the paper makes a useful contribution: it extends policy-gradient RL to velocity-only flow/SI models, reports what appears to be the first RL application to CSP, and shows that a learned time-dependent annealing schedule can replace handcrafted ones. The release of the OMatG-IRL code is a concrete strength, as are the controlled comparisons between score-based and velocity-based variants and the extensive hyperparameter documentation. However, the theoretical justification of the core surrogate-SDE construction is not rigorous, and the energy-based evaluation partly overlaps with the reward, which tempers the strength of the headline claims. The central empirical claim is nevertheless plausible and worth publishing after revision.

major comments (3)
  1. [§3.2, Eq. (13)] The claim that the omitted score correction is O(σ_ref²) and that Girsanov gives an O(σ_ref²) bound on the deviation from the true marginal is not substantiated. For the square-root schedule σ_ref(t)=a√((1−t)/t) used in the RL experiments, and for the OMatG trig interpolants where γ(t) vanishes at the endpoints, the Girsanov integrand |σ_ref(t) z_θref/(2γ(t))|² is not square-integrable in t (near t=0 it behaves as 1/t³ for typical trig γ). The path-measure KL is therefore not finite in general, and the stated bound does not follow. Moreover, Girsanov applies to continuous-time SDEs, while Eq. (13) is a discrete Euler–Maruyama update; no bound for the discretized process is given. Figure 2 reports only final-time scalar metrics, which do not control the pathwise deviation relevant to the likelihood ratios in Eqs. (10)–(11). Since the KL regularizer in Eq. (11) uses this surrogate as the r
  2. [§4.2 / §2.4, Fig. 3] The energy-based reward (negative clipped MACE-MPA-0 energy per atom) and the headline evaluation metric ('relative energy per atom') use the same MACE-MPA-0 model. The observed ~0.5 eV/atom reduction is therefore in part a tautology: the policy is directly optimizing the metric being reported. The manuscript should state this explicitly and place the emphasis on the non-reward metrics (match rate, RMSE, invalid-energy rate), which do show that optimization does not destroy overall generation quality. As written, the energy improvement is over-interpreted.
  3. [§3.3, Eqs. (14)–(15)] The velocity-annealing surrogate scales the exploration noise by the pretrained velocity b_θref(t, x_t), giving state-dependent multiplicative noise. No justification is provided for this choice, and the noise vanishes wherever b_θref is small, potentially inhibiting exploration exactly where the velocity changes sign or is near zero. The learned-annealing results rest on this surrogate; please clarify the rationale and discuss the behavior, or consider a noise schedule independent of the velocity.
minor comments (6)
  1. [Eq. (14)] The notation b_θref is used without arguments; define b_θref(t, x_t) explicitly.
  2. [§3.3] The expression 'st' in the handcrafted annealing rule is ambiguous; write s·t or define it explicitly.
  3. [Fig. 2 / Appendix D] The values of a_s, a_m, a_l are not given in the text. Report the actual numerical values for reproducibility.
  4. [Appendix G] The chosen reference noise scales are denoted a^pos_ref twice; the second should be a^cell_ref.
  5. [Appendix H] The hyperparameter sweep only varies scalar coefficients of a handcrafted (presumably linear-in-time) annealing schedule. The conclusion that 'no configuration recovers' should be limited to that restricted family; a more general sweep over nonlinear schedules would be needed to fully substantiate the claim that the improvement arises from the learned policy.
  6. [Abstract / §1.2] The claim of 'the first application of RL to CSP' should be hedged as 'to our knowledge' to avoid overclaiming.

Circularity Check

2 steps flagged

Energy and cRMSE headline results are the RL rewards by construction; core score-free RL claim is independent.

specific steps
  1. fitted input called prediction [Section 4.2 / Section 2.4 / Appendix E]
    "As the reward, we use a clipped relative energy per atom to prevent outliers from dominating the group-normalized advantages and to keep policy updates stable (see Appendix E). ... The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards."

    The RL reward is the clipped MACE-MPA-0 energy per atom (Appendix E), and the reported evaluation metric is the same MACE-MPA-0 relative energy per atom (Section 2.4). The headline result that RL lowers relative energy by ~0.5 eV/atom is therefore the training objective itself evaluated at convergence, not an independent prediction: the improvement is forced by maximizing the reward. The paper does add independent evidence (RMSE, match rate, held-out test evaluation), so this is partial circularity, not the whole claim.

  2. fitted input called prediction [Section 4.3 / Appendix G]
    "Because the learned annealing schedule depends only on time (and not on composition), and because velocity annealing is known to directly improve structural metrics, we can reinforce the model using a cRMSE-like objective in this setting (see Appendix G for details). ... The reward is defined as r(x^i_{t=1}) = 0.5 − cRMSE^i, where cRMSE^i is the normalized root-mean-square displacement to this best match, with missing matches penalized in the mean using stol=0.5."

    The policy is trained to minimize the cRMSE-like reward r = 0.5 − cRMSE computed against training-set polymorphs, and the paper then reports cRMSE as a key test-set evaluation metric. The cRMSE improvements/recovery reported in Fig. 4 are thus directly aligned with the fitted objective rather than being an independent check. Independent content comes from the METRe metric, the explicit handcrafted-annealing sweep at N_t=10, and robustness across integration step counts.

full rationale

Core methodological claim—policy-gradient RL on velocity-only SI models via the surrogate SDE (Eq. 13)—is not circular: the surrogate policy, GRPO objective, and KL regularization are defined independently of the reward and of the reported metrics. The Girsanov/O(σ²) justification for Eq. (13) is a mathematical-rigor concern (the bound is asserted, not proven, and may fail for the square-root schedule), but it is a correctness issue, not a circularity, and was not scored here. The circularity is limited to the evaluation of the trained policies: two headline 'predictions' (relative energy reduction in Sec. 4.2; cRMSE recovery in Sec. 4.3) use evaluation metrics that are the same as (energy) or directly derived from (cRMSE-like) the RL rewards. Because the paper also reports independent metrics (match rate, RMSE, METRe) and compares against handcrafted baselines, the central derivation retains independent content. Self-citations to OMatG/Höllmer et al. supply pretrained models and the SI framework as external resources, not as a load-bearing uniqueness argument. Overall, partial circularity in the headline metrics: score 6.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central method rests on one nonstandard assumption (the surrogate SDE's validity), one surrogate reward/evaluation loop (MACE), and one unmeasured claim (diversity). All other inputs are standard SI/RL machinery from the cited literature.

free parameters (5)
  • square-root noise scale a = a_s, a_m, a_l selected via validation sweep; exact values not reported
    Controls exploration and performance of the perturbed ODE; chosen per Fig. 2.
  • reference noise scale a_ref = a_s or a_m
    Defines the KL-regularized reference policy; selected by validation reward.
  • reward clipping band and invalid-energy penalty = 3 eV/atom penalty; clipping to ±3 std around group mean
    Hand-chosen in Appendix E to stabilize GRPO advantages.
  • RL hyperparameters (G, PPO epochs, ε, KL weight, denoiser-distillation weight, learning rate) = Tuned via Optuna/Ray Tune; final values not listed
    Selected by validation reward sweep in Appendix F.
  • number of integration steps Nt = 50 for energy RL, 100 for annealing RL
    Reduced from 740/950 to limit cost and variance; this is a key experimental choice.
axioms (5)
  • standard math The SI velocity field trained by Eq. (2) produces correct marginals when integrated by the ODE Eq. (3).
    Standard stochastic-interpolants theory, cited from Albergo et al. (2025).
  • domain assumption The omitted score correction in Eq. (13) leads to a negligible deviation from the true marginal, O(σ²_ref).
    This is the paper's core heuristic; the Girsanov bound is asserted, not proven uniformly, and γ(t)→0 near t=1.
  • domain assumption MACE-MPA-0 energies are a faithful proxy for chemical stability.
    Used both as the RL reward and as the evaluation metric in Section 4.2; no DFT validation is provided.
  • domain assumption Composition conditioning preserves diversity in CSP without explicit diversity rewards.
    Stated in Section 1.2 and 4.1; no diversity metric is reported to verify this.
  • domain assumption A cRMSE-like reward computed against training-set polymorphs transfers to test-set improvements.
    Used in Section 4.3; the polymorph split keeps test compositions out of training, but the reward still relies on structural matching to the training set.

pith-pipeline@v1.3.0-alltime-deepseek · 18439 in / 10278 out tokens · 125950 ms · 2026-08-03T06:01:46.204058+00:00 · methodology

0 comments
read the original abstract

Continuous-time generative models for crystalline materials enable inverse materials design by learning to predict stable crystal structures, but incorporating explicit target properties into the generative process remains challenging. Policy-gradient reinforcement learning (RL) provides a principled mechanism for aligning generative models with downstream objectives but typically requires access to the score, which has prevented its application to flow-based models that learn only velocity fields. We introduce Open Materials Generation with Inference-time Reinforcement Learning (OMatG-IRL), a policy-gradient RL framework that operates directly on the learned velocity fields and eliminates the need for the explicit computation of the score. OMatG-IRL leverages stochastic perturbations of the underlying generation dynamics preserving the baseline performance of the pretrained generative model while enabling exploration and policy-gradient estimation at inference time. Using OMatG-IRL, we present the first application of RL to crystal structure prediction (CSP). Our method enables effective reinforcement of an energy-based objective while preserving diversity through composition conditioning, and it achieves performance competitive with score-based RL approaches. Finally, we show that OMatG-IRL can learn time-dependent velocity-annealing schedules, enabling accurate CSP with order-of-magnitude improvements in sampling efficiency and, correspondingly, reduction in generation time. The OMatG-IRL code is included in a new release of the Open Materials Generation (OMatG) framework available at https://github.com/FERMat-ML/OMatG.

Figures

Figures reproduced from arXiv: 2602.00424 by Philipp Hoellmer, Stefano Martiniani.

Figure 1
Figure 1. Figure 1: Inference-time RL for CSP in velocity-based OMatG-IRL. (a) The deterministic base ODE with pretrained velocity field b θref(t, xt) is augmented with a small noise schedule σref(t), yielding a surrogate SDE that leaves evaluation metrics (e.g., deviation from a reference structure) of the final samples xt=1 virtually unchanged. (b) The frozen surrogate defines a reference policy for KL regularization, while… view at source ↗
Figure 2
Figure 2. Figure 2: Test-set evaluation metrics for score-based SDE and perturbed velocity-based ODE integration of the atomic positions under different noise schedules. Small, medium, and large noise scales are denoted by as, am, and al, respectively. 4.2. Inference-Time Energy Reinforcement To compare the score-based OMatG-IRL approach of Sec￾tion 3.1 to the velocity-based OMatG-IRL approach of Sec￾tion 3.2, we use the publ… view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of validation metrics for score-based and velocity-based OMatG-IRL as a function of RL training iteration, shown for three random seeds of the same setup (Nt = 50). The colored dashed lines indicate the test-set performance of the OMatG-IRL checkpoint selected by the validation optimum. For reference, we also show the test-set performance of the original velocity-annealed OMatG model evaluated wi… view at source ↗
Figure 4
Figure 4. Figure 4: Test-set evaluation metrics for velocity-annealing OMatG and OMatG-IRL as a function of the number of integration steps Nt, highlighting the improved robustness of OMatG-IRL to ag￾gressive time discretization. We verified that a hyperparameter sweep over the hand￾crafted velocity-annealing schedule at the low integration step count of Nt = 10 does not recover the performance achieved by the learned velocit… view at source ↗
Figure 5
Figure 5. Figure 5: Velocity-annealing schedules for the atomic positions and lattice vectors, either learned with OMatG-IRL or obtained from a hyperparameter sweep over a handcrafted schedule at a given number of integration steps. The learned schedules adapt qualitatively differently from handcrafted ones. crystalline materials, and demonstrated its effectiveness on CSP. A key result is that effective policy-gradient RL is … view at source ↗
Figure 6
Figure 6. Figure 6: Evolution of validation metrics for score-based OMatG-IRL with different noise scales as a function of RL training iteration, shown for three random seeds of the same setup (Nt = 50). For reference, we also show the test-set performance of the original velocity-annealed OMatG model evaluated with Nt = 950 integration steps. E. Energy-Based Reward As the energy-based rewards for a GRPO group of structures i… view at source ↗
Figure 7
Figure 7. Figure 7: Test-set evaluation metrics for perturbed velocity-annealing ODE integration of the atomic positions and lattice vectors under different noise schedules and noise scales. The chosen reference noise scales for positions and lattice vectors are denoted by a pos ref and a pos ref , respectively. As the reward for policy-gradient RL, we use a cRMSE-like objective (see Section 2.4). For each generated structure… view at source ↗
Figure 8
Figure 8. Figure 8: Evolution of validation metrics for velocity-annealing OMatG-IRL as a function of RL training iteration, shown for three random seeds of the same setup (Nt = 100). The dashed lines indicate the test-set performance of the OMatG-IRL checkpoint selected by the validation optimum. For reference, we also show the test-set performance of the original velocity-annealed OMatG model evaluated with Nt = 950 integra… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VASP Plugins: Linking the Vienna ab-initio Simulation Package with Python

    cond-mat.mtrl-sci 2026-07 accept novelty 5.5

    A C++/pybind11 shared-memory plugin layer exposes VASP SCF and ionic data as NumPy arrays so Python can modify structure, forces, local potential, and occupancies in place.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    The unit-cell volume is smaller than0.1 ˚A 3

  2. [2]

    Any pairwise atomic distance (accounting for periodic boundary conditions) is smaller than0.5 ˚A

  3. [3]

    The polar sine of the lattice vectors is smaller than10 −3, indicating a nearly degenerate unit cell. C. Sensitivity of OMatG to Integration Steps and V elocity Annealing In Table 1, we show that the performance of the pretrained Trig-SDE-Gamma model in OMatG is largely unchanged when switching between SDE integration via Eq. (7) and ODE integration via E...

  4. [4]

    Øksendal, B.Stochastic Differential Equations

    URL https://www.doi.org/10.1039/ D0SC00594K. Øksendal, B.Stochastic Differential Equations. Springer, sixth edition, 2013. Ong, S. P., Richards, W. D., Jain, A., Hautier, G., Kocher, M., Cholia, S., Gunter, D., Chevrier, V . L., Persson, K. A., and Ceder, G. Python Materials Genomics (py- matgen): A robust, open-source python library for mate- rials analy...

  5. [7]

    11 Open Materials Generation with Inference-Time Reinforcement Learning Sriram, A., Miller, B

    URL https://openreview.net/forum? id=PxTIG12RRHS. 11 Open Materials Generation with Inference-Time Reinforcement Learning Sriram, A., Miller, B. K., Chen, R. T. Q., and Wood, B. M. FlowLLM: Flow Matching for Material Gener- ation with Large Language Models as Base Distribu- tions. InThe Thirty-eighth Annual Conference on Neural Information Processing Syst...

  6. [12]

    The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards

    and clip the resulting values to a band of three standard deviations around the group mean to prevent outliers from dominating the advantages. The reward is then defined as the negative of the (clipped) energy per atom, such that lower energies correspond to higher rewards. Since all structures within a GRPO group share the same composition, we do not nee...

  7. [2018]

    Satorras, V

    doi: https://www.doi.org/10.1126/science.aat2663. Satorras, V . G., Hoogeboom, E., and Welling, M. E(n) Equivariant Graph Neural Networks. InProceedings of the 38th International Conference on Machine Learning,

  8. [2020]

    5555/3495724.3496298

    URL https://dl.acm.org/doi/abs/10. 5555/3495724.3496298. H¨ollmer, P., Egg, T., Martirossyan, M., Fuemmeler, E., Shui, Z., Gupta, A., Prakash, P., Roitberg, A., Liu, M., Karypis, G., Transtrum, M., Hennig, R., Tadmor, E. B., and Martiniani, S. Open Materials Generation with Stochastic Interpolants. InForty-second International Conference on Machine Learni...

  9. [2021]

    Schmidt, J., Marques, M

    URL https://proceedings.mlr.press/ v139/satorras21a.html. Schmidt, J., Marques, M. R. G., Botti, S., and Marques, M. A. L. Recent advances and applications of machine learn- ing in solid-state materials science.npj Computational Materials, 5:83, 2019. URL https://doi.org/10. 1038/s41524-019-0221-0. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and ...

  10. [2023]

    Batatia, I., Benner, P., Chiang, Y ., Elena, A

    URL https://openreview.net/forum? id=li7qeBbCR1t. Batatia, I., Benner, P., Chiang, Y ., Elena, A. M., Kov´acs, D. P., Riebesell, J., Advincula, X. R., Asta, M., Avaylon, M., Baldwin, W. J., Berger, F., Bernstein, N., Bhowmik, A., Bigi, F., Blau, S. M., C˘arare, V ., Ceriotti, M., Chong, S., Darby, J. P., De, S., Della Pia, F., Deringer, V . L., Elijoˇsius...

  11. [2024]

    Yim, J., Campbell, A., Foong, A

    URL https://openreview.net/forum? id=wm4WlHoXpC. Yim, J., Campbell, A., Foong, A. Y . K., Gastegger, M., Jim´enez-Luna, J., Lewis, S., Satorras, V . G., Veeling, B. S., Barzilay, R., Jaakkola, T., and No´e, F. Fast protein backbone generation with SE(3) flow matching, 2023. URLhttps://arxiv.org/abs/2310.05297. Zeni, C., Pinsler, R., Z ¨ugner, D., Fowler, ...

  12. [2025]

    De Breuck, P.-P., Wang, H.-C., Rignanese, G.-M., Botti, S., and Marques, M

    URL https://openreview.net/forum? id=7J1kwZY72h. De Breuck, P.-P., Wang, H.-C., Rignanese, G.-M., Botti, S., and Marques, M. A. L. Generative AI for crys- tal structures: a review.npj Computational Materials, 11:370, 2025. URL https://doi.org/10.1038/ s41524-025-01881-2. Fan, Y . and Lee, K. Optimizing DDPM sampling with shortcut fine-tuning. InProceeding...