Recognition: no theorem link
Gradient-Free Noise Optimization for Reward Alignment in Generative Models
Pith reviewed 2026-05-14 21:34 UTC · model grok-4.3
The pith
ZeNO achieves reward alignment in generative models by optimizing noise inputs using only zeroth-order reward evaluations, without requiring gradients or backpropagation.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
ZeNO formulates noise optimization as a path-integral control problem estimable from zeroth-order reward evaluations alone. When instantiated with an Ornstein-Uhlenbeck reference process, the update connects to Langevin dynamics implicitly targeting a reward-tilted distribution. This enables effective inference-time scaling and strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible.
What carries the argument
ZeNO (Zeroth-order Noise Optimization) is the central mechanism, which treats noise input optimization as a path-integral control problem whose updates are estimated directly from zeroth-order reward evaluations without backpropagation.
If this is right
- Reward alignment extends to deterministic generators that lack stochastic trajectories.
- Non-differentiable rewards become usable, including protein structure generation.
- Inference-time scaling improves performance by increasing computation on noise optimization.
- The same framework applies without modification across multiple generator types and reward functions.
Where Pith is reading between the lines
- The method could apply directly to black-box simulators in domains such as robotics where gradients are unavailable.
- Hybrid schedules that occasionally insert gradient steps might lower variance while retaining the gradient-free core.
- Scaling studies on higher-dimensional models would reveal whether sampling requirements remain practical as problem size grows.
Load-bearing premise
Zeroth-order reward evaluations alone can produce low-variance estimates of the path-integral control updates sufficient to target the reward-tilted distribution without excessive sampling cost or bias.
What would settle it
A side-by-side test on a differentiable diffusion model where ZeNO with a practical sampling budget fails to match the reward improvement of gradient-based baselines would falsify the sufficiency of zeroth-order estimates.
Figures
read the original abstract
Existing reward alignment methods for diffusion and flow models rely on multi-step stochastic trajectories, making them difficult to extend to deterministic generators. A natural alternative is noise-space optimization, but existing approaches require backpropagation through the generator and reward pipeline, limiting applicability to differentiable settings. To address this, here we present ZeNO (Zeroth-order Noise Optimization), a gradient-free framework that formulates noise optimization as a path-integral control problem, estimable from zeroth-order reward evaluations alone. When instantiated with an Ornstein--Uhlenbeck reference process, the update connects to Langevin dynamics implicitly targeting a reward-tilted distribution. ZeNO enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces ZeNO, a gradient-free framework for reward alignment in generative models. It recasts noise optimization as a path-integral control problem whose solution is recovered from zeroth-order reward evaluations alone. For an Ornstein-Uhlenbeck reference process the resulting update is claimed to implicitly follow Langevin dynamics on the reward-tilted distribution, enabling inference-time scaling for both differentiable and non-differentiable generators, including a protein structure generation task.
Significance. If the zeroth-order Monte Carlo estimators can be shown to achieve sufficiently low variance and bias at practical sampling budgets, the approach would meaningfully extend reward alignment to non-differentiable and deterministic generators where back-propagation is infeasible. The path-integral formulation supplies a clean theoretical link to control theory and Langevin dynamics that is not present in prior noise-optimization work.
major comments (2)
- [Path-integral control derivation and Monte Carlo estimator] The Monte Carlo estimator for the path-integral control update (described in the section deriving the ZeNO update) is presented without variance bounds, bias analysis, or sample-complexity guarantees. Standard results for such estimators show variance scaling with noise dimension and reward Lipschitz constant; absent explicit variance-reduction mechanisms, the estimator may require far more reward evaluations than implied by the inference-time scaling claim, especially on the non-differentiable protein task.
- [Experiments] The abstract and experimental claims assert strong performance across generators and reward functions, yet the manuscript supplies no quantitative tables, baselines, variance statistics, or ablation studies. This absence makes it impossible to evaluate whether the zeroth-order updates actually reach the reward-tilted distribution at feasible cost.
minor comments (1)
- [Langevin dynamics connection] Clarify the precise parameterization of the Ornstein-Uhlenbeck process (drift and diffusion coefficients) used to obtain the implicit Langevin connection; the current notation leaves the scaling with time step ambiguous.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback and for recognizing the theoretical connections to path-integral control and Langevin dynamics. We address each major comment below and will incorporate the suggested additions in the revised manuscript.
read point-by-point responses
-
Referee: [Path-integral control derivation and Monte Carlo estimator] The Monte Carlo estimator for the path-integral control update (described in the section deriving the ZeNO update) is presented without variance bounds, bias analysis, or sample-complexity guarantees. Standard results for such estimators show variance scaling with noise dimension and reward Lipschitz constant; absent explicit variance-reduction mechanisms, the estimator may require far more reward evaluations than implied by the inference-time scaling claim, especially on the non-differentiable protein task.
Authors: We agree that the manuscript currently lacks explicit variance bounds, bias analysis, and sample-complexity guarantees for the Monte Carlo estimator. The derivation in the paper establishes the connection to implicit Langevin dynamics on the reward-tilted distribution via the Ornstein-Uhlenbeck reference process, which provides a theoretical justification for the update rule, but does not include finite-sample guarantees. In the revision we will add a dedicated subsection with (i) a bias-variance decomposition based on standard path-integral Monte Carlo results, (ii) empirical variance statistics measured across repeated runs on the reported tasks, and (iii) the exact number of reward evaluations used for the protein structure generation experiment. These additions will clarify the practical sampling budget required and directly address feasibility concerns. revision: yes
-
Referee: [Experiments] The abstract and experimental claims assert strong performance across generators and reward functions, yet the manuscript supplies no quantitative tables, baselines, variance statistics, or ablation studies. This absence makes it impossible to evaluate whether the zeroth-order updates actually reach the reward-tilted distribution at feasible cost.
Authors: We acknowledge that the current version does not include quantitative tables, baseline comparisons, variance statistics, or ablation studies, which limits the ability to assess performance and cost. In the revised manuscript we will add (i) tables reporting mean reward, standard deviation across seeds, and number of reward evaluations for each generator-reward pair, (ii) comparisons against relevant baselines (including gradient-based noise optimization where applicable and standard sampling), and (iii) ablations on the number of Monte Carlo samples and reference-process parameters. These results will demonstrate whether the zeroth-order updates achieve the claimed reward-tilted behavior at practical cost. revision: yes
Circularity Check
No significant circularity; derivation is self-contained
full rationale
The paper formulates noise optimization as a path-integral control problem and shows that, under an Ornstein-Uhlenbeck reference, the resulting zeroth-order update implicitly targets the reward-tilted distribution via a connection to Langevin dynamics. These links are derived from the control formulation and stochastic process definitions rather than by fitting parameters to the target quantities or by self-citation chains. Zeroth-order reward evaluations are used directly as the estimator without renaming fitted inputs as predictions. No load-bearing step reduces to its own inputs by construction, and the central claims rest on the mathematical equivalence shown in the path-integral setup rather than on external self-references.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Noise optimization can be cast as a path-integral control problem whose solution is estimable from zeroth-order reward samples.
Reference graph
Works this paper leans on
-
[1]
Training diffusion models with reinforcement learning
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024
work page 2024
-
[2]
Diffusion posterior sampling for general noisy inverse problems
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. InInternational Conference on Learning Representations, 2023
work page 2023
-
[3]
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. InThe Twelfth International Conference on Learning Repre- sentations, 2024
work page 2024
-
[4]
J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y . Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning–based protein sequence design using proteinmpnn.Science, ...
work page 2022
-
[5]
ReNO: Enhancing one-step text-to-image models through reward-based noise optimization
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. ReNO: Enhancing one-step text-to-image models through reward-based noise optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
work page 2024
-
[6]
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning.Transactions on Machine Learning Research, 2023
work page 2023
-
[7]
Mean Flows for One-step Generative Modeling
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling.arXiv preprint arXiv:2505.13447, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[8]
Geneval: An object-focused framework for evaluating text-to-image alignment
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023
work page 2023
-
[9]
Symbolic music generation with non-differentiable rule guided diffusion
Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli Shama Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proc...
work page 2024
-
[10]
Diffusion fine-tuning via reparameterized policy gradient of the soft q-function
Hyeongyu Kang, Jaewoo Lee, Woocheol Shin, Kiyoung Om, and Jinkyoo Park. Diffusion fine-tuning via reparameterized policy gradient of the soft q-function. InThe Fourteenth International Conference on Learning Representations, 2026
work page 2026
-
[11]
H. Kappen. Path integrals and symmetry breaking for optimal control theory.J Stat Mech- Theory E, 2005, 05 2005
work page 2005
-
[12]
Byungjun Kim, Soobin Um, and Jong Chul Ye. Diverse text-to-image generation via contrastive noise optimization.arXiv preprint arXiv:2510.03813, 2025
-
[13]
Free2guide: Training-free text-to-video alignment using image lvlm
Jaemin Kim, Bryan Sangwoo Kim, and Jong Chul Ye. Free2guide: Training-free text-to-video alignment using image lvlm. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17920–17929, October 2025
work page 2025
-
[14]
Test-time alignment of diffusion models without reward over-optimization
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. InThe Thirteenth International Conference on Learning Representations, 2025
work page 2025
-
[15]
Pick-a-pic: An open dataset of user preferences for text-to-image generation
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 10
work page 2023
-
[16]
PCPO: Proportionate credit policy optimization for preference alignment of image generation models
Jeongjae Lee and Jong Chul Ye. PCPO: Proportionate credit policy optimization for preference alignment of image generation models. InThe Fourteenth International Conference on Learning Representations, 2026
work page 2026
-
[17]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic- level protein structure with a language model.Science, 379(6637):1123–1130, 2023
work page 2023
-
[18]
Flow-GRPO: Training flow matching models via online RL
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di ZHANG, and Wanli Ouyang. Flow-GRPO: Training flow matching models via online RL. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026
work page 2026
-
[19]
Xiao, Carles Domingo-Enrich, Weiyang Liu, and Dinghuai Zhang
Zhen Liu, Tim Z. Xiao, Carles Domingo-Enrich, Weiyang Liu, and Dinghuai Zhang. Value gradient guidance for flow matching alignment. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026
work page 2026
-
[20]
Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang
Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang. Efficient diversity- preserving diffusion alignment via gradient-informed GFlownets. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025
work page 2025
-
[21]
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[22]
ORIGEN: Zero-shot 3d orientation grounding in text-to-image generation
Yunhong Min, Daehyeon Choi, Kyeongmin Yeo, Jihyun Lee, and Minhyuk Sung. ORIGEN: Zero-shot 3d orientation grounding in text-to-image generation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026
work page 2026
-
[23]
Stochastic differential equations
Bernt Øksendal. Stochastic differential equations. InStochastic differential equations: an introduction with applications, pages 38–50. Springer, 2003
work page 2003
-
[24]
Fast high-resolution image synthesis with latent adversarial diffusion distillation
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024
work page 2024
-
[25]
Adversarial diffusion distillation
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. InEuropean Conference on Computer Vision, pages 87–103. Springer, 2024
work page 2024
-
[26]
Christoph Schuhmann. Improved aesthetic predictor. https://github.com/ christophschuhmann/improved-aesthetic-predictor, 2022. GitHub repository
work page 2022
-
[27]
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models.arXiv preprint arXiv:2303.01469, 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[28]
Df-gan: A simple and effective baseline for text-to-image synthesis
Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022
work page 2022
-
[29]
Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review.arXiv preprint arXiv:2407.13734, 2024
-
[30]
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine- tuning of continuous-time diffusion models as entropy-regularized control.arXiv preprint arXiv:2402.15194, 2024
-
[31]
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 7 2011
work page 2011
-
[32]
Natural evolution strategies.Journal of Machine Learning Research, 15(27):949–980, 2014
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies.Journal of Machine Learning Research, 15(27):949–980, 2014. 11
work page 2014
-
[33]
Dongyeop Woo, Marta Skreta, Seonghyun Park, Kirill Neklyudov, and Sungsoo Ahn. Rieman- nian meanflow, 2026
work page 2026
-
[34]
Improved distribution matching distillation for fast image synthesis
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455–47487, 2024
work page 2024
-
[35]
One-step diffusion with distribution matching distillation
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613–6623, 2024
work page 2024
-
[36]
TaeHoon Yoon, Yunhong Min, Kyeongmin Yeo, and Minhyuk Sung. $\psi$-sampler: Initial particle sampling for SMC-based inference-time reward alignment in score models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. 12 A Path-Integral Control Formulation [11] We briefly derive the optimal control form used in the main tex...
work page 2026
-
[37]
A teddy bear wearing holding a whiteboard,
"A teddy bear wearing holding a whiteboard, "ZeNO" written on it."
-
[38]
A beautifully decorated birthday cake with smooth blue icing, the letters
"A beautifully decorated birthday cake with smooth blue icing, the letters "Happy 30th Jake" elegantly spelled out on top, surrounded by colorful candles and sparkling decorations, set on a white tablecloth in a cozy living room."
-
[39]
Studio shot of intricate shoe sculptures crafted from vibrant colored wires, with the text
"Studio shot of intricate shoe sculptures crafted from vibrant colored wires, with the text "rte" prominently displayed on a clean, white background."
-
[40]
A campfire in a dense forest, with smoke curling upwards and forming the words
"A campfire in a dense forest, with smoke curling upwards and forming the words "Send Help" against a twilight sky, partially obscured by the treetops."
-
[41]
An astronaut proudly displays the
"An astronaut proudly displays the "Mission Success" patch on their spacesuit, standing against the backdrop of a stunning Earthrise over the lunar horizon, with the vast cosmos stretching beyond." 16 CFG scaleWe used a CFG scale of 0.0 for SDXL-Turbo and DMD2, and 8.0 for LCM, following the default recommended settings for each model. F.2 Protein Structu...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.