REVIEW 4 major objections 4 minor 10 cited by
This paper claims that a single VLM reward model, queried with a task identity and an evaluation dimension, can supervise one mask-guided editor across image fill, image extend, object removal, and text rendering without any task-specific s
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A single VLM-based reward model, conditionable on task and evaluation dimension, drives multi-task RL that improves mask-guided image editing across four tasks without task-specific SFT.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A genuinely new reward-model construction, but the paper hides the architectural details that make or break the central claim. the 4 major comments →
OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
OneReward is a multi-task reinforcement-learning framework built around a single VLM reward model. For each training sample, two images are generated from the same condition: a reference image from the frozen pre-trained model with a full denoising trajectory, and an evaluation image from the trainable policy with partial denoising and direct latent prediction, following ReFL. The reward model receives both images plus a query naming the task and the evaluation dimension, and returns the probability of the token 'Yes', meaning the first image is better. The policy objective is J(θ) = max(0, λ − P(y+ | πθ(c), πref(c), q)), so the policy is pushed to outperform the reference under each criteri
What carries the argument
Central object: OneReward, a pairwise vision-language reward model built on a pre-trained VLM. It converts any evaluation into a comparison question: given task id s_k, dimension e, and optionally the prompt, a query template asks whether the first image is better than the second; the probability of the 'Yes' token is the reward. Because task and dimension are in the query, one model covers the four tasks and five evaluation dimensions. The policy update uses the ReFL-style truncated backprop: the evaluation image is produced by partial denoising plus one-step latent prediction, and gradients flow through the reward model into that single step. A frozen reference model (or, in the dynamic va
Load-bearing premise
The method assumes the pre-trained text-to-image base model already has a working pathway that feeds the binary mask into generation, because the paper never specifies how the mask enters the architecture.
What would settle it
On a held-out benchmark with simple, clearly delineated masks, measure whether pixels outside the mask change between the source image and the final output. If a non-trivial fraction of edits alter the unmasked region, then the policy did not learn mask fidelity from the RL reward, and the central claim of mask-guided multi-task editing would be contradicted.
If this is right
- A single VLM can act as the reward model for many tasks at once, so adding a task or evaluation dimension reduces to writing a new query template rather than training a new reward model.
- Multi-task RL can replace task-specific SFT for mask-guided editing, so a pre-trained text-to-image model can become a unified editor through reward feedback alone.
- Averaging dimension-wise rewards steers one policy toward balanced improvement on conflicting objectives, which is what lets object-removal quality and image-fill generation coexist in one model.
- The dynamic EMA-reference variant removes the separate frozen reference model and gives the policy a progressively harder baseline, achieving competitive results with lower memory cost.
- The recipe transfers to an open-source base: FLUX Fill [dev] improved with OneReward becomes a new public baseline for unified mask-guided editing.
Where Pith is reading between the lines
- An implicit consequence: the reward model's reported accuracies of roughly 72-85% mean the policy is trained on moderately noisy win/lose labels; a direct test would be to run OneReward on a fifth editing task not in the training set and measure reward-model accuracy before policy training.
- The no-SFT claim is architecture-dependent: if the base model only became mask-aware during RL, the paper would need to explain how gradients through the reward model teach the mask pathway. A useful ablation would compare final outputs with masks replaced by random masks or all-one masks; if outputs are unchanged, the mask is not load-bearing.
- The query-conditioned reward mechanism may transfer to other conditional generation domains, such as video inpainting/outpainting or control-to-image, where tasks share a conditioning format but differ in objectives; this is a testable extension the paper does not run.
- A controlled comparison with a task-specific SFT baseline trained on the same preference dataset would isolate what multi-task RL adds over supervised fine-tuning; the paper compares against the base model and external competitors but not against an SFT baseline on identical data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OneReward, a single VLM-based reward model that is conditioned on task identity and evaluation dimension, and uses it to train a mask-guided image generation model (Seedream 3.0 Fill) by multi-task reinforcement learning directly from a pre-trained text-to-image model, without task-specific SFT. The policy objective (Eq. 5) maximizes the reward model's Yes-token probability for the policy output versus a reference output. Experiments cover image fill, image extend, object removal, and text rendering, and report that the resulting model outperforms commercial and open-source competitors on most dimensions in human evaluation. The authors also release a FLUX Fill [dev][OneReward] variant.
Significance. If the central claim holds, the paper makes a useful contribution: a single reward model that can provide multi-task, multi-dimensional preference signals, avoiding per-task SFT and separate reward models per evaluation criterion. The use of independent human GSB evaluation against the base model (Sec. 5.2.2, Fig. 7) is an appropriate, non-circular way to validate the trained policy, and the open-sourcing of a FLUX-based variant is a concrete benefit to the community. However, the evidence as presented has several gaps: the mask-conditioning mechanism of the base model is never specified, the reward model's input omits the source image and mask, and the headline human-evaluation comparisons lack statistical support. These issues are central to the paper's claims and need to be resolved before the results can be fully assessed.
major comments (4)
- [§5.1, §4.4, Eq. (5)] The mask-conditioning pathway is never specified. Section 3.1 defines conditioning c as including "text or binary mask," but Section 5.1 identifies the base model as "Seedream 3.0" text-to-image model. If the base network has no mask channel/encoder, then the binary mask M is not fed into πθ, and the RL loss in Eq. (5) has no gradient path that depends on M; the policy cannot learn mask fidelity. The data pipeline in Section 4.2 already presupposes a base generator that accepts (Isrc,M,P), so this is not merely a missing hyperparameter. Please state explicitly how M is injected into the architecture (or confirm that Seedream 3.0 already supports mask conditioning) and provide evidence that gradients flow from the reward through the mask-dependent path.
- [§4.3, Eq. (2), Fig. 5] The reward model input is defined as a pair of images (xw,xl) and query q = Φ(sk,e,P), with no source image Isrc or mask M. Yet dimensions such as consistency, structure, and removal quality are defined relative to the original image and the edit region (Section 4.2). Without (Isrc,M), the VLM cannot tell whether texture is consistent with surrounding content or whether a removed object is truly gone. If the reward model does receive the source and mask, Eq. (2) and Fig. 5 must be corrected; otherwise the reward signal is not grounded in the edit region and the claimed multi-dimensional preferences are not being learned.
- [§5.2.2, Table 2, Fig. 1] The headline comparisons lack statistical support. Usability rates and MOS in Table 2 are reported without confidence intervals, significance tests, or inter-rater agreement; Fig. 1 shows only point estimates. Several margins are small (e.g., Image Extend w Prompt usability 64.72 vs. 63.13; text alignment 4.26 vs. 4.23), and the benchmark is only 130/100/200 images rated by 40 participants. Report confidence intervals (e.g., bootstrap) and paired significance tests for the claimed victories, and describe how images and participants were selected.
- [§4.3, Table 1, §4.4] Reward model accuracy is 70–85% (Table 1), with most dimensions in the low-to-mid 70s. Since Eq. (5) uses the reward model's token probability as the training signal, the sensitivity of the final policy to reward-model noise is a central concern. The paper does not report calibration, agreement with human labels at the pair level, or any ablation of reward-model quality. At minimum, discuss how the 72–75% accuracy on consistency/structure/aesthetics affects the reliability of the learned reward and show that the trained reward model's judgments correlate with held-out human preference on the evaluation benchmark.
minor comments (4)
- [Throughout] Typographical errors: "Seeddream" (Sec. 5.1), "mulit-task" (Abstract/Intro), "trainging" (Sec. 3.1), "mathching" (Conclusion). Please proofread.
- [Fig. 7] The Good/Same/Bad percentages are not accompanied by sample sizes or confidence intervals. Also clarify that "Seedream 3.0 [OneReward]" and "Seedream 3.0 [Base]" are the same base model with and without reward-guided training.
- [Alg. 1/2] Algorithm 1 initializes and updates an EMA model but never uses it; Algorithm 2 reuses the EMA as the reference. Please clarify the role of EMA in Alg. 1 and report quantitative results for the dynamic variant rather than only a schematic (Fig. 10).
- [§5.1] The dataset description says a "large number" of preference pairs were generated and annotated, but no dataset size, number of annotators, or inter-annotator agreement is given. These details are important for reproducibility and for interpreting reward-model accuracy.
Circularity Check
No significant circularity: the reward model is trained on independent human labels and the final policy is evaluated by separate human raters.
full rationale
The paper's claimed derivation chain is a standard RLHF pipeline: (1) collect human preference pairs per task and evaluation dimension (Sec. 4.2); (2) train a VLM reward model with cross-entropy on those pairs (Eq. 3); (3) freeze that reward model and optimize a diffusion/flow policy with Eq. 5, using the reward model's Yes-token probability as reward; (4) evaluate the resulting policy with fresh human raters (Sec. 5.2.2, Fig. 1, Fig. 7). The reward-model training objective (Eq. 3) and the policy objective (Eq. 5) are not the same functional: Eq. 3 is supervised on human-annotated winner/loser pairs, while Eq. 5 is an RL reward signal on policy/reference generations with a margin λ. The final model's quality is not measured by the reward model's own outputs; the headline comparisons use independent human evaluation. The paper's use of its own Seedream 3.0 as a base model and its citations to ByteEdit, DanceGRPO, and related work are contextual and not load-bearing mathematical premises. The only notable weakness is an unstated architectural assumption: Section 5.1 says 'We adopt Seeddream 3.0 as the text-to-image base model' and Section 3.1 defines conditioning c as including 'text or binary mask', but the exact mechanism by which the binary mask enters the base model's forward pass is not specified. That is a correctness/engineering gap, not a circular derivation, because no equation or definition in the paper makes the mask pathway equivalent to the outcome it is supposed to explain. Therefore no circularity steps are identified.
Axiom & Free-Parameter Ledger
free parameters (4)
- lambda (reward upper bound) =
not reported
- task sampling probabilities p_k =
fill 0.5, extend 0.25, removal 0.25
- partial denoising timestep interval [t1, t2] =
not reported
- reward model fine-tuning LR and batch size =
lr=1e-6, batch=16
axioms (5)
- standard math Flow matching / rectified flow objective is a valid generative model for Seedream 3.0
- domain assumption A VLM fine-tuned on pairwise comparisons can provide a reliable, dimension-specific reward signal across diverse editing tasks
- domain assumption Human Best-of-N/Worst-of-N per-dimension annotations yield consistent, high-quality preference pairs
- ad hoc to paper The base text-to-image model can be adapted to mask-guided editing purely through RL without task-specific SFT
- domain assumption Truncated backprop through a single denoising step (ReFL-style) yields a useful gradient for minimizing the reward loss
invented entities (1)
-
OneReward
no independent evidence
Cite this review
Pith. "Pith review of OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning." pith.science (2026). https://pith.science/paper/ZRV3FT7L
@misc{pith2026250821066,
author = {Pith},
title = {Pith review of: OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZRV3FT7L}},
note = {Machine review of arXiv:2508.21066}
}
read the original abstract
In this paper, we introduce OneReward, a unified reinforcement learning framework that enhances the model's generative capabilities across multiple tasks under different evaluation criteria using only \textit{One Reward} model. By employing a single vision-language model (VLM) as the generative reward model, which can distinguish the winner and loser for a given task and a given evaluation criterion, it can be effectively applied to multi-task generation models, particularly in contexts with varied data and diverse task objectives. We utilize OneReward for mask-guided image generation, which can be further divided into several sub-tasks such as image fill, image extend, object removal, and text rendering, involving a binary mask as the edit area. Although these domain-specific tasks share same conditioning paradigm, they differ significantly in underlying data distributions and evaluation metrics. Existing methods often rely on task-specific supervised fine-tuning (SFT), which limits generalization and training efficiency. Building on OneReward, we develop Seedream 3.0 Fill, a mask-guided generation model trained via multi-task reinforcement learning directly on a pre-trained base model, eliminating the need for task-specific SFT. Experimental results demonstrate that our unified edit model consistently outperforms both commercial and open-source competitors, such as Ideogram, Adobe Photoshop, and FLUX Fill [Pro], across multiple evaluation dimensions. Code and model are available at: https://one-reward.github.io
Figures
Forward citations
Cited by 10 Pith papers
-
RewardHarness: Self-Evolving Agentic Post-Training
RewardHarness self-evolves a tool-and-skill library from 100 preference examples to reach 47.4% accuracy on image-edit evaluation, beating GPT-5, and yields stronger RL-tuned models.
-
Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling
DiNa-LRM introduces a diffusion-native latent reward model using a noise-calibrated Thurstone likelihood on noisy states, matching VLM performance at lower compute in image alignment and preference optimization.
-
Leveraging Verifier-Based Reinforcement Learning in Image Editing
Edit-R1 trains a CoT-based reasoning reward model with GCPO and uses it to boost image editing performance over VLMs and models like FLUX.1-kontext via GRPO.
-
Leveraging Verifier-Based Reinforcement Learning in Image Editing
Edit-R1 builds a CoT-based reasoning reward model (RRM) via SFT and GCPO, then applies it with GRPO to improve image editing models such as FLUX.1-kontext.
-
DDA-Thinker: Decoupled Dual-Atomic Reinforcement Learning for Reasoning-Driven Image Editing
DDA-Thinker decouples planning from generation and applies dual-atomic RL with checklist-based rewards to boost reasoning in image editing, yielding competitive results on RISE-Bench and KRIS-Bench.
-
Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
JRM jointly optimizes preference ranking and language modeling on a shared vision-language backbone, yielding an efficient discriminative reward model that outperforms both discriminative and generative baselines on i...
-
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Using an LLM to generate prompt-specific visual rubrics and grade each criterion independently gives a more interpretable reward that improves text-to-image model alignment beyond composite and learned scalar rewards.
-
Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
UniWorld-V2 applies policy optimization via DiffusionNFT and MLLM logit feedback with group filtering to reach state-of-the-art scores of 4.49 on ImgEdit and 7.83 on GEdit-Bench while remaining model-agnostic.
-
Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
Dynamic-TreeRPO replaces independent trajectory sampling with a tree-structured search using dynamic noise intensities and integrates SFT into RL via a weighted Progress Reward Model to achieve better semantic consist...
-
Edit-GRPO: A Locality-Preserving Policy Optimization Framework for Image Editing
Edit-GRPO decouples editing and preservation objectives via region-specific signals in a policy optimization framework to improve locality in image editing tasks.
Reference graph
Works this paper leans on
-
[1]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...
-
[6]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,
-
[7]
Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300,
-
[8]
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b. Hongzu Su, Lichao M...
Pith/arXiv arXiv 2010
-
[10]
Dancegrpo: Unleashing grpo on visual generation
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818,
-
[11]
Inst-inpaint: Instructing to remove objects with diffusion models
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint: Instructing to remove objects with diffusion models. arXiv preprint arXiv:2304.03246,
-
[12]
On- linevpo: Align video diffusion model with online video-centric preference optimization
Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Ji, Xuefeng Xiao, Weilin Huang, and Kai Han. On- linevpo: Align video diffusion model with online video-centric preference optimization. arXiv preprint arXiv:2412.15159,
-
[1952]
Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report. arXiv preprint arXiv:2504.11346 , 2025a. Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundari...
-
[2022]
Flow-grpo: Training flow matching models via online rl
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025a. Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video ...
-
[2023]
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human prefer- ence learning for image and video generation. arXiv preprint arXiv:2412.21059,
-
[2024]
Flow matching for generative modeling
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,
-
[2025]
Training diffusion models with reinforcement learning
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.