{"id":"206e8261-6e61-4450-903b-0f0a48042bc9","arxiv_id":"2608.06794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PAST reduces the computational cost of RL fine-tuning for diffusion models by adding an intrinsic denoising reward and adaptively terminating episodes when denoising and semantic alignment are achieved.","lead":"This paper introduces PAST, a plugin for reinforcement-learning fine-tuning of text-to-image diffusion models that adds a denoising-progress reward and adaptively shortens training episodes. It reports cutting training time by up to 66.7% while improving preference optimization quality by up to 29.5%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 11's min-based termination rule is internally inconsistent with the 'both metrics' requirement and can truncate under-denoised episodes; without code or a max-based ablation, the central efficiency/quality claim is not established.","rationale":"The reader's conditional verdict is well aligned with the main uncertainty. The central claim requires that adaptive termination stops only when the image is adequately denoised and semantically aligned. The paper's own Eq. 11 appears to contradict this requirement by taking the min of the two completion steps rather than the max, which would terminate at the first satisfied criterion. This is not merely a proxy-quality concern: even with perfect denoising and semantic estimators, the formula as written can return under-denoised images in any episode where the two signals do not saturate at the same step. The empirical tables show aggregate outcomes but no ablation or logged distribution of the two termination signals, so the reported efficiency/quality gains cannot be attributed to the stated mechanism without additional evidence. The proof of Theorem 2.1 is also unsound in its current form, but since the experimental central claim is empirical, the min/max issue is the more load-bearing technical concern. No code is released, which makes the formula issue irresolvable from the paper alone. These are addressable issues, so a conditional rather than a reject verdict remains appropriate.","tokens_in":19486,"tokens_out":9000,"duration_ms":79660,"concrete_test":"Request the training code (or, failing that, ask the authors to instrument their pipeline) and log t_s_len and t_d_len for every episode on the Table 2 setting. Re-run three configurations: the published min rule, the max rule, and a fixed 31-step schedule. If the max rule preserves or improves AES/PS/CLIP while the min rule matches the reported numbers, then the published formula is a typo and the headline result depends on unreported code; if the max rule produces different metrics, the 'both metrics' claim and the quality/efficiency trade-off need to be re-evaluated. Also report the fraction of episodes with t_s_len < t_d_len to quantify how often the min rule is the active truncation signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 2.3.3 defines t_d_len as the first step where alpha^d_t approaches 1 and t_s_len as the first step where alpha^s_t approaches 0 with vanishing variation, then sets t_len = min(t_s_len, t_d_len). Since the abstract and surrounding text require both noise and semantic criteria to be satisfied, the correct joint stopping time is max(t_s_len, t_d_len), not min. With min, the episode terminates as soon as either signal fires. Whenever semantic alignment saturates before full denoising (or vice versa), the returned image is taken from an incomplete denoising step. Tab. 2 reports only the average adaptive length (31) and aggregate metrics; it does not report the joint distribution of t_s_len and t_d_len or an ablation using max. The same section's theoretical justification is also not dependable: the proof of Theorem 2.1 invokes a soft-Q form with inconsistent alpha indices, drops R_ext in the recursion, and does not establish convergence of Eq. 5. But the min/max inconsistency alone is enough to make the central efficiency/quality claim conditional on implementation details that are not currently disclosed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAST, a plugin for reinforcement-learning fine-tuning of text-to-image diffusion models. PAST adds an intrinsic reward based on the squared distance between the current denoised state and the initial noise, which is claimed to accelerate denoising and improve exploration, and it introduces an adaptive episode-termination mechanism that stops training episodes early based on a Denoise-Aware Model (DAM) and a cross-attention semantic-alignment metric. The authors report up to 66.7% computational savings and up to 29.5% preference-optimization improvement when PAST is added to DDPO and DPOK, and they provide a theorem intended to justify the intrinsic reward. Experiments span multiple backbones (SDv1.4, SDv1.5, SDv2.1, SDXL, SD3.5), reward functions (Aesthetic Score, PickScore, JPEG compressibility, incompressibility), and evaluation metrics including human and large-vision-model scoring.","tokens_in":19800,"tokens_out":3490,"duration_ms":30460,"significance":"If the empirical claims are correct, PAST addresses a practically important bottleneck: the high computational cost of RL fine-tuning for text-to-image diffusion models. The paper's strengths include a model-agnostic design, a broad experimental evaluation across backbones, reward functions, and datasets, and an explicit effort to measure diversity (LPIPS, TCE) in addition to preference scores. The reported wall-clock improvements (Tab. 1) are substantial and, if reproducible, would make the method useful to practitioners. However, the theoretical justification (Theorem 2.1) is not a convergence guarantee in its current form, and the termination rule in Eq. (11) is inconsistent with the stated requirement that both denoising and semantic criteria be satisfied. These issues are load-bearing for the central claims of efficiency and quality preservation, so the manuscript requires substantive revision before the result can be accepted.","major_comments":[{"comment":"The termination rule is inconsistent with the stated requirement. The text says 'the final episode length needs to satisfy both noise and semantic requirements,' and the abstract says termination occurs when 'both metrics satisfy generation requirements.' However, Eq. (11) defines t_len = min(t_s_len, t_d_len), which stops the episode as soon as either signal fires. If semantic alignment saturates before full denoising (or vice versa), the min rule returns an image from an incomplete denoising step. The correct joint stopping time for 'both' is t_len = max(t_s_len, t_d_len). Table 2 reports only the average adaptive length (31) and aggregate metrics; it does not report the joint distribution of t_s_len and t_d_len, nor an ablation comparing min versus max. Since the efficiency claim depends on early termination and the quality claim depends on not truncating under-denoised episodes, the authors must either justify min as a deliberate early-stop heuristic with evidence that quality is preserved in the cases where one signal lags, or change the rule to max and re-run the experiments.","section":"Sec. 2.3.3, Eq. (11)"},{"comment":"Theorem 2.1 is stated as an expression for the optimal policy, but the text immediately after the theorem says 'The theorem guarantees the convergence of Eq. 5.' This is a category error: a closed-form expression for the optimal policy, even if correct, does not by itself guarantee convergence of any training algorithm to that policy. In the proof, the alpha indices are inconsistent across lines: alpha_{t-1} appears in the exponent of Eq. (6), alpha_{t-2} in the soft-Q recursions, and alpha_{t-3} in the expansion. The recursion also drops the extrinsic reward R_ext: after the second line, the term exp{R_int(x_{t-1})/alpha_{t-2}} is factored out, but R_ext(x_{t-1}) disappears entirely, and the final factorization into Q_ext^* and Q_int^* is not derived from the preceding algebra. As written, the proof does not establish the theorem, and the theorem does not establish convergence. The authors should either repair the statement and proof with consistent indexing and a complete derivation, or remove the convergence claim and present the intrinsic reward as a heuristic whose effectiveness is demonstrated empirically.","section":"Sec. 2.2, Theorem 2.1 and proof"},{"comment":"The adaptive termination mechanism relies on two proxies whose validity is not demonstrated. First, the DAM classifier is trained with labels derived from the endpoints x_T and x_0 of the same denoising trajectories that define the intrinsic reward, and the neighborhood range nu and endpoint sample count n are tunable hyperparameters. The paper does not report DAM accuracy, calibration, or any comparison of its output against a ground-truth noise level on held-out trajectories, so it is unclear whether alpha_t^d reliably indicates denoising completion for intermediate states. Second, the semantic alignment metric alpha_s^t is defined as the average pairwise cosine distance among cross-attention maps in noun-attribute groups. This is a plausible proxy, but no evidence is given that this distance correlates with actual prompt-image semantic alignment (e.g., against CLIP score or human ratings on intermediate steps). Given that the termination rule uses both signals to decide episode length, the paper should include a validation study of both DAM and the attention-based alignment metric, for example by reporting the correlation of alpha_s^t with a reference alignment score across denoising steps.","section":"Sec. 2.1.3, Eq. (4) and Sec. 2.3.1, Eqs. (7)-(8)"}],"minor_comments":[{"comment":"There are several typos: 'seciton' should be 'section,' 'epsoide' should be 'episode,' and the section heading 'Sementic alignment evaluator' should be 'Semantic alignment evaluator.'","section":"Section 2"},{"comment":"The label 'None-AttrGroups' in Figure 1 should be 'Noun-AttrGroups' to match the terminology used in the text.","section":"Figure 1"},{"comment":"Equation (5) contains a formatting error in the KL term: 'KL(p_theta(·))||p_pre(·))' has an extra closing parenthesis and is missing the second argument of the KL divergence. Please fix the notation.","section":"Sec. 2.2, Eq. (5)"},{"comment":"Table 3 lists hyperparameters but omits the DAM neighborhood range nu and the endpoint sample count n, both described as tunable in Sec. 2.1.3. Please report the values used in the experiments, since the termination behavior is sensitive to them.","section":"Table 3"},{"comment":"The statement that DeepSeek-V3 and Qwen2.5-32B-Instruct 'achieved 100% consistency' in noun-attribute decomposition is vague. Please specify what the 100% consistency is measured against (e.g., human annotations, agreement between the two models) and how many prompts were tested.","section":"Sec. 3.2"},{"comment":"The claim that CLIP 'struggle[s] to provide accurate real-time assessments of semantic alignment during denoising' is supported by citation [7], but [7] is 'Diffusion models beat GANs on image synthesis' and does not address CLIP-based alignment evaluation. Please cite an appropriate reference for this claim.","section":"Section 2.3.1"}],"recommendation":"major_revision","confidential_remarks":"The min/max termination issue is exactly the kind of detail that determines whether the reported 66.7% speedup is a genuine early-stopping result or an artifact of premature truncation. I recommend asking the authors to provide the joint statistics of t_s_len and t_d_len and an ablation with max-based stopping. The theoretical section also needs to be rewritten: presenting a policy-form expression as a convergence guarantee will not survive review. The paper otherwise has a strong empirical component, and I would be willing to review a revised version. I also suggest the editor ask for the code or at least a detailed pseudo-code for the termination rule, since the ambiguity between min and max cannot be resolved from the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PAST has a plausible and well-tested efficiency idea for RL fine-tuning of diffusion models, but the paper undermines itself with a wrong termination rule (min vs max) and a theorem that doesn't prove what it claims. The empirical results are worth a look, but the manuscript as written isn't solid enough to trust without those fixed.\n\nWhat's actually new: the combination of an intrinsic reward based on distance to the initial noise with adaptive episode termination using a lightweight denoising classifier and cross-attention similarity is a neat, practical contribution. The plug-and-play framing is attractive: you bolt this onto DDPO/DPOK and see 60%+ wall-clock savings with modest reward gains, across several backbones and reward models. The ablations (Fig 5, Tab 2) give some evidence that both the intrinsic reward and the adaptive termination matter. That's real work and plausibly useful for practitioners.\n\nThe soft spots are serious. First, Eq 11 uses min(t_s_len, t_d_len) for the joint stopping time, but the text requires both denoising and semantic conditions. The correct joining is max. With min, training terminates as soon as one signal fires, which can cut episodes before the image is actually ready. This is not a cosmetic typo: it changes the algorithm and the reported step distribution. There's no ablation with max, and the paper doesn't report the joint distribution of t_s and t_d, so we can't tell how often the min actually goes wrong. Second, Theorem 2.1 is not a convergence guarantee. The proof gives a Boltzmann form for the soft-Q policy but never shows Eq 5 attains an optimum or that the iterates converge. The algebra also has inconsistent alpha indices and drops R_ext in the recursion. That section needs to be rewritten honestly or removed. Third, the termination hyperparameters (nu, n, and the thresholds for alpha^d -> 1 / alpha^s -> 0) are not given in the main text, and no code is released, so the central efficiency claim is hard to reproduce. Finally, the novelty overlap with the authors' own ref [55] is not clearly addressed.\n\nFor whom: researchers working on efficient RL fine-tuning of diffusion models. The idea is worth engaging, and a serious referee could help the authors fix the termination rule and tighten the theory. I'd say send it to review, but reject in current form, not desk-reject.","headline":"Plausible efficiency idea for RL fine-tuning of diffusion models undermined by a wrong termination rule (min vs max) and an overstated theorem.","tokens_in":20268,"tokens_out":3480,"would_cite":false,"duration_ms":29041,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAST, a plug-in for RL fine-tuning of diffusion models, claims up to 66.7% savings in training time and up to 29.5% better preference optimization by terminating episodes adaptively.","keywords":["Text to Image Generation","Computational Efficiency Optimization","Diffusion Models","Reinforcement Learning Fine-tuning","Intrinsic Reward","Adaptive Episode Termination","Cross-Attention Alignment","Reward Sparsity"],"falsifier":"Run PAST on a prompt set where the decisive visual content, such as object identity, count, or spatial relation, first appears in the last few denoising steps. If the denoise or alignment proxies fire early on such prompts, adaptive episodes will terminate before that content is fixed, and quality metrics will measurably fall relative to full-length episodes.","tokens_in":1601,"feed_emoji":"⚡","tokens_out":5538,"duration_ms":77811,"temperature":0.7,"pith_summary":"PAST is a plugin for reinforcement-learning fine-tuning of text-to-image diffusion models. It claims that current RL methods waste most of their training compute on denoising steps whose marginal quality gain has already saturated, and that this waste can be avoided without hurting the final reward. The paper proposes an intrinsic reward based on distance from initial noise to relieve sparse feedback, and a prompt-adaptive termination rule that stops an episode as soon as a denoise-completion classifier and a cross-attention semantic-alignment signal both indicate the image is ready. In experiments, adding PAST to existing RL fine-tuning methods reduces time-to-target reward by up to 66.7% and improves preference scores by up to 29.5%.","feed_headline":"PAST cuts diffusion RL training cost by up to 66.7%","feed_subtitle":"Skipping saturating denoising steps cuts compute and, the paper reports, lifts preference scores by up to 29.5%.","key_machinery":"The Denoise-Aware Model (DAM), a small binary classifier trained on endpoint labels with a tunable neighborhood $\\nu$, produces the real-time cleanliness estimate $\\alpha_t^d$. The semantic-alignment evaluator computes pairwise cosine distances between cross-attention maps inside noun-attribute groups, producing $\\alpha_t^s$. The termination rule takes the earlier of the denoising-ready and alignment-ready steps: $t_{\\text{len}} = \\min\\{t^d_{\\text{len}}, t^s_{\\text{len}}\\}$. Intrinsic reward $R_{\\text{int}}(s_t,a_t)=\\|x_t-x_T\\|_2^2$ provides dense mid-trajectory feedback, while the adaptive factor $(1-\\alpha_t)$ orchestrates the extrinsic-intrinsic and exploration-convergence trade-offs. Theorem 2.1 gives the optimal policy in closed form and is used to justify stable convergence of the training objective.","core_discovery":"The paper's central claim is that episode length in RL fine-tuning does not need to be a fixed full denoising schedule. A Denoise-Aware Model learns to classify intermediate latents as noise-side or clean-side, giving a real-time cleanliness estimate; a second signal measures whether the attention maps of a prompt's noun and its attributes focus on the same regions, which the paper treats as semantic alignment. PAST terminates an episode at the earlier of the two 'ready' steps, and uses the adaptive factor $(1-\\alpha_t)$ to shift from intrinsic exploration reward to extrinsic task reward and to strengthen KL regularization as denoising proceeds. The paper reports comparable or better image quality on the same or shorter episodes, with computational savings up to 66.7% and preference optimization gains up to 29.5%.","pith_inferences":["A natural extension of the paper's logic is to reuse the same real-time denoising and attention-alignment signals as inference-time stopping rules, which would make every generation adaptive rather than only the RL training episodes.","Because the termination rule takes a minimum of two imperfect monitors, a conservative variant that requires both signals to persist for several consecutive steps could avoid premature termination at a small compute cost.","If the cross-attention alignment proxy truly measures semantic binding, the same signal could serve as an auxiliary loss or reward for non-RL fine-tuning methods, a direction the paper does not test.","The reported savings may grow when reward models become more expensive to query, since adaptive termination prevents paying those query costs on steps whose marginal value has saturated."],"forward_implications":["If the two readiness proxies are reliable, every RL fine-tuning run can skip steps beyond the earlier of denoising completion and semantic alignment, so compute savings multiply with prompt difficulty.","Adding PAST to reward-based methods preserves the base method's gains while substantially shortening time-to-target, as reported for Aesthetic Score and PickScore objectives.","The intrinsic reward shifts the policy toward reaching clean latents earlier, so per-episode step budgets can be set dynamically rather than fixed for all prompts.","The adaptive termination mechanism transfers across backbone architectures, including SD1.4, SD2.1, SDXL, and the flow-matching SD3.5, and across heterogeneous reward objectives.","By reducing over-optimization of late-step rewards, PAST also reduces reward-hacking artifacts such as style or background collapse, improving fidelity and diversity metrics."],"supporting_citations":[{"why":"The RL fine-tuning baseline that PAST wraps for reward optimization; provides the DDPO method as a main comparison.","marker":"[2]"},{"why":"A second RL fine-tuning baseline used to test plug-and-play acceleration.","marker":"[11]"},{"why":"A preference-based RL fine-tuning baseline used as a comparison.","marker":"[58]"},{"why":"A preference-based method that addresses reward over-optimization and is compared for reward curves.","marker":"[65]"},{"why":"Observation that image quality gains become marginal near the end of denoising, motivating early episode termination.","marker":"[32]"},{"why":"The paper's source of reward signals (PickScore) and the prompts from the Pick-a-pic dataset.","marker":"[24]"},{"why":"Provides the Aesthetic Score reward signal used in the fine-tuning experiments.","marker":"[41]"},{"why":"Supplies the soft Bellman optimality form used in the proof of Theorem 2.1.","marker":"[6]"}],"fun_headline_variants":["PAST adaptively ends diffusion RL episodes early, saving 66.7%","Diffusion RL: PAST stops early, cuts compute by 66.7%","PAST: adaptive episode termination for efficient diffusion RL","PAST saves 66.7% compute by knowing when diffusion is done"],"cache_read_input_tokens":22400,"weakest_assumption_plain":"The whole gain rests on the Denoise-Aware Model's binary cleanliness estimate and the noun-attribute attention-map similarity being accurate proxies for whether the image is actually ready; if either lags reality, the min-rule ends episodes too early and quality drops.","fun_headline_variants_meta":{"raw":{"variants":["PAST adaptively ends diffusion RL episodes early, saving 66.7%","Diffusion RL: PAST stops early, cuts compute by 66.7%","PAST: adaptive episode termination for efficient diffusion RL","PAST saves 66.7% compute by knowing when diffusion is done"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000837,"raw_usage":{"total_tokens":3645,"prompt_tokens":937,"completion_tokens":2708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2627}},"tokens_in":553,"tokens_out":2708,"duration_ms":16124,"temperature":1.0,"reasoning_tokens":2627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:29:12.397761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PAST on a prompt set where the decisive visual content, such as object identity, count, or spatial relation, first appears in the last few denoising steps. If the denoise or alignment proxies fire early on such prompts, adaptive episodes will terminate before that content is fixed, and quality metrics will measurably fall relative to full-length episodes.","supporting_citations":[{"cited_title":"Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song","cited_arxiv_id":null,"evidence_quote":"Supplies the soft Bellman optimality form used in the proof of Theorem 2.1."}],"review_version":2}