{"id":"f997befe-71fe-42f6-aaba-b3e5ae18bb0d","arxiv_id":"2601.22679","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new self-distillation objective with time-condition relaxation and training-time CFG stabilizes from-scratch consistency training and reduces variance across random seeds.","lead":"This paper analyzes why consistency-model training from scratch is unstable, and proposes a reformulated self-distillation objective (iSD) with training-time classifier-free guidance that trains stably from random initialization. It reaches FID 15.20 on ImageNet 256x256 (DiT-B/4, 2-step) with lower run-to-run variance than several baselines, and shows the same objective works for diffusion-based policy learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.2's optimality condition rests on an invalid Euler–Lagrange manipulation; the central diagnosis that direct training converges to degenerate flow maps is currently unproven.","rationale":"The reader's weakest_assumption already identifies the Appendix A.6 derivation of Proposition 4.2 as dubious; that is the single most load-bearing concern because the paper's central theoretical claim about direct training degeneracy depends on it. The derivation indeed contains a clear operator/adjoint error and an incorrect pointwise equivalence. However, the paper's empirical contribution — iSD trained from random initialization with low variance and competitive FID — is concrete, reproducible in description, and not undermined by the proof gap. Thus the appropriate verdict remains CONDITIONAL, not REJECT: the method may be sound, but the paper's theoretical justification needs correction or an explicit downgrade to conjecture. The proposed closed-form test would settle the validity of Prop 4.2 in a simple but nontrivial setting.","tokens_in":40565,"tokens_out":7089,"duration_ms":68986,"concrete_test":"Set up a tractable closed-form case: 1D linear interpolation with a two-mode Gaussian mixture, and a linear parameterization f_θ(x;t,s) = a(t,s)x + b(t,s) (or a small MLP). Compute L_DT(θ) by exact integration or high-precision Monte Carlo, minimize it, and check whether the minimizer satisfies the claimed condition L_ED + ∇·(Σ∇f) = 0 and whether it is actually a degenerate flow map (L_ED > 0). Compare with the prediction from a corrected Euler–Lagrange derivation. If the exact minimizer does not match the paper's condition, Prop 4.2 is invalid and the theoretical framing should be downgraded to a conjecture until a corrected proof is supplied.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is Proposition 4.2 : direct training (L_DT) has degenerate optima characterized by L_ED + C = 0 with C = ∇·(Σ_{Δv|x_t} ∇f). This is derived in Appendix A.6 via Euler–Lagrange. The derivation is not mathematically valid as written. For J(f)=E[∥∂_t f + v^T∇f∥^2], the stationarity condition is E[∂_t R + ∇·(vR)] = 0, where R = L^* f + Δv·∇f. This reduces to L^{*†}L^* f + ∇·(Σ∇f) = 0, with L^* = ∂_t + v^*·∇ and L^{*†} = −∂_t − ∇·(v^*·). The paper instead states L^*L^* f − ∇·(Σ∇f) = 0 ⇔ ∥L^*f∥_2^2 + ∇·(Σ∇f) = 0. This conflates the operator with its adjoint, drops the adjoint sign, and replaces a differential equation with a scalar equation. The stated equivalence is dimensionally inconsistent and no sufficiency argument is given. Since the paper's central diagnosis—that conditional-velocity training can converge to degenerate flow maps, motivating marginal-velocity guidance—rests on Prop 4.2, that part of the central claim is currently unsupported. The toy experiment in Figure 2 is suggestive but cannot validate the proposition without a correct variational statement. The empirical iSD results are credible and reproducible, so this is an addressable proof gap rather than a refutation of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a flow-map-based analysis of consistency models, arguing that (i) direct training with conditional velocity has degenerate optima (Proposition 4.2), (ii) consistency training admits fixed points satisfying the Eulerian equation but lacks second-order curvature needed for convergence (Proposition 4.3), and (iii) a reformulated self-distillation objective, iSD/iSD-T, provides stable training from random initialization by aligning with the marginal velocity and optionally with a classifier-free guidance trajectory. The empirical section reports ImageNet-1K 256×256 FID improvements from a consistency-training baseline of 120.5 to 15.20 with iSD-T, lower variance across seeds than baselines, and extensions to CelebA-HQ and diffusion-based policy learning. The paper includes detailed ablations, loss landscapes, and algorithmic descriptions.","tokens_in":40982,"tokens_out":5275,"duration_ms":53982,"significance":"If the theoretical analysis were correct, the paper would make a substantial contribution: it would unify several consistency/self-distillation methods under a flow-map representation and explain why direct conditional-velocity training can fail while marginal-velocity training succeeds. The practical recipe is also valuable: the reported variance reduction and training-from-scratch capability directly address a known reproducibility problem in consistency models. Strengths include controlled comparisons with fixed backbone/data/optimizer, multiple seeds with reported standard deviations, detailed hyperparameters and algorithms, and ablations isolating time-condition relaxation, joint flow matching, and Pre-CFG. The loss-landscape analysis is a useful empirical diagnostic. However, the central theoretical claims rest on a variational derivation in Appendix A.6 that is, as written, not valid, and the fixed-point argument for the final iSD objective is imported from a theorem about a different objective. The empirical results are credible and the gaps appear addressable, so the contribution is currently stronger as an empirical study than as a theory paper.","major_comments":[{"comment":"The Euler–Lagrange derivation of the optimality condition is not mathematically valid as written. The stationarity condition for L_DT = E[||∂_t f + v·∇f||^2] is ∂_t E[R] + ∇·E[R v] = 0 with R = ∂_t f + v·∇f. The manuscript replaces E[∂_t R] with ∂_t(L^*f) without accounting for the t-dependence of the conditional distribution p(x|xt), and then writes the adjoint step as L^*L^*f − ∇·(Σ∇f)=0 ⇔ ||L^*f||_2^2 + ∇·(Σ∇f)=0. This conflates a differential operator applied to f with an L^2 norm, drops the adjoint sign, and is dimensionally inconsistent. Consequently Proposition 4.2, which is the paper's central diagnosis of degenerate optima under direct training, is unsupported. The toy experiment in Figure 2 is suggestive but cannot substitute for a correct variational statement.","section":"A.6, Proposition 4.2, Eq. (10)"},{"comment":"The claim that the continuous-time consistency objective 'reduces in value to Eulerian distillation' is incorrect as stated. The derivation gives L_CT = E[ f·(L^*f^-) + f·(Δv·∇f^-) ], but Eulerian distillation is E[||L^*f||^2], not E[f·L^*f]. The two have different gradients even with a detached target, since ∇_θ E[f·L^*f^-] = E[∇f·L^*f^-], whereas the Eulerian gradient is E[2 L^*f·∇(L^*f)]. The fixed-point condition E[L^*f]=0 may still be derivable from the linear objective, but the 'reduces to ED' wording and the resulting discussion of second-order structure need to be corrected. This also affects the interpretation of small-batch behavior in Section 4.2 and the summary paragraph calling Theorem 4.2/4.3.","section":"A.5/A.7, Prop. 4.3"},{"comment":"The assertion that LSD-R 'admits the desired fixed points (Theorem 4.3)' is not supported by the cited theorem. Proposition 4.3 analyzes the continuous-time consistency objective L_CT with conditional velocity v_t(x_t|x); it does not analyze the self-distillation target F_tgt = F_θ + A''_{t,s} x_t + A'_{t,s}(v_θ−F_θ) − A_{t,s} dF_θ/dt, which uses the network's own instantaneous velocity v_θ = F_θ(x_t;t,t) under joint flow-matching training. A separate fixed-point or stationarity analysis for Eq. (13) is needed, or the theoretical claim should be weakened. The strong empirical results do not depend on this specific theorem, but the paper's theoretical framing of iSD does.","section":"Section 5.2, Eq. (13)–(15)"}],"minor_comments":[{"comment":"The text refers to 'Theorem 4.2' and 'Theorem 4.3' in the Section 4 summary and Section 5.2, but the paper only contains Proposition 4.2 and Proposition 4.3. Please correct the cross-references.","section":"General"},{"comment":"In Algorithm 1, the velocity assignment is written as 'vt ← α'_t x + σ'_t x'; the second term should presumably be σ'_t z. Please fix this typo.","section":"Algorithm 1"},{"comment":"The displayed equation for the reformulated self-distillation loss has malformed norm delimiters. Please clean up the LaTeX/rendering.","section":"Eq. (12)"},{"comment":"The dichotomy in the final paragraph of A.6 ('if this scenario is admitted... otherwise...') is not a proof; it is an informal case analysis. If retained, it should be clearly labeled as heuristic, not as part of the proposition proof.","section":"A.6"},{"comment":"The table and text use inconsistent notation for the iSD objective: Table 1 lists 'L_SD-R' while Section 5.2 defines 'LSD-R' and 'LiSD'. Please unify notation.","section":"Table 1 / Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is substantial and appears reproducible, but the theory section is not yet at the standard required for the claims made in the abstract. The Euler–Lagrange derivation in A.6 is the main blocker; it is a load-bearing error, but it is localized and potentially fixable. I would encourage the editor to request a corrected derivation or a clear reframing of Proposition 4.2 as a heuristic/empirical observation, and a dedicated fixed-point argument for the iSD objective before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical contribution is real: iSD is a reproducible scratch-training objective for consistency models, and the ablation shows a clear path from LCT at 120.5 FID to iSD-T at 15.20 with lower variance. The training recipe is detailed, the ablations are sensible, and the policy-learning extension is a nice bonus. The writing is honest and the authors flag their own limitations, including the failure of the network-free Pre-CFG variant. That part deserves credit.\n\nThe theoretical story, as you noted, is where the paper strains. Stress-testing the Euler-Lagrange derivation in Appendix A.6 confirms the problem: the operator composition L*L*f and the equivalence to ||L*f||^2 + div(Σ∇f) = 0 do not hold as written — the adjoint sign is dropped, the composition is dimensionally inconsistent, and no sufficiency argument is given. That means Proposition 4.2, the paper's central diagnosis for why direct training converges to degenerate flow maps, is currently unproven. The toy experiment is suggestive but cannot carry that weight alone. Also, the claim in Section 5.2 that LSD-R 'admits the desired fixed points (Theorem 4.3)' overshoots: Theorem 4.3 treats the original LCT objective, not the iSD objective, and the appendix does not fill that gap. So the theoretical framing should either be corrected or explicitly downgraded to conjecture.\n\nWhat's genuinely new: the flow-map representation as a unifying lens, the fixed-point perspective on consistency training, the stop-gradient reformulation LSD-R, and the Pre-CFG training-time guidance. The loss landscape analysis supports the time-relaxation hypothesis, and the empirical reproducibility is better than most in this subfield. The math is the soft spot, not the experiments.\n\nOverall: this paper deserves a serious referee, but with a clear expectation of major revision on the theory. I would send it to peer review rather than desk reject. Ask for a corrected variational statement for Prop 4.2 (or a plain conjecture label), a proper fixed-point analysis for LSD-R, and code release. Then it could be a solid contribution.","headline":"Useful training recipe with a shaky theoretical core; don't cite the theory until Prop 4.2 and the LSD-R fixed-point claims are fixed.","tokens_in":41471,"tokens_out":1835,"would_cite":false,"duration_ms":22394,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that consistency-model instability in scratch training stems from guiding the flow map by conditional rather than marginal velocity, and that a reformulated self-distillation fixes it, making pretrained diffusion precondit","keywords":["consistency models","flow maps","self-distillation","training instability","marginal velocity","few-step generation","classifier-free guidance","generative modeling"],"falsifier":"On a small Gaussian-mixture dataset where marginal and conditional velocities are known in closed form, numerically minimize the direct-training objective LDT with a rich enough network; if the global optimum satisfies the Eulerian equation (zero residual) rather than the paper's predicted condition with a nonzero divergence term, then Proposition 4.2 is refuted, and the instability must have a different cause.","tokens_in":40425,"feed_emoji":"","tokens_out":6928,"duration_ms":68981,"temperature":0.7,"pith_summary":"The paper tries to establish that consistency models—generators that map any point on a probability-flow trajectory straight to the data—become unstable when trained from scratch because the standard objective guides them with the conditional velocity of a single data sample instead of the marginal velocity of the true flow. That mismatch, the authors argue, creates degenerate optima and explains the large loss variance and seed-to-seed irreproducibility seen in practice. As a remedy, they reformulate self-distillation so the network learns from its own marginal velocity, relax the time condition, and add classifier-free guidance during training. If the claim is right, few-step generative models can be trained from random initialization without a pretrained diffusion backbone, and training is reproducible across seeds. The method is demonstrated on ImageNet-1K and on diffusion-based robot-policy learning.","feed_headline":"Consistency models can train from scratch without pretrained diffusion","feed_subtitle":"Flow-map analysis traces the instability to conditional velocity; self-distillation fixes it.","key_machinery":"The central object is the flow map f_{t,s}(x_t), the mapping that transports a point x_t along a probability-flow trajectory back to x_s, represented as f_θ(x_t;t,s)=ν^{-1}(A'_{t,s}x_t - A_{t,s}F_θ(...)). The argument hinges on the distinction between the conditional velocity v_t(x_t|x) of an individual sample and the marginal velocity v*_t(x_t) that defines the true flow: the paper shows the former biases the objective (via a conditional-covariance divergence term in the Euler–Lagrange condition), while the latter yields the desired optimum. Fixing the instability then turns on a reformulated self-distillation loss (LSD-R) that regresses the network onto a stop-gradient self-target driven b","core_discovery":"The paper's central claim is that the optimum of 'direct training'—Eulerian distillation under a conditional velocity field—is not the true flow map. An Euler–Lagrange analysis shows the optimum satisfies a first-order condition with a divergence term involving the conditional covariance of the velocity error, ∇·(Σ_{Δv|x} ∇f)=0, so a learned map can satisfy the objective by flattening its gradient rather than by reproducing the true flow. Consistency training, the paper argues, has the Eulerian equation as a fixed point but lacks the second-order structure needed to converge to it, which is why small batches, random weight init, and continued training drift toward degenerate maps. The remedy","pith_inferences":["If the marginal-velocity diagnosis is correct, then any consistency-style objective that conditions on per-sample paths—including variants with learned couplings—should exhibit a similar bias, and the paper's stop-gradient self-distillation fix could serve as a general stabilization module for those methods.","The theory suggests a sharp, checkable prediction: on a toy distribution with known conditional covariance, direct training should converge to a map with nonzero Eulerian residual; a careful numerical optimization should exhibit the predicted degeneracy rather than the true flow map.","The result hints that the instability of consistency models is not primarily an optimization-difficulty phenomenon but an optimum-shift phenomenon; if so, increasing model capacity or training time alone will not fix it, and guidance by marginal velocity is necessary.","Extending iSD-style marginal-guidance training to other flow-map learners (e.g., in video, audio, or offline RL) is a natural next step, since the same conditional-vs-marginal gap appears wherever flow maps are learned from paired samples."],"forward_implications":["Consistency models for few-step generation can be trained from random initialization, removing the need for pretrained diffusion preconditioners and making results reproducible across seeds.","Plain consistency training guided by conditional velocity is predicted to have a degenerate global optimum, meaning the bias is not merely a finite-batch or initialization effect.","Small batch sizes shift consistency training toward direct training, so the observed instability and degenerate solutions should worsen as batch size shrinks; large batch sizes mask the problem.","The time-condition relaxation s