{"id":"85cd6c2c-4c07-4b2a-a181-c0c4891453cd","arxiv_id":"2411.08954","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Directly supervising a consistency model against an ODE solver lowers ODE solving error yet degrades image quality, so better ODE solving does not imply better samples.","lead":"A new training variant for consistency models, Direct CMs, matches an ODE solver more closely than standard consistency models but produces worse images. The result challenges the common assumption that better ODE solving leads to better samples in few-step diffusion distillation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central measure E (Eq. 6) is the same distance Direct CMs train on, so 'better ODE solving' may only mean better matching of a coarse numerical solver; until compared against a high-accuracy reference, the headline dissociation is not established.","rationale":"To accept the headline, one must accept that E measures how well a model solves the empirical PF ODE. The reader's weakest_assumption identifies this; I agree. The paper's experiments are controlled and the code is released, which is real evidence of reproducibility, but the measure itself is not independently validated. Because Eq. 5 trains Direct CMs on exactly the distance reported in Eq. 6, the 'better ODE solving' result is partly built into the training procedure. The open question is whether fsolver is close enough to f^s that matching fsolver is equivalent to solving the ODE. The paper's ablations over N and ω show the E ordering persists, but they do not bound the solver's own error; increasing N from 50 to 200 changes E values only mildly in Figure 3, which could mean the solver is converged or that E is insensitive. The high-accuracy reference test distinguishes these. Secondary concerns, such as lack of error bars and the four-step metrics that favor Direct CM on FD-DINO and CLIP, further weaken the 'significantly worse' phrasing, but the primary issue is the E proxy. I therefore recommend no change to the reader's CONDITIONAL verdict; the paper is a useful diagnostic study, but the central counterexample is not established until the reference-solver check is run.","tokens_in":11162,"tokens_out":5162,"duration_ms":48983,"concrete_test":"Recompute Eq. 6 with a high-accuracy reference solution f_ref(xT, T, 0) obtained with the same teacher s but a much finer solver (e.g., 1000-step Heun or DPM-Solver-3 with tight tolerance), on the same fixed seeds; also report average d(fsolver, f_ref) for N ∈ {50, 100, 200}. If Direct CMs no longer have lower E_ref, or if the coarse-solver error is comparable to the CM/Direct gap, the claimed dissociation is an artifact of matching the specific coarse solver. If Direct CMs still have lower E_ref, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines E (Eq. 6) as the expected distance between fθ(xT, T) and fsolver(xT, T, 0). This is exactly the n = N term of the Direct CM training objective in Eq. 5. Thus Direct CMs are trained to minimize E, and observing E_DirectCM < E_CM is a direct consequence of the objective, not an independent measurement of ODE-solving quality. The paper never quantifies the discretization error between fsolver and f^s, the exact solution of the empirical PF ODE in Eq. 3. If fsolver is a poor approximation of f^s—plausible for N = 50–200 with guidance scales up to 11 on high-dimensional latent trajectories—then the result reduces to 'matching the training target at every timestep can hurt sample quality,' which does not challenge the premise that accurate PF ODE solving improves samples. The authors' explanation (ii) allows f^s ≠ f^*, but the distinct step fsolver ≈ f^s is not verified. The load-bearing assumption is that E faithfully proxies PF ODE error; without it, the counterexample does not land as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"Direct CMs,\" a variant of consistency-model distillation trained with Eq. (5) to directly minimize the distance between the student's one-step prediction and the numerical solver's output along the probability-flow ODE. Using SDXL as the teacher and LoRA fine-tuning, the authors compare Direct CMs against standard CMs across three ODE solvers (DDIM, Euler, Heun), for one, two, and four sampling steps, and across ablations over discretization intervals N and guidance scales ω. The reported results show that Direct CMs achieve lower ODE-solving error E (Eq. 6) but worse image-quality metrics (FID, FD-DINO, CLIP, aesthetic score). The paper concludes that better PF ODE solving does not imply better sample quality, and questions the theoretical motivation of consistency models.","tokens_in":11359,"tokens_out":4880,"duration_ms":49614,"significance":"If the dissociation between ODE-solving fidelity and sample quality were firmly established, this would be a valuable counterexample for the distillation community, with implications for consistency models and trajectory-based methods more broadly. The study has strengths: it is a controlled comparison where only the loss is changed, it uses a large-scale teacher, it releases code, and it includes sweeps over solver type, discretization, and guidance scale. However, the central ODE-error measure E is the final-timestep term of the Direct CM training objective, so the headline claim is not supported by an independent measurement of ODE-solving quality. As it stands, the experiments convincingly show that better matching the training target does not imply better samples, which is weaker than the paper's stated conclusion.","major_comments":[{"comment":"The definition of E in Eq. (6) is the n = N term of the Direct CM training objective in Eq. (5). Since Direct CMs are trained to minimize Eq. (5) for all n, including n = N, observing E_DirectCM < E_CM is substantially built into the training procedure rather than being an independent measurement of ODE-solving error. The paper never verifies that the numerical solver fsolver with N = 50–200 and guidance scales up to 11 is close to the exact solution f^s of the empirical PF ODE in Eq. (3), for example by comparing against a high-accuracy reference solution with much larger N or adaptive step size. Without such verification, the results support the weaker statement that directly minimizing the solver-distance objective does not imply better samples, not the stronger claim in the abstract and Section 4 that better ODE solving does not imply better samples.","section":"Section 3, Eqs. (5) and (6)"},{"comment":"All reported numbers are point estimates from a single training run of 250 LoRA steps, with no error bars, multiple seeds, or significance tests. The image-quality gap is large, but the ODE-error differences are modest (E of 0.23–0.25 for Direct CMs versus 0.29–0.30 for CMs), and without run-to-run variability estimates the claim that Direct CMs are \"consistently\" better at ODE solving is not statistically supported. At minimum, the paper should report confidence intervals or repeated training runs for the main comparisons.","section":"Section 4, Table 1 and Figures 3–4"},{"comment":"The claim that CMs produce higher-quality images than Direct CMs is not uniformly true in the reported multi-step results. At four steps, Direct CMs outperform CMs on FD-DINO for both DDIM (155.1 versus 159.8) and Heun (150.6 versus 159.4), and on CLIP score for both (0.28 versus 0.27). The paper attributes these exceptions to imperfections in automated metrics, but this is a post hoc explanation. The conclusions should either be scoped to single-step generation or supported by additional analysis showing that the metric exceptions do not reflect genuine sample-quality differences.","section":"Appendix A.2, Table 3"}],"minor_comments":[{"comment":"The first author's name contains an erroneous space: \"Noël V ouitsis\" should read \"Noël Vouitsis.\"","section":"Author list"},{"comment":"The notation \"n∼U J1,N K\" is malformed; it should be written as n ∼ Uniform({1, ..., N}) or equivalent.","section":"Section 2, Eq. (4)"},{"comment":"The paper states that metric and loss curves stabilized around 250 training steps, but no learning curves are shown. Since Direct CMs optimize a different objective, a brief sensitivity check over training length, LoRA rank, or loss weighting λ(t) would make the controlled comparison more persuasive.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical measure is too closely tied to the Direct CM objective for the headline claim to stand as written. The paper is a useful case study, but the authors should either add an independent high-accuracy ODE-error evaluation or reframe the conclusion as showing that minimizing the training objective does not imply better sample quality. The multi-step metric exceptions should also be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean empirical probe that deserves a serious referee, but the headline needs an independent ODE-error check before it fully lands.\n\nThe new thing is the dissociation itself. They make the smallest possible modification to the consistency distillation loss—regress fθ(xtn, tn) directly against the solver output fsolver(xtn, tn, 0) at every timestep—and find that this \"Direct CM\" achieves lower ODE solving error E but worse FID, FD-DINO, CLIP, and aesthetic scores across DDIM, Euler, and Heun, across N ∈ {25, 50, 100, 200} and guidance scales 1–11. That is genuinely surprising and it undercuts the standard story that consistency models work because they learn to solve the PF ODE. The experiments are well-controlled: identical parameterization, identical LoRA setup, only the loss differs, and the qualitative gap is large and consistent.\n\nThe soft spot is exactly the one in the stress-test note. E in Eq. 6 is the same distance that Direct CMs are trained to minimize in Eq. 5 (the n = N term). So the lower E is substantially guaranteed by the objective; it is not an independent measurement of ODE-solving quality. The paper never quantifies how close fsolver is to the exact PF ODE solution f^s. With N = 50–200 and guidance up to 11, the discrete solver error could be nontrivial, and if fsolver is a poor proxy for f^s, the finding reduces to \"matching a coarse training target can hurt samples,\" which does not challenge the premise that accurate ODE solving helps. The authors list (ii) f^s ≠ f^*, but they do not address the separately load-bearing assumption fsolver ≈ f^s. That is a real gap.\n\nSecond-order issues: single run, 250 LoRA steps, no error bars or multiple seeds; at four steps two of four metrics favor Direct CM, dismissed as metric artifacts. Those are minor in context but contribute to the conditional verdict.\n\nI think the right reading is: the paper is a good diagnostic, not a settled counterexample. A referee should ask for an independent ODE-error evaluation (e.g., compare both models against a high-accuracy solver such as DPM-Solver++ or a fine RK45 with many steps) and a couple of seeds. The authors have been honest about limitations and ship code. This belongs in the review process; it's a thought-provoking probe for the distillation community, and I'd bring it to a reading group.","headline":"A well-controlled empirical probe showing that better ODE-solver matching can hurt sample quality, but the headline needs an independent ODE-error check before it fully lands.","tokens_in":11982,"tokens_out":2038,"would_cite":true,"duration_ms":17857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Directly training consistency models to minimize ODE solving error produces worse sample quality than the standard weakly-supervised objective.","keywords":["consistency models","diffusion distillation","probability flow ODE","ODE solving error","few-step generation","sample quality","Direct Consistency Models","latent diffusion"],"falsifier":"Compute $E$ for both models against a reference solution obtained from a much finer discretization or a high-order solver with thousands of steps, and check whether Direct CMs still have lower error to that reference while producing worse images; if the error ordering reverses, the central claim reduces to a statement about matching a fixed training target.","tokens_in":10913,"feed_emoji":"🎨","tokens_out":4214,"duration_ms":34628,"temperature":0.7,"pith_summary":"Consistency models distill a diffusion model by training a network to mimic a numerical solver of the probability-flow ODE, with the implicit assumption that more faithful ODE solving means better samples. The paper tests this assumption directly by introducing Direct Consistency Models, which replace the standard self-consistency loss with a loss that explicitly minimizes the distance to the solver's output at every point along the trajectory. In controlled experiments distilling SDXL with three different solvers and across ablations of discretization and guidance scale, Direct CMs achieve lower ODE solving error $E$ but noticeably worse image quality on every image metric. The authors conclude that the success of consistency models cannot be attributed to their ODE-solving fidelity, and that other factors must drive sample quality.","feed_headline":"Better ODE solving yields worse images, distillation study finds","feed_subtitle":"Consistency models trained to match the solver exactly produce lower error but visibly poorer samples, challenging a core assumption.","key_machinery":"The load-bearing object is the Direct CM objective (Equation 5), $L_{\\mathrm{CD}}^{\\mathrm{Direct}} = \\mathbb{E}[\\lambda(t_n)\\, d(f_\\theta(x_{t_n}, t_n), f_{\\mathrm{solver}}(x_{t_n}, t_n, 0))]$, which replaces the weak self-consistency loss (Equation 4) with direct, per-point supervision from the discrete teacher solver. The companion metric $E$ (Equation 6) measures exactly the expected distance this loss minimizes, so the paper's ODE-solving comparison is, by construction, a comparison of how well each model matches $f_{\\mathrm{solver}}$ from the same initial noise. The controlled setup isolates the loss as the only changed component, with identical teacher (SDXL), LoRA parameterization, seeds, and evaluation protocol.","core_discovery":"The central discovery is a counterexample to the assumption that better PF ODE solving entails better sample quality. The paper constructs Direct CMs, trained with Equation 5, which supervises $f_\\theta(x_{t_n}, t_n)$ to match $f_{\\mathrm{solver}}(x_{t_n}, t_n, 0)$ directly, so that minimizing the objective is essentially minimizing the measured ODE error $E$ of Equation 6. Across DDIM, Euler, and Heun solvers, and across $N \\in \\{25, 50, 100, 200\\}$ and $\\omega \\in \\{1, 4, 8, 11\\}$, Direct CMs consistently reduce $E$ relative to standard CMs yet produce worse FID, FD-DINO, CLIP, and aesthetic scores, with visibly noisier images. The paper argues this falsifies the implicit assumption behind CM-style distillation and calls for investigation of confounding factors that make CMs work.","pith_inferences":["The paper's own caveats suggest a stronger reading: because $E$ is measured against the discrete solver, the result may show that matching the training target better does not improve samples, rather than that matching the true solution does not; testing against a much finer reference solver would settle which statement holds.","The observed gap hints that the self-consistency loss acts as an implicit regularizer or denoiser, and removing it removes a beneficial inductive bias; one testable extension is to add the self-consistency term back to the Direct CM loss and see whether sample quality recovers while $E$ stays low.","A practical consequence the authors do not draw is that ODE-faithful distillation may be the wrong target for few-step generation, and methods that add distribution-matching or adversarial losses may be aligning with the right objective for reasons unrelated to the ODE."],"forward_implications":["If valid, the counterexample refutes the notion that better PF ODE solving is the mechanism by which consistency models obtain high sample quality.","Improvements attributed to follow-up CM variants that add trajectory splitting or bidirectional solving should not be credited to better ODE fidelity without independent evidence.","ODE-solving fidelity alone is not a reliable proxy for evaluating distillation quality, so sample-based metrics remain necessary.","The Direct CM design provides a reusable probe: any distillation method can be tested against $f_{\\mathrm{solver}}$ with Equation 6 to separate ODE accuracy from generation quality."],"supporting_citations":[{"why":"Defines consistency model distillation, the weak self-consistency loss that Direct CMs modify, and the theory that perfect minimization yields the solver.","marker":"[49]"},{"why":"Establishes the probability-flow ODE formulation that both the CM and Direct CM objectives are built on.","marker":"[48]"},{"why":"Provides the latent-consistency-model training recipe, hyperparameters, and cskip/cout parameterization used in the controlled experiments.","marker":"[25]"},{"why":"Supplies one of the numerical solvers used to define fsolver and measure E.","marker":"[45]"},{"why":"Supplies the SDXL teacher model whose empirical score defines the PF ODE being distilled.","marker":"[30]"},{"why":"Supplies the Euler solver used as another definition of fsolver.","marker":"[8]"},{"why":"Supplies the Heun solver used as another definition of fsolver.","marker":"[38]"},{"why":"Introduces latent diffusion, the space in which the ODEs here are defined, relevant to the paper's explanation of why latent-space closeness may not survive decoding.","marker":"[36]"}],"fun_headline_variants":["Better ODE solving, worse samples in consistency models","Direct consistency models: less ODE error, poorer images","Consistency model paradox: lower error, worse sample quality","Solving ODE better hurts sample quality in consistency models","Distillation flaw: accurate ODE solving degrades images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on treating the numerical solver used for supervision as a faithful stand-in for the true probability-flow solution; if that solver is far from the true solution, the result only shows that matching the training target better does not improve samples.","fun_headline_variants_meta":{"raw":{"variants":["Better ODE solving, worse samples in consistency models","Direct consistency models: less ODE error, poorer images","Consistency model paradox: lower error, worse sample quality","Solving ODE better hurts sample quality in consistency models","Distillation flaw: accurate ODE solving degrades images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2964,"prompt_tokens":935,"completion_tokens":2029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1950}},"tokens_in":551,"tokens_out":2029,"duration_ms":12622,"temperature":1.0,"reasoning_tokens":1950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:13:47.218891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $E$ for both models against a reference solution obtained from a much finer discretization or a high-order solver with thousands of steps, and check whether Direct CMs still have lower error to that reference while producing worse images; if the error ordering reverses, the central claim reduces to a statement about matching a fixed training target.","supporting_citations":[{"cited_title":"Consistency models","cited_arxiv_id":null,"evidence_quote":"Defines consistency model distillation, the weak self-consistency loss that Direct CMs modify, and the theory that perfect minimization yields the solver."},{"cited_title":"SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the SDXL teacher model whose empirical score defines the PF ODE being distilled."},{"cited_title":"Griffiths and Desmond J","cited_arxiv_id":null,"evidence_quote":"Supplies the Euler solver used as another definition of fsolver."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Heun solver used as another definition of fsolver."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Introduces latent diffusion, the space in which the ODEs here are defined, relevant to the paper's explanation of why latent-space closeness may not survive decoding."}],"review_version":1}