Pith. sign in

REVIEW 5 minor 42 references

A closed-form mixture of true and teacher velocities provably improves a suboptimal rectified-flow teacher on fixed interpolation pairs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:36 UTC pith:EXAEFRVD

load-bearing objection A clean, honest transfer of ridge self-distillation to linear rectified flow; the math is solid and the sign rule is useful, but the scope is narrower than the abstract suggests.

arxiv 2607.14947 v1 pith:EXAEFRVD submitted 2026-07-16 stat.ML cs.LG

Optimal Self-Distillation for Rectified Flow via Linear Probing

Saptarshi Roy , Debepsita Mukherjee , Pratik Patil This is my paper
classification stat.ML cs.LG MSC 62J0768T07
keywords rectified flowself-distillationridge regressionlinear probingflow matchingvelocity field riskWasserstein generation boundssynthetic data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a generative rectified-flow model can improve itself by training a student on a mixture of true velocity targets and its own teacher predictions. For linear rectified flow with ridge regularization on fixed interpolation pairs, the paper proves that the self-distilled student is exactly an affine interpolation between the teacher and a pure-distilled refit. That makes the integrated velocity risk a quadratic function of the mixing coefficient, so the best mixture is given in closed form and strictly improves the teacher whenever the teacher's risk is nonstationary along the regularization path. The optimal coefficient follows a sign rule—positive mixing for under-regularized teachers, negative for over-regularized ones—and the same gain flows into the velocity-error terms of standard Wasserstein generation bounds. The result matters because it isolates, in one tractable setting, when training on model-generated signals helps rather than collapses, and gives a one-shot tuning procedure that requires no grid search.

Core claim

The paper's central discovery is the affine path identity: for ridge-regularized linear rectified flow, the student fitted on mixed velocity targets u(ξ)=(1−ξ)u+ξû_λ equals (1−ξ) times the teacher field plus ξ times the pure-distilled refit field. Because the ridge estimator is linear in the targets, the integrated RF risk becomes exactly R_sd(λ,ξ)=R(λ)−2ξ{R(λ)−C(λ)}+ξ²D(λ), so the optimal mixing coefficient is ξ⋆(λ)=(R(λ)−C(λ))/D(λ)=−(λ/2)R'(λ)/D(λ). The risk reduction equals (R−C)²/D=(λ²/4)(R')²/D; therefore any teacher whose risk is not at a stationary point of the ridge path is strictly improved, and the sign of ξ⋆ is opposite to the slope of the risk curve. The same gain enters the velo

What carries the argument

The central object is the affine path identity for linear ridge RF on fixed interpolation pairs. Since the ridge fit is linear in the response matrix, the student trained on mixed targets is an affine combination of the teacher coefficient and the pure-distilled refit coefficient: W_sd,λ,ξ = (1−ξ)W_λ + ξ W_pd,λ. The squared-loss geometry then turns the integrated RF risk into a quadratic in ξ, with the cross term C(λ) and the distance D(λ) between teacher and pure-distilled predictions. A derivative identity, W_λ − W_pd,λ = λ ∂_λ W_λ, connects the optimal ξ to the slope of the teacher risk curve, giving the closed form and the sign rule. The paper's one-shot tuning estimates R, R_pd, and C u

Load-bearing premise

The load-bearing premise is that the interpolation covariates (T_i, X_{T_i,i}) and the feature map φ stay fixed across the teacher, the pure-distilled refit, and the student; if the interpolants are regenerated or the feature map changes, the affine path identity and the closed-form strict improvement no longer follow.

What would settle it

At a ridge level λ where the empirical teacher risk curve is visibly non-flat, keep the interpolation pairs fixed, train the ridge teacher and the pure-distilled refit, then numerically minimize the integrated RF risk over ξ. If the minimum is not below the teacher's risk, or if the sign of the minimizing ξ is not opposite to the sign of the local slope of R(λ), the central claim is contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At every ridge level where the teacher's risk is nonstationary, the optimally mixed student strictly lowers integrated velocity risk relative to both the teacher and pure distillation.
  • The optimal mixing coefficient is closed-form and sign-determined by the slope of the teacher risk: positive for under-regularized teachers, negative for over-regularized ones.
  • The velocity-risk gain transfers directly to generation: the same reduction appears in the velocity-error term of continuous-time and Euler Wasserstein bounds, tightening those bounds.
  • One-shot GCV or validation tuning selects the mixing coefficient without grid search or repeated student refits, requiring only a teacher fit, a pure-distilled refit, and a scalar search.
  • Pure distillation alone can worsen over-regularized teachers because it repeats the same shrinkage; the strict improvement requires extrapolation (ξ<0), which conventional mixing constrained to [0,1] cannot deliver.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The affine-path structure is not limited to ridge regression: any estimator that is linear in its targets and whose pure-distilled version is a repeated application of the same operator should admit the same quadratic risk decomposition, so the closed-form optimal mixing may extend to kernel interpolants or shrinkage estimators with early stopping.
  • Negative mixing acts as a de-shrinking operation on the teacher field, suggesting a direct connection between self-distillation for flow models and ridge-regression debiasing; this could help predict when synthetic supervision amplifies rather than collapses diversity.
  • A testable extension is to apply the fixed-interpolant mixing to non-ridge teachers, such as random-feature or neural models trained with gradient descent, and check whether the sign of the optimal correction still tracks the slope of the validation risk as a function of the effective regularization.
  • Because the one-shot GCV estimates depend only on the smoother matrix, the method may scale to very high-dimensional feature maps without additional refits; the main risk is that approximating S_λ in nonlinear models degrades the estimates, so validation on the affine family may be the safer route in neural settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper studies self-distillation for rectified flow: a student is trained on a mixture of true RF velocities and teacher velocities. For linear RF with ridge regularization and fixed interpolation pairs, the paper proves an affine path identity (Prop. 1) under which the self-distilled student field is an affine combination of the teacher and the pure-distilled refit. Consequently, the integrated RF risk is an exact quadratic in the mixing coefficient ξ (Theorem 2), yielding the closed-form optimum ξ*(λ) = −(λ/2)R′(λ)/D(λ), strict velocity-risk improvement for every nonstationary teacher along the ridge path, and the sign rule. The paper then connects this risk reduction to RF Wasserstein generation bounds (Prop. 3, Theorem 4), limiting claims to the velocity-error component. It proposes one-shot GCV/validation tuning and validates the theory on Gaussian, Gaussian-mixture, and image experiments, including a neural CIFAR-10 stress test. The exact theory is explicitly scoped to fixed interpolants and linear probing; the neural extension is labeled empirical.

Significance. If the result holds, this is a valuable provable positive result on self-distillation in generative modeling: a suboptimal RF teacher can be improved by a single extrapolation along the teacher-to-pure-distillation direction, without recursive endpoint replacement. The algebraic core—Prop. 1 and Theorem 2—is self-contained, correct, and cleanly proved in Appendix A. The paper gives a falsifiable sign rule and a one-shot tuning procedure that is honest model selection rather than curve fitting. The generation-bound consequences are carefully scoped to the velocity-error terms, and the limitations are stated explicitly in Section 6 and Supplement D. The experiments include oracle, GCV, and validation comparisons and a controlled neural test, which strengthens the empirical claims. Overall, this is a solid and focused contribution.

minor comments (5)
  1. [§4.1, Eq. (24); §4.2, after Eq. (31)] The phrase 'using a common Lipschitz bound L for these fields' is an extra assumption. For ξ<0, v_sd is an extrapolation and its Lipschitz constant can be larger than the teacher's. The paper is careful to claim improvement only in the velocity-error term, so this is not a defect, but a sentence noting that the student's own W2 bound may require a different L would prevent over-reading.
  2. [§3.2, Eq. (18)] The GCV plug-in estimates bR, bRpd, and bC are presented as direct analogues of standard ridge GCV. The cross-denominator form of bC is nonstandard; a brief derivation or a more explicit citation for this particular estimator would improve rigor.
  3. [Algorithm 1, Step 4] The method is called 'one-shot' even though Step 4 involves a one-dimensional validation search over ξ. The key point—that no refitting is needed for each candidate—is clear, but the wording could be sharpened to avoid the impression that ξ is obtained in closed form in the neural extension.
  4. [Theorem 2, Eq. (16)] The assumption D(λ)>0 is stated, but note that R′(λ)≠0 already implies D(λ)>0 via the identity R(λ)−C(λ)=−(λ/2)R′(λ). Stating this equivalence would make the nonstationarity condition tidier.
  5. [Figures 1 and 3] Check typesetting of axis labels and exponents in the final PDF: several labels render as '10 4' and '10 3' where negative powers (e.g., 10^{-4}) are clearly intended, and the word 'Teacher' is split in Figure 1.

Circularity Check

0 steps flagged

No significant circularity: the core affine-path identity and risk quadratic are derived from ridge linearity and are not equivalent to their inputs.

full rationale

The paper's central claim—Proposition 1 and Theorem 2—is self-contained. The affine path identity cW_sd,λ,ξ = (1−ξ)cW_λ + ξ cW_pd,λ follows immediately from linearity of the ridge estimator in the response matrix (Appendix A.1), and the risk quadratic (14) is an algebraic expansion of squared residuals. The optimal mixing coefficient ξ⋆ = (R−C)/D is derived, not imposed, and the derivative identity R−C = −(λ/2)R′ is proved from ∂λ cW_λ = −(Σ̂+λI)^{-1} cW_λ and the relation cW_λ−cW_pd,λ = −λ∂λ cW_λ. No fitted parameter is renamed as a prediction; the one-shot GCV/validation tuning is presented as an estimator of the derived oracle coefficient, which is honest model selection. The neural extension is explicitly labeled as an empirical extension (Section 6), not as a theorem. The Wasserstein bounds borrowed from Bansal et al. [3] are used only to translate velocity-risk improvement into generation-error bounds; the cited result is an external, parameter-free RF convergence theorem that does not assume self-distillation or the present result. The Dang et al. [6] self-citation supplies context for ridge self-distillation, but the present proof does not reduce to it. The fixed-interpolant assumption is stated explicitly and scoped, and the paper even notes where the theory would fail under recursive endpoint replacement. Therefore no equation reduces to its own input by construction, and no load-bearing step is equivalent to a fit or to a self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The theoretical contribution requires no invented entities. The fixed-interpolant assumption and the ridge linearity are the main load-bearing premises along with the external Wasserstein bounds. The free parameters ξ, λ, and teacher scale a are standard hyperparameters or experiment-design choices; the closed-form ξ⋆ is not fitted in theory but is estimated in practice.

free parameters (3)
  • Mixing coefficient ξ = Selected by GCV/validation in experiments; oracle optimum (R(λ)−C(λ))/D(λ) in theory
    The practical procedure tunes ξ from data (Algorithm 1). The theory derives the optimum, but all empirical gains depend on the selection method's ability to estimate the true optimum; the negative-mixing result is the paper's key practical claim.
  • Ridge penalty λ = Swept across a grid in experiments (e.g., λ≈7.5 for MoG, λ≈31.6 for Gaussian)
    The strict improvement requires a fixed λ with R'(λ)≠0; the magnitude of the gain depends on λ. In experiments λ is a user-chosen hyperparameter, not learned in the theory.
  • Teacher scale a (neural stress tests) = a=0.75 (CIFAR-10 main), a∈{0.55,0.65,0.75,0.85} in Table S4, 0.35 for Fashion-MNIST
    The neural image experiments deliberately scale the teacher output to emulate over-regularization. The observed gains depend on this stress-test design; the supplement reports that gains diminish as a→1.
axioms (7)
  • standard math Ridge estimator closed form: Ŵ_λ = (Φ^TΦ/N + λI)^{-1}Φ^TU/N
    This linearity in the response matrix U is the basis of the affine path identity (Prop 1).
  • domain assumption Fixed interpolation triples (T_i, X_{T_i,i}) and fixed feature map φ across teacher, pure-distilled, and self-distilled fits
    The entire theory (Section 3, Eq. (9)) relies on bU_λ = S_λU and W_pd = S_λ Ŵ_λ evaluated at the same interpolation points. If the interpolants changed, the affine identity and quadratic risk would break.
  • standard math Differentiability of R(λ) along the ridge path and the derivative ∂_λ Ŵ_λ = −(Σ̂+λI)^{-1}Ŵ_λ
    Used to obtain the sign rule ξ⋆ = −(λ/2)R'(λ)/D(λ) in Theorem 2.
  • domain assumption Wasserstein bound W_2²(ρ_v, ρ_1) ≤ exp(1+2L)E(v) (Bansal et al. [3], Prop 3)
    Cites an external theorem to convert velocity-risk improvement into generation-error upper-bound improvement; requires L-Lipschitz velocity fields and well-posed ODEs.
  • domain assumption Euler discretization bound (29) with constant C_L (Bansal et al. [3], Theorem 4)
    Used for the finite-step generation guarantee; depends on the same Lipschitz and straightness assumptions.
  • domain assumption Independent coupling π = ρ_0 ⊗ ρ_1 and linear interpolant X_t = (1−t)X_0 + tX_1
    The standard rectified-flow setup (Section 2); the theory is stated for this coupling.
  • domain assumption In the Gaussian and MoG experiments, the population RF velocity v⋆ is exactly representable in the chosen feature map
    These experiments validate the exact theory; the representation is designed so the linear model is correctly specified.

pith-pipeline@v1.3.0-alltime-deepseek · 20668 in / 12528 out tokens · 128791 ms · 2026-08-02T00:36:18.585728+00:00 · methodology

0 comments
read the original abstract

Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.

Figures

Figures reproduced from arXiv: 2607.14947 by Debepsita Mukherjee, Pratik Patil, Saptarshi Roy.

Figure 1
Figure 1. Figure 1: Optimal self-distillation corrects degraded RF teachers. In controlled stress tests on real images from handwritten digits and Fashion-MNIST, we scale down the teacher output to emulate an over-regularized or otherwise miscalibrated teacher field. The resulting teacher and pure-distilled student produce degraded samples, whereas validation selects a negative mixing coefficient for unconstrained SD, which r… view at source ↗
Figure 2
Figure 2. Figure 2: Schematic of self-distillation for RF velocity targets on fixed interpolants. Unlike recursive Reflow, the endpoint coupling and interpolation covariates remain fixed. The teacher predicts velocities at the same RF training locations, and the student is trained on their mixture with the true targets. representations [8]; the affine path identity below applies to any fixed choice of ϕ. Once ϕ is fixed, trai… view at source ↗
Figure 3
Figure 3. Figure 3: The sign rule in Gaussian RF. (a) In a correctly specified Gaussian RF model, one-shot GCV closely tracks oracle mixing and selects negative mixing in over-regularized regimes. (b) Constraining ξ ∈ [0, 1] often returns the teacher when the optimal correction is negative. unconditional field toward a conditional field to strengthen conditioning. Here the path joins a teacher trained on true targets and its … view at source ↗
Figure 4
Figure 4. Figure 4: Generated samples for nonlinear Gaussian mixture RF. At λ ≈ 7.5, the over-regularized teacher blurs the mixture. Oracle and one-shot self-distillation use negative mixing and recover the target modes much more accurately. At a fixed grid time ti , the self-distilled conditional risk has the decomposition Rsd,ti (λ, ξ) = Rti (λ) − 2ξAti (λ) + ξ 2Dti (λ), Ati (λ) := Rti (λ) − Cti (λ), (26) where Rsd,ti (λ, ξ… view at source ↗
Figure 5
Figure 5. Figure 5: Neural CIFAR-10 RF experiment. Left: the teacher and pure-distilled samples are degraded. Clipping ξbto [0, 1] returns the teacher because validation selects ξ <b 0, whereas uncon￾strained optimal SD produces clearer samples. Right: optimal SD reduces RF risk across the sampled time grid. Appendix B.3 gives exact Wasserstein and KL errors in a Gaussian linear model and shows explicitly how they depend on t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 28 linked inside Pith

  1. [1]

    Albergo and Eric Vanden-Eijnden

    Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. InProceedings of the International Conference on Learning Representations,

  2. [2]

    Baraniuk

    Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go MAD. InProceedings of the International Conference on Learning Representations,

  3. [3]

    On the convergence and straightness of rectified flow

    Vansh Bansal, Saptarshi Roy, Alessandro Rinaldo, and Purnamrita Sarkar. On the convergence and straightness of rectified flow. InProceedings of the International Conference on Artificial Intelligence and Statistics, 2026. URLhttps://arxiv.org/abs/2410.14949

    Pith/arXiv arXiv 2026

  4. [4]

    Error bounds for flow matching methods

    Joe Benton, George Deligiannidis, and Arnaud Doucet. Error bounds for flow matching methods. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2305. 16860

    2024

  5. [5]

    Nepotistically trained generative-AI models collapse

    Matyas Bohacek and Hany Farid. Nepotistically trained generative-AI models collapse. arXiv: 2311.12202, 2023. URLhttps://arxiv.org/abs/2311.12202

    Pith/arXiv arXiv 2023

  6. [6]

    Optimal unconstrained self-distillation in ridge regression: Strict improvements, precise asymptotics, and one-shot tuning

    Hien Dang, Pratik Patil, and Alessandro Rinaldo. Optimal unconstrained self-distillation in ridge regression: Strict improvements, precise asymptotics, and one-shot tuning. arXiv: 2602.17565,

    arXiv

  7. [7]

    Understanding self-distillation in the presence of label noise

    Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. InProceedings of the International Conference on Machine Learning, 2023. URL https://arxiv.org/abs/2301.13304

    Pith/arXiv arXiv 2023

  8. [8]

    Implicit regularization paths of weighted neural representations

    Jin-Hong Du and Pratik Patil. Implicit regularization paths of weighted neural representations. InAdvances in Neural Information Processing Systems, 2024. URL https://pratikpatil. io/papers/weighted-neural-representations.pdf

    2024

  9. [9]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. InProceedings of the International Conference on Machine Learning, 2024. URL...

    Pith/arXiv arXiv 2024

  10. [10]

    Bayesian knowledge distillation: A bayesian perspective of distillation with uncertainty quantification

    Luyang Fang, Yongkai Chen, Wenxuan Zhong, and Ping Ma. Bayesian knowledge distillation: A bayesian perspective of distillation with uncertainty quantification. InProceedings of the International Conference on Machine Learning, 2024. URL https://proceedings.mlr. press/v235/fang24a.html

    2024

  11. [11]

    Towards theoretical understandings of self-consuming generative models

    Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models. InProceedings of the International Conference on Machine Learning, 2024. URLhttps://arxiv.org/abs/2402.11778

    Pith/arXiv arXiv 2024

  12. [12]

    Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar

    Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Born again neural networks. InProceedings of the International Conference on Machine Learning, 2018. URLhttps://arxiv.org/abs/1805.04770

    Pith/arXiv arXiv 2018

  13. [13]

    Will large-scale generative models corrupt future datasets? InProceedings of the International Conference on Computer Vision, 2023

    Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? InProceedings of the International Conference on Computer Vision, 2023. URL https://arxiv.org/abs/2211.08095

    Pith/arXiv arXiv 2023

  14. [14]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv: 1503.02531, 2015. URLhttps://arxiv.org/abs/1503.02531

    Pith/arXiv arXiv 2015

  15. [15]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. InNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://arxiv.org/ abs/2207.12598. 10

    Pith/arXiv arXiv 2021

  16. [16]

    Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher

    Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. InAdvances in Neural Information Processing Systems,

  17. [17]

    Pyramidal flow matching for efficient video generative modeling

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. InProceedings of the International Conference on Learning Representations, 2025. URLhttps://arxiv.org/abs/2410.05954

    arXiv 2025

  18. [18]

    Balanced conic rectified flow

    Shin Seong Kim, Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Balanced conic rectified flow. InAdvances in Neural Information Processing Systems, 2025. URL https://arxiv. org/abs/2510.25229

    arXiv 2025

  19. [19]

    Dynamic negative guidance of diffusion models

    Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni. Dynamic negative guidance of diffusion models. InProceedings of the International Conference on Learning Representations, 2025. URLhttps://arxiv.org/abs/2410.14398

    Pith/arXiv arXiv 2025

  20. [20]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. InProceedings of the International Conference on Learning Representations, 2022. URL https://arxiv. org/abs/2202.10054

    Pith/arXiv arXiv 2022

  21. [21]

    V oicebox: Text-guided multilingual universal speech generation at scale

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. V oicebox: Text-guided multilingual universal speech generation at scale. InAdvances in Neural Informa- tion Processing Systems, 2023. URLhttps://arxiv.org/abs/2306.15687

    Pith/arXiv arXiv 2023

  22. [22]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InProceedings of the International Conference on Learning Representations, 2023. URLhttps://arxiv.org/abs/2210.02747

    Pith/arXiv arXiv 2023

  23. [23]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. InProceedings of the International Conference on Learning Representations, 2023. URLhttps://arxiv.org/abs/2209.03003

    Pith/arXiv arXiv 2023

  24. [24]

    A comprehensive survey on knowledge distillation of diffusion models

    Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. arXiv: 2304.04262, 2023. URLhttps://arxiv.org/abs/2304.04262

    Pith/arXiv arXiv 2023

  25. [25]

    Combining generative artificial intelligence (AI) and the Internet: Heading towards evolution or degradation? arXiv: 2303.01255, 2023

    Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Combining generative artificial intelligence (AI) and the Internet: Heading towards evolution or degradation? arXiv: 2303.01255, 2023. URL https://arxiv.org/abs/2303. 01255

    Pith/arXiv arXiv 2023

  26. [26]

    Bartlett

    Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distillation amplifies regular- ization in Hilbert space. InAdvances in Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2002.05715

    Pith/arXiv arXiv 2020

  27. [27]

    Understanding the gains from repeated self-distillation

    Divyansh Pareek, Simon Shaolei Du, and Sewoong Oh. Understanding the gains from repeated self-distillation. InAdvances in Neural Information Processing Systems, 2024. URL https: //arxiv.org/abs/2407.04600

    Pith/arXiv arXiv 2024

  28. [28]

    Tibshirani

    Pratik Patil, Yuting Wei, Alessandro Rinaldo, and Ryan J. Tibshirani. Uniform consis- tency of cross-validation estimators for high-dimensional ridge regression. InProceed- ings of the International Conference on Artificial Intelligence and Statistics, 2021. URL https://pratikpatil.io/papers/ridgecv-combined.pdf

    2021

  29. [29]

    Tibshirani

    Pratik Patil, Alessandro Rinaldo, and Ryan J. Tibshirani. Estimating functionals of the out-of- sample error distribution in high-dimensional ridge regression. InInternational Conference on Artificial Intelligence and Statistics, 2022. URL https://pratikpatil.io/papers/ functionals-combined.pdf

    2022

  30. [30]

    Mary Phuong and Christoph H. Lampert. Towards understanding knowledge distillation. InProceedings of the International Conference on Machine Learning, 2019. URL https: //arxiv.org/abs/2105.13093. 11

    Pith/arXiv arXiv 2019

  31. [31]

    Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky T. Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings. InProceedings of the International Conference on Machine Learning, 2023. URL https://arxiv.org/abs/2304.14772

    Pith/arXiv arXiv 2023

  32. [32]

    Shreshth Saini, Shashank Gupta, and Alan C. Bovik. Rectified-cfg++ for flow based models. In Advances in Neural Information Processing Systems, volume 38, pages 149034–149074, 2025. URLhttps://arxiv.org/abs/2510.07631

    arXiv 2025

  33. [33]

    Neta Shaul, Ricky T. Q. Chen, Maximilian Nickel, Matthew Le, and Yaron Lipman. On kinetic optimal probability paths for generative models. InProceedings of the International Conference on Machine Learning, 2023. URLhttps://arxiv.org/abs/2306.06626

    Pith/arXiv arXiv 2023

  34. [34]

    AI models collapse when trained on recursively generated data.Nature, 631(8022): 755–759, 2024

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data.Nature, 631(8022): 755–759, 2024. URLhttps://www.nature.com/articles/s41586-024-07566-y

    2024

  35. [35]

    Understanding linear probing then fine-tuning language models from NTK perspective

    Akiyoshi Tomihari and Issei Sato. Understanding linear probing then fine-tuning language models from NTK perspective. InAdvances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/abs/2405.16747

    Pith/arXiv arXiv 2024

  36. [36]

    Improving and generalizing flow-based generative models with minibatch optimal transport.Transactions on Machine Learning Research, 2024

    Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport.Transactions on Machine Learning Research, 2024. URLhttps://arxiv.org/abs/2302.00482

    Pith/arXiv arXiv 2024

  37. [37]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. InProceedings of the International Conference on Computer Vision, 2019. URL https://arxiv.org/abs/1905.08094

    Pith/arXiv arXiv 2019

  38. [38]

    Optimal Self-Distillation for Rectified Flow via Linear Probing

    Huminhao Zhu, Fangyikang Wang, Tianyu Ding, Qing Qu, and Zhihui Zhu. Analyzing and mitigating model collapse in rectified flow models. arXiv:2412.08175, 2024. URL https: //arxiv.org/abs/2412.08175. 12 Supplement This supplement accompanies the paper “Optimal Self-Distillation for Rectified Flow via Linear Probing”. It contains proofs, further theoretical ...

    Pith/arXiv arXiv 2024

  39. [2020]

    URLhttps://arxiv.org/abs/2010.10090

    Pith/arXiv arXiv 2010

  40. [2023]

    URLhttps://arxiv.org/abs/2209.15571

    Pith/arXiv arXiv

  41. [2024]

    URLhttps://arxiv.org/abs/2307.01850

    Pith/arXiv arXiv

  42. [2026]

    URLhttps://pratikpatil.io/papers/sd-ridge.pdf