Pith. sign in

REVIEW 4 major objections 4 minor 124 references

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LeapTalk claims that real-time, open-ended talking-head generation is achievable in a single forward step per video chunk, using a Brownian-bridge student distilled from a diffusion teacher, with identity preserved over arbitrarily long str

desk verdict Genuinely novel bridge-forcing chunking idea, but the heterogeneous DMD core is under-derived and the experimental support is too thin to accept the headline numbers at face value. read the letter →

arxiv 2608.00079 v1 pith:2FHKMLSU submitted 2026-07-29 cs.CV cs.SD

Rongxiang Zhang , Songhua Liu This is my paper · ORCID
classification cs.CVcs.SD
keywords talking-headgenerationsingle-stepdistillationBrownianbridgestreamingvideosynthesisdistributionmatchingSNR-alignedtimetransformationaudio-drivenanimationidentityconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LeapTalk claims that real-time, open-ended talking-head generation is possible at one diffusion step per chunk, without the usual speed-versus-quality trade-off. Its idea is to stop generating each new video chunk from random noise, which causes errors to accumulate; instead, every chunk is generated along a Brownian bridge that starts from the same reference image and lands on the target frames. A flow-matching diffusion teacher is distilled into this one-step bridge student using a time transformation that matches signal-to-noise ratios, and an audio-conditioned guidance term keeps lip motion sharp. If correct, a single GPU can synthesize 55-200 frames per second while preserving identity over 30 minutes or more, which would make live interactive digital humans practical.

What carries the argument

The Brownian bridge X_tau = (1-tau)I + tau*X1 + sqrt(tau(1-tau))*epsilon, with I being a persistent reference image and X1 the target chunk, turns each chunk into data-to-data transport. Bridge Forcing builds chunk inputs by splicing the last K frames of the previous chunk onto repeated reference frames, preserving motion continuity while re-anchoring identity. The SNR-aligned time transform t = Phi(tau) = 1/(1 + sqrt((1-tau)/tau)) lets a flow-matching teacher supervise a bridge student at matched noise levels. Audio-driven classifier-free guidance replaces the teacher score with a conditional/unconditional combination to preserve lip detail and motion diversity under one step.

What would settle it

Train the same bridge student with only the weighted reconstruction and perceptual losses, leaving out the score-matching DMD term under identical hyperparameters, and compare FID/FVD and Sync-C on HDTF and CelebV-HQ. If removing the DMD term does not degrade the metrics at one step, the SNR-aligned heterogeneous distillation is not the source of the gains and the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the latency-quality trade-off in talking-head generation is not fundamental: by reformulating chunk generation as a data-to-data transport problem anchored to a persistent reference image, a single forward step of a Brownian-bridge student can replace dozens of denoising steps of a flow-matching diffusion teacher. The student is trained with a heterogeneous distribution-matching distillation that maps teacher time to student time through the SNR equality t = 1/(1 + sqrt((1-tau)/tau)), and augments the teacher score with audio-driven classifier-free guidance. The result is a streaming model that generates each chunk in one step, matches or beats multi-step baselines

Load-bearing premise

The load-bearing premise is that matching signal-to-noise ratios via the time transformation makes the flow-matching teacher's score and the Brownian-bridge student's score comparable enough for distribution-matching distillation; if that comparison is not well-defined, the one-step student may only be learning from direct reconstruction loss, and the claimed DMD advantage would be unsupported.

Editorial extensions

If this is right

  • Real-time streaming: at chunk size 33 frames and one step per chunk, the next chunk is generated in about 20% of the current chunk's playback time, so playback can be continuous and gap-free.
  • Open-ended identity: because every chunk is anchored to the same reference image, identity consistency stays flat in long rollouts of 30 minutes or more, whereas noise-to-data autoregressive baselines visibly drift.
  • Lip-sync survives extreme step reduction: the audio-driven CFG term prevents static or mismatched mouth motion, giving Sync-C/Sync-D scores comparable to or better than 4-50-step diffusion baselines.
  • The heterogeneous distillation scheme offers a reusable recipe: a pre-trained flow-matching diffusion teacher can be distilled into a bridge-based student via the SNR-aligned time transformation, not only for talking heads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the SNR-aligned heterogeneous distillation could extend to other image-to-video or paired-translation tasks where a pre-trained noise-to-data diffusion model is too slow and a data-to-data bridge is a better prior, such as general image animation or controllable video editing.
  • Beyond the paper: the persistent-reference bridge design suggests the same architecture could be steered by non-audio controls (e.g., pose or text) while keeping identity anchored, enabling interactive agents; the paper does not test this.
  • Beyond the paper: because the bridge treats the reference image as a deterministic bias, extremely large deviations from the reference identity or head pose may be under-produced; a targeted test would measure head-pose diversity on side-view or profile inputs, which the paper only evaluates qualitatively.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LeapTalk, a framework for real-time, streaming talking-head generation. It replaces the usual noise-to-data autoregressive diffusion formulation with a reference-anchored Brownian-bridge process: each chunk is generated from a fixed reference image to a target frame, with the previous chunk's last frames as a motion prefix. To distill a pretrained flow-matching teacher into this one-step bridge student, the authors introduce a 'heterogeneous DMD' scheme with an SNR-aligned time transformation Φ(τ), plus an audio-driven classifier-free guidance term. The reported experiments claim 1-step inference at up to 200 FPS, stable identity over 30-minute streams, and FID/FVD/Sync-C/Sync-D improvements over multi-step baselines on HDTF and CelebV-HQ.

Significance. If the central claims are correct, this is a significant systems contribution: stable open-ended talking-head generation at 1 NFE with 55–200 FPS would break a real latency-quality trade-off. The bridge-forcing idea is simple and appealing, and the long-video qualitative results are striking. The paper also includes a fairly complete ablation structure and a user study. However, the theoretical centerpiece—heterogeneous DMD—is not rigorously defined: the score-matching gradient as written compares scores at points from two different forward processes, and the connection to any proper distribution-matching objective is asserted rather than derived. The empirical evaluation also lacks error bars and tunes hyperparameters on the test set. These issues are load-bearing for the paper's main novelty claim.

major comments (4)
  1. [Methods, 'Heterogeneous Distribution Matching Distillation', Eqs. (15)–(17); Algorithm 1 lines 12–19] The central DMD gradient is not well-defined. Eq. (17) subtracts a 'real' teacher score evaluated at xΦ(τ) from a 'fake' student score evaluated at xτ. These are two different random vectors produced by two different forward processes (flow-matching vs. Brownian bridge); equalizing a scalar SNR does not make the score functions comparable, and the difference is not the derivative of any KL divergence between student and teacher. Algorithm 1 never constructs xΦ(τ): line 15 calls V_real(xt, t, ...) and line 19 uses xt, but xt is never defined anywhere in the paper. If xt is a teacher-process sample, the teacher score is evaluated at a point unrelated to the student sample; if xt is the bridge state xτ, the teacher is queried outside its training distribution. The paper needs to (i) specify xΦ(τ) explicitly, (ii) derive the gradient from a well-defined distribution-matching objective, or (i
  2. [Methods, 'Bridge Training' and Eq. (18); Algorithm 1 lines 22–25] The final loss also contains L_rec and L_perc, which directly regress the one-step output to multi-step teacher rollouts. The reported ablation removes the time transformation but never removes the DMD term itself, so it is impossible to attribute the one-step quality to the distribution-matching component. Add an ablation trained with only L_rec + L_perc (no DMD) and one with only DMD (no regression/perceptual losses). Without these, the claim that heterogeneous DMD 'enables' high-quality one-step generation is unsupported.
  3. [Experiments, Implementation Details; Tables 1 and 3; Figure 8] All main quantitative results appear to come from a single run, with no error bars or random seeds reported. In addition, α and λ_perc are selected by looking at HDTF metrics (Figure 8 and Table 3) and then the same HDTF numbers are used in the headline table. The reported differences on HDTF (e.g., Sync-C 8.38 vs. 8.11) are within plausible run-to-run variation. Report mean±std over at least 3 seeds and perform hyperparameter selection on a validation split.
  4. [Methods, 'Reference-Anchored Bridge Forcing'; Figure 5] The DINO-similarity identity metric measures similarity to the reference image I, which is explicitly concatenated/repeated into every chunk input (Eqs. 10–11). A degenerate model that always copies the reference would score highly on this metric, so high DINO-vs-reference similarity is partly by construction. Report also identity consistency between generated frames at increasing temporal offsets, or evaluate baselines under the same reference-anchored chunk input, to separate the method's contribution from the conditioning signal.
minor comments (4)
  1. [Eqs. (12)–(17) and Algorithm 1] The paper mixes score functions s and velocity fields v without stating the conversion (e.g., Tweedie/score-velocity relation). Algorithm 1 computes v_cfg and then uses it as if it were a score. Please define the notation precisely.
  2. [Appendix, 'Derivation of the SNR-aligned Time Transformation'] Equation (14) is called 'Theorem 1,' but it is an algebraic solution to an SNR equality, not a theorem about score comparability. Rename and clarify what the mapping actually guarantees.
  3. [Abstract and Appendix Table 9] The main text claims 'up to 200 FPS,' but Appendix Table 9 shows that this is on H200; on A100 at 512×512 the speed is 104 FPS. Include the hardware and resolution caveat in the abstract or main text.
  4. [User Study, Table 4] User-study percentages are reported without confidence intervals or significance tests. Add them if space permits.

Circularity Check

1 steps flagged · score 4.0 of 10

The SNR-aligned time transformation is a definitional construction, and the claimed well-defined DMD score matching rests on that definition rather than on a derived equivalence.

  1. self definitional [Methods, Eq. (13)-(17); Appendix 'Derivation of the SNR-aligned Time Transformation', Eq. (32)-(37)]
    "Let SNRteacher(t) and SNRstudent(τ) denote the SNR of the flow-matching teacher and the bridge student, respectively. There exists a monotonic time transformation t=Φ(τ) that aligns the two processes such that SNRteacher(t)=SNRstudent(τ). ... Since the objective is to recover Xtgt, we take (1−τ)Xtgt as the signal component and treat τXsrc as a deterministic bias term. ... We align the two processes by enforcing SNRTeacher(t)=SNRStudent(τ) ... Solving for t yields ... t=Φ(τ)=1/(1+sqrt((1−τ)/τ))."

    The theorem is an algebraic restatement of the definition: Φ is solved from the very equality it is claimed to establish, and the 'signal vs bias' split is chosen to make SNRStudent take the desired form. The paper then calls this 'consistent score evaluation' and uses it to justify the DMD gradient in Eq. (17), where s_real(xΦ(τ)) and s_fake(xτ) are evaluated at states from different forward processes. Scalar SNR equality does not make the score difference a valid divergence gradient; the conclusion of 'well-defined score matching' is not derived from the theorem but assumed in the definition of alignment.

full rationale

Most empirical validation is external: FID/FVD, Sync-C/D, FPS, and user studies compare against unrelated baselines on HDTF and CelebV-HQ, and the one-step generator is also trained with direct regression and perceptual losses to teacher rollouts, so the framework as a whole is not a closed self-consistent loop. However, the central theoretical justification of heterogeneous DMD is circular in a load-bearing way: the SNR-aligned time transformation is constructed by solving for t=Φ(τ) from the equality SNR_teacher(t)=SNR_student(τ), so the theorem states only what was put into the definition. The paper then uses that definitional equality to claim 'consistent score evaluation' and a valid DMD objective, even though the teacher and student scores are evaluated at different states generated by different forward processes. This is a self-definitional step in the derivation chain, not an independent mathematical justification. Separately, Algorithm 1 calls the teacher on an undefined xt, which is a missing-definition issue rather than circularity. Overall, partial circularity in the theoretical claim, but the empirical comparisons are independent, so the score is moderate.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central contribution rests on (i) a representational assumption that a fixed reference image plus audio is sufficient to span the target chunk via a bridge, (ii) a pretrained flow-matching teacher whose multi-step rollouts are reliable targets, and (iii) an alignment assumption that matching a hand-defined SNR makes teacher and student scores comparable for DMD. Hyperparameters α and λperc are tuned on the same evaluation sets. No new physical or conceptual entities are introduced with independent falsifiable handles.

free parameters (4)
  • Audio CFG scale α = 1.6
    Chosen from HDTF sensitivity analysis (Fig 8) to maximize Sync-C; used for final HDTF/CelebV-HQ results without a separate validation split.
  • Perceptual loss weight λ_perc = 4.0
    Chosen via Table 3 sensitivity on PSNR/SSIM/LPIPS; λ_perc=8.0 severely degrades metrics, indicating instability.
  • Chunk length = 33 frames
    Selected from Table 8 to minimize T_gen/T_chunk at 512×512 on A100.
  • Bridge loss and spatial-mask weights (λ_br, λ_face, λ_lip) = not disclosed
    Appear in Eq. 18 and Algorithm 1 but no values are reported; they modulate the final objective and affect reproduction.
assumptions (5)
  • domain assumption Each target chunk can be modeled as the endpoint of a Brownian bridge from a fixed reference image I; the reference image carries enough identity information.
    Methods 'Reference-Anchored Bridge Forcing', Eq. 8-9. If I cannot represent new appearance or identity variation, the model cannot generate it.
  • domain assumption A pretrained flow-matching video diffusion model (Wan2.1-T2V) is a valid teacher, and its multi-step rollouts are reliable training targets.
    Used throughout the distillation; Algorithm 1 line 9 calls multiStepRollout(V_real). Teacher quality on VividHead is not independently verified.
  • ad hoc to paper Equalizing the hand-defined SNR makes teacher and student score functions comparable for DMD.
    Appendix 'Derivation of the SNR-aligned Time Transformation', Eq. 33-37. The mapping only equates a chosen SNR; it does not prove the two conditional distributions are comparable, yet Eq. 16 subtracts their scores.
  • standard math Brownian bridge Gaussian conditioning and the DMD gradient formula hold as standard mathematics.
    Preliminaries Eq. 3-6; Eq. 12 from Yin et al. 2024. No errors apparent in these background statements.
  • domain assumption SyncNet, DINOv2, FID/FVD, and the small user study reliably measure lip-sync, identity, and quality.
    Used in Experiments; automated metrics are partial proxies and the user study has n=30 with no confidence intervals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation." pith.science (2026). https://pith.science/paper/2FHKMLSU

@misc{pith2026260800079,
  author       = {Pith},
  title        = {Pith review of: LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FHKMLSU}},
  note         = {Machine review of arXiv:2608.00079}
}
abstract

Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation $\Phi(\tau)$, which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/

Figures

Figures reproduced from arXiv: 2608.00079 by the authors.

Figure 1
Figure 1. Overview of LeapTalk. Given audio and a reference image, our method enables open-ended streaming talking-head generation with consistent identity. It achieves 1-step inference per chunk at up to 200 FPS, delivering up to 15000× speedup while maintaining strong lip-sync accuracy. and static motion in one-step generation. In summary, our key contributions include: • We propose LeapTalk, a real-time, streaming talking￾… view at source ↗
Figure 2
Figure 2. Comparison between conventional noise-to-data flow matching and our Bridge Forcing paradigm. The identity consistency is preserved along bridge process, while error accumulates in the forward diffusion process. Heterogeneous Distribution Matching Distillation Real-time talking-head generation requires autoregressive generation with few-step (ideally 1 step) synthesis. We propose a heterogeneous DMD (See [PITH_FULL_… view at source ↗
Figure 3
Figure 3. Distillation pipeline of our approach. The one-step student generates frames conditioned on a static reference. To enable stable distillation across heterogeneous processes, we apply a time transformation t = Φ(τ ) to align noise levels between the flow-matching teacher and the bridge-based student, allowing consistent score supervision and a well-defined DMD objective. the student is defined on a Brownian bridge, r… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on long-video streaming generation. Frames are sampled from streaming rollouts at increasing time indices. The red boxes mark identity incon￾sistency and drift, and the blue boxes mark slight or incorrect lip-sync. LeapTalk maintains stable ident…
Figure 6
Figure 6. Figure 6: Ablation results demonstrating the contribution of each component in our framework. …Under... …Administration... …Job... …More... LeapTalk w/o Audio- driven CFG [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Visualization of ablation effects of audio-driven [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity analysis of audio-driven CFG. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: Generated results under diverse and challenging [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 9
Figure 9. Figure 9: Visual comparison across different VAEs. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 11
Figure 11. Figure 11: More qualitative results on challenging condi [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

124 extracted references · 33 linked inside Pith

  1. [2]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  2. [3]

    Zhu, Hao and Wu, Wayne and Zhu, Wentao and Jiang, Liming and Tang, Siwei and Zhang, Li and Liu, Ziwei and Loy, Chen Change , booktitle=

  3. [4]

    Diffusion Schr\"

    De Bortoli, Valentin and Thornton, James and Heng, Jeremy and Doucet, Arnaud , booktitle =. Diffusion Schr\"

  4. [6]

    International Conference on Machine Learning , year=

    FrameBridge: Improving Image-to-Video Generation with Bridge Models , author=. International Conference on Machine Learning , year=

  5. [7]

    2025 , eprint=

    Stochastic Interpolants: A Unifying Framework for Flows and Diffusions , author=. 2025 , eprint=

    2025

  6. [8]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  7. [9]

    Arxiv , year=

    Packing Input Frame Contexts in Next-Frame Prediction Models for Video Generation , author=. Arxiv , year=

  8. [10]

    2026 , eprint=

    Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation , author=. 2026 , eprint=

    2026

Show all 124 references
  1. [11]

    2024 , eprint=

    Rolling Diffusion Models , author=. 2024 , eprint=

    2024

  2. [12]

    2025 , eprint=

    End-to-End Training for Autoregressive Video Diffusion via Self-Resampling , author=. 2025 , eprint=

    2025

  3. [13]

    2025 , eprint=

    Stable Video Infinity: Infinite-Length Video Generation with Error Recycling , author=. 2025 , eprint=

    2025

  4. [14]

    2024 , eprint=

    Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion , author=. 2024 , eprint=

    2024

  5. [15]

    NeurIPS , year =

    FIFO-Diffusion: Generating Infinite Videos from Text without Training , author =. NeurIPS , year =

  6. [16]

    arXiv preprint arXiv:2603.04379 , year=

    Helios: Real Real-Time Long Video Generation Model , author=. arXiv preprint arXiv:2603.04379 , year=

  7. [17]

    2026 , eprint=

    SoulX-FlashHead: Oracle-guided Generation of Infinite Real-time Streaming Talking Heads , author=. 2026 , eprint=

    2026

  8. [18]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Ditto: Motion-space diffusion for controllable realtime talking head synthesis , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  9. [20]

    2026 , eprint=

    REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation , author=. 2026 , eprint=

    2026

  10. [21]

    2025 , eprint=

    Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation , author=. 2025 , eprint=

    2025

  11. [22]

    2025 , eprint=

    Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length , author=. 2025 , eprint=

    2025

  12. [23]

    arXiv:2304.07193 , year=

    DINOv2: Learning Robust Visual Features without Supervision , author=. arXiv:2304.07193 , year=

    arXiv

  13. [25]

    arXiv preprint arXiv:2403.17694 , year=

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation , author=. arXiv preprint arXiv:2403.17694 , year=

    arXiv

  14. [26]

    arXiv preprint arXiv:2412.04448 , year=

    MEMO: Memory-guided diffusion for expressive talking video generation , author=. arXiv preprint arXiv:2412.04448 , year=

    arXiv

  15. [27]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    SadTalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  16. [28]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  17. [29]

    2024 , booktitle=

    Video generation models as world simulators , author=. 2024 , booktitle=

    2024

  18. [31]

    arXiv preprint arXiv:1312.6114 , year=

    Auto-encoding variational bayes , author=. arXiv preprint arXiv:1312.6114 , year=

    arXiv

  19. [33]

    arXiv e-prints , pages=

    SoulX-LiveTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation , author=. arXiv e-prints , pages=

  20. [34]

    International Conference on Medical image computing and computer-assisted intervention , pages=

    U-net: Convolutional networks for biomedical image segmentation , author=. International Conference on Medical image computing and computer-assisted intervention , pages=. 2015 , organization=

    2015

  21. [35]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  22. [39]

    arXiv preprint arXiv:2501.00103 , year=

    LTX-Video: Realtime video latent diffusion , author=. arXiv preprint arXiv:2501.00103 , year=

    arXiv

  23. [43]

    arXiv preprint arXiv:2410.10122 , year=

    MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling , author=. arXiv preprint arXiv:2410.10122 , year=

    arXiv

  24. [45]

    Learning Transferable Visual Models From Natural Language Supervision , booktitle =

    Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever , editor =. Learning Transferable Visual Models From Natural La...

    2021

  25. [46]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    EchoMimic: Lifelike audio-driven portrait animations through editable landmark conditions , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  26. [47]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Sonic: Shifting focus to global audio perception in portrait animation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  27. [48]

    arXiv preprint arXiv:2411.19509 , year=

    Ditto: Motion-space diffusion for controllable realtime talking head synthesis , author=. arXiv preprint arXiv:2411.19509 , year=

  28. [49]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vfhq: A high-quality dataset and benchmark for video face super-resolution , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  29. [50]

    ECCV , month = Augest, year =

    Wang, Kaisiyuan and Wu, Qianyi and Song, Linsen and Yang, Zhuoqian and Wu, Wayne and Qian, Chen and He, Ran and Qiao, Yu and Loy, Chen Change , title =. ECCV , month = Augest, year =

  30. [51]

    Castellano, Brandon , title =

  31. [52]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Scalable diffusion models with transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  32. [53]

    Advances in neural information processing systems , volume=

    Gans trained by a two time-scale update rule converge to a local nash equilibrium , author=. Advances in neural information processing systems , volume=

  33. [54]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    One-step diffusion with distribution matching distillation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  34. [55]

    Advances in neural information processing systems , volume=

    wav2vec 2.0: A framework for self-supervised learning of speech representations , author=. Advances in neural information processing systems , volume=

  35. [56]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Effective whole-body pose estimation with two-stages distillation , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  36. [57]

    arXiv preprint arXiv:1804.03619 , year=

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation , author=. arXiv preprint arXiv:1804.03619 , year=

    arXiv

  37. [58]

    arXiv preprint arXiv:2510.02283 , year=

    Self-Forcing++: Towards Minute-Scale High-Quality Video Generation , author=. arXiv preprint arXiv:2510.02283 , year=

    arXiv

  38. [59]

    Asian conference on computer vision , pages=

    Out of time: automated lip sync in the wild , author=. Asian conference on computer vision , pages=. 2016 , organization=

    2016

  39. [60]

    FVD: A new metric for video generation , author=

  40. [61]

    Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei , booktitle=

  41. [62]

    arXiv preprint arXiv:2312.17090 , year=

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels , author=. arXiv preprint arXiv:2312.17090 , year=

    arXiv

  42. [63]

    arXiv preprint arXiv:2507.09862 , year=

    SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation , author=. arXiv preprint arXiv:2507.09862 , year=

    arXiv

  43. [65]

    arXiv preprint arXiv:2309.14509 , year=

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models , author=. arXiv preprint arXiv:2309.14509 , year=

    arXiv

  44. [66]

    arXiv preprint arXiv:2310.01889 , year=

    Ring attention with blockwise transformers for near-infinite context , author=. arXiv preprint arXiv:2310.01889 , year=

    arXiv

  45. [67]

    GitHub repository , howpublished =

    LightX2V Contributors , title =. GitHub repository , howpublished =. 2025 , publisher =

    2025

  46. [68]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Flash. Advances in Neural Information Processing Systems (NeurIPS) , year=

  47. [69]

    Dao, Tri , booktitle=. Flash

  48. [70]

    Advances in Neural Information Processing Systems , volume=

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision , author=. Advances in Neural Information Processing Systems , volume=

  49. [72]

    2025 , journal =

    LBM: Latent Bridge Matching for Fast Image-to-Image Translation , author=. 2025 , journal =

    2025

  50. [73]

    2025 , howpublished =

    Boer Bohan, Ollin , title =. 2025 , howpublished =

    2025

  51. [74]

    CVPR , year=

    Shen, Shuai and Zhao, Wenliang and Meng, Zibin and Li, Wanhua and Zhu, Zheng and Zhou, Jie and Lu, Jiwen , title=. CVPR , year=

  52. [75]

    2022 , eprint=

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author=. 2022 , eprint=

    2022

  53. [76]

    and Shechtman, Eli and Wang, Oliver , title =

    Zhang, Richard and Isola, Phillip and Efros, Alexei A. and Shechtman, Eli and Wang, Oliver , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

  54. [77]

    2026 , eprint=

    TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation , author=. 2026 , eprint=

    2026

  55. [78]

    2026 , eprint=

    EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control , author=. 2026 , eprint=

    2026

  56. [79]

    2023 , eprint=

    Flow Matching for Generative Modeling , author=. 2023 , eprint=

    2023

  57. [80]

    2021 , eprint=

    Score-Based Generative Modeling through Stochastic Differential Equations , author=. 2021 , eprint=

    2021

  58. [81]

    2023 , howpublished =

    Google , title =. 2023 , howpublished =

    2023

  59. [82]

    2025 , eprint=

    TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesis , author=. 2025 , eprint=

    2025

  60. [83]

    Proceedings of the IEEE international conference on computer vision , pages=

    Flownet: Learning optical flow with convolutional networks , author=. Proceedings of the IEEE international conference on computer vision , pages=

  61. [84]

    arXiv preprint arXiv:2304.11277 , year=

    Pytorch fsdp: experiences on scaling fully sharded data parallel , author=. arXiv preprint arXiv:2304.11277 , year=

    arXiv

  62. [85]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. J. Mach. Learn. Res. , volume =. 2020 , url =

    2020

  63. [86]

    arXiv preprint arXiv:2207.12598 , year=

    Classifier-free diffusion guidance , author=. arXiv preprint arXiv:2207.12598 , year=

    arXiv

  64. [87]

    2023 , eprint=

    FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling , author=. 2023 , eprint=

    2023

  65. [88]

    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Bardia Doosti and Shujon Naha and Majid Mirbagheri and David Crandall , title =. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

  66. [89]

    Baevski, A.; Zhou, Y.; Mohamed, A.; and Auli, M. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449--12460

    2020

  67. [90]

    Boer Bohan, O. 2025. TAEHV: Tiny AutoEncoder for Hunyuan Video. https://github.com/madebyollin/taehv

    2025

  68. [91]

    Brooks, T.; Peebles, B.; Holmes, C.; DePue, W.; Guo, Y.; Jing, L.; Schnurr, D.; Taylor, J.; Luhman, T.; Luhman, E.; Ng, C.; Wang, R.; and Ramesh, A. 2024. Video generation models as world simulators. In openai

    2024

  69. [92]

    Chadebec, C.; Tasar, O.; Sreetharan, S.; and Aubin, B. 2025. LBM: Latent Bridge Matching for Fast Image-to-Image Translation. arXiv preprint arXiv:2503.07535

    2025 arXiv

  70. [93]

    M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V

    Chen, B.; Monso, D. M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V. 2024. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. arXiv:2407.01392

    2024 arXiv

  71. [94]

    Chen, Z.; Cao, J.; Chen, Z.; Li, Y.; and Ma, C. 2025. EchoMimic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2403--2410

    2025

  72. [95]

    Chern, E.; Hu, Z.; Tang, B.; Su, J.; Chern, S.; Deng, Z.; and Liu, P. 2025. LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation. arXiv preprint arXiv:2512.23576

    2025

  73. [96]

    S.; and Zisserman, A

    Chung, J. S.; and Zisserman, A. 2016. Out of time: automated lip sync in the wild. In Asian conference on computer vision, 251--263. Springer

    2016

  74. [97]

    Cui, J.; Li, H.; Zhan, Y.; Shang, H.; Cheng, K.; Ma, Y.; Mu, S.; Zhou, H.; Wang, J.; and Zhu, S. 2025. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, 21086--21095

    2025

  75. [98]

    De Bortoli, V.; Thornton, J.; Heng, J.; and Doucet, A. 2021. Diffusion Schr\" o dinger Bridge with Applications to Score-Based Generative Modeling. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Syst...

    2021

  76. [99]

    Doosti, B.; Naha, S.; Mirbagheri, M.; and Crandall, D. 2020. HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  77. [100]

    Gan, Q.; Yang, R.; Zhu, J.; Xue, S.; and Hoi, S. 2025. OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation. arXiv preprint arXiv:2506.18866

    2025 arXiv

  78. [101]

    Gao, X.; Hu, L.; Hu, S.; Huang, M.; Ji, C.; Meng, D.; Qi, J.; Qiao, P.; Shen, Z.; Song, Y.; et al. 2025. Wan-s2v: Audio-driven cinematic video generation. arXiv preprint arXiv:2508.18621

    2025 arXiv

  79. [102]

    Google. 2023. Cross-platform, customizable ML solutions for live and streaming media. https://github.com/google-ai-edge/mediapipe

    2023

  80. [103]

    Guo, J.; Zhang, D.; Liu, X.; Zhong, Z.; Zhang, Y.; Wan, P.; and Zhang, D. 2024. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168

    2024 arXiv

  81. [104]

    Guo, Y.; Yang, C.; He, H.; Zhao, Y.; Wei, M.; Yang, Z.; Huang, W.; and Lin, D. 2025. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling. arXiv:2512.15702

    2025 arXiv

  82. [105]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30

    2017

  83. [106]

    Huang, X.; Li, Z.; He, G.; Zhou, M.; and Shechtman, E. 2025 a . Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. arXiv preprint arXiv:2506.08009

    2025 arXiv

  84. [107]

    Huang, Y.; Guo, H.; Wu, F.; Zhang, S.; Huang, S.; Gan, Q.; Liu, L.; Zhao, S.; Chen, E.; Liu, J.; and Hoi, S. 2025 b . Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length. arXiv:2512.04677

    2025 arXiv

  85. [108]

    Kim, J.; Kang, J.; Choi, J.; and Han, B. 2024. FIFO-Diffusion: Generating Infinite Videos from Text without Training. In NeurIPS

    2024

  86. [109]

    Li, W.; Pan, W.; Luan, P.-C.; Gao, Y.; and Alahi, A. 2025. Stable Video Infinity: Infinite-Length Video Generation with Error Recycling. arXiv:2510.09212

    2025

  87. [110]

    Lipman, Y.; Chen, R. T. Q.; Ben-Hamu, H.; Nickel, M.; and Le, M. 2023. Flow Matching for Generative Modeling. arXiv:2210.02747

    2023 arXiv

  88. [111]

    Liu, X.; Gong, C.; and Liu, Q. 2022. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. arXiv:2209.03003

    2022 arXiv

  89. [112]

    Mazumdar, S.; and Rakesh, V. K. 2026. TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation. arXiv:2603.06057

    2026

  90. [113]

    Meng, R.; Wang, Y.; Wu, W.; Zheng, R.; Li, Y.; and Ma, C. 2025. Echomimicv3: 1.3 b parameters are all you need for unified multi-modal and multi-task human animation. arXiv preprint arXiv:2507.03905

    2025

  91. [114]

    Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H. V.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; Howes, R.; Huang, P.-Y.; Xu, H.; Sharma, V.; Li, S.-W.; Galuba, W.; Rabbat, M.; Assran, M.; Ballas, N.; Synnaeve, G.; Misra, I.; Jegou, H.; Maira...

    2023

  92. [115]

    Qiu, H.; Xia, M.; Zhang, Y.; He, Y.; Wang, X.; Shan, Y.; and Liu, Z. 2023. FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling. arXiv:2310.15169

    2023 arXiv

  93. [116]

    Ruhe, D.; Heek, J.; Salimans, T.; and Hoogeboom, E. 2024. Rolling Diffusion Models. arXiv:2402.09470

    2024 arXiv

  94. [117]

    Shen, L.; Qian, Q.; Yu, T.; Zhou, K.; Yu, T.; Zhan, Y.; Wang, Z.; Tao, M.; Yin, S.; and Liu, S. 2025. SoulX-LiveTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation. arXiv e-prints, arXiv--2512

    2025

  95. [118]

    Shen, S.; Zhao, W.; Meng, Z.; Li, W.; Zhu, Z.; Zhou, J.; and Lu, J. 2023. DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation. In CVPR

    2023

  96. [119]

    P.; Kumar, A.; Ermon, S.; and Poole, B

    Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. arXiv:2011.13456

    2021 arXiv

  97. [120]

    Tan, Z.; Wang, Z.; Yang, X.; Liu, S.; and Wang, X. 2025. Vision Bridge Transformer at Scale. arXiv preprint arXiv:2511.23199

    2025

  98. [121]

    Tu, S.; Pan, Y.; Huang, Y.; Han, X.; Xing, Z.; Dai, Q.; Luo, C.; Wu, Z.; and Jiang, Y.-G. 2025. Stableavatar: Infinite-length audio-driven avatar video generation. arXiv preprint arXiv:2508.08248

    2025 arXiv

  99. [122]

    Unterthiner, T.; Van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2019. FVD: A new metric for video generation

    2019

  100. [123]

    Wan, T.; Wang, A.; Ai, B.; Wen, B.; Mao, C.; Xie, C.-W.; Chen, D.; Yu, F.; Zhao, H.; Yang, J.; et al. 2025. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314

    2025 arXiv

  101. [124]

    Wang, H.; Weng, Y.; Du, J.; Xu, H.; Wu, X.; He, S.; Yin, B.; Liu, C.; and Liu, Q. 2026. REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation. arXiv:2512.11229

    2026

  102. [125]

    Wang, Q.; Liu, M.; Hu, J.; Jiang, F.; and Xu, M. 2025 a . FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis. arXiv preprint arXiv:2504.04842

    2025 arXiv

  103. [126]

    Wang, Y.; Chen, Z.; Chen, X.; Zhu, J.; and Chen, J. 2025 b . FrameBridge: Improving Image-to-Video Generation with Bridge Models. In International Conference on Machine Learning

    2025

  104. [127]

    Weng, Y.; Wang, H.; Yu, Y.; Du, J.; He, S.; Wu, X.; and Xu, H. 2026. EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control. arXiv:2603.20307

    2026

  105. [128]

    Xiao, S.; Zhang, X.; Meng, D.; Wang, Q.; Zhang, P.; and Zhang, B. 2025. Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation. arXiv:2512.21734

    2025

  106. [129]

    Yang, S.; Kong, Z.; Gao, F.; Cheng, M.; Liu, X.; Zhang, Y.; Kang, Z.; Luo, W.; Cai, X.; He, R.; et al. 2025. Infinitetalk: Audio-driven video generation for sparse-frame video dubbing. arXiv preprint arXiv:2508.14033

    2025 arXiv

  107. [130]

    T.; and Park, T

    Yin, T.; Gharbi, M.; Zhang, R.; Shechtman, E.; Durand, F.; Freeman, W. T.; and Park, T. 2024. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6613--6623

    2024

  108. [131]

    Yu, T.; Qiao, Q.; Shen, L.; Zhou, K.; Hu, J.; Sheng, D.; Hu, B.; Qin, H.; Gao, J.; Zhou, C.; Yin, S.; and Liu, S. 2026. SoulX-FlashHead: Oracle-guided Generation of Infinite Real-time Streaming Talking Heads. arXiv:2602.07449

    2026

  109. [132]

    Zhang, L.; and Agrawala, M. 2025. Packing Input Frame Contexts in Next-Frame Prediction Models for Video Generation. Arxiv

    2025

  110. [133]

    Zhang, L.; Cai, S.; Li, M.; Wetzstein, G.; and Agrawala, M. 2025. Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  111. [134]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  112. [135]

    Zhang, Z.; Li, L.; Ding, Y.; and Fan, C. 2021. Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3661--3670

    2021

  113. [136]

    Zheng, Z.; Peng, X.; Yang, T.; Shen, C.; Li, S.; Liu, H.; Zhou, Y.; Li, T.; and You, Y. 2024. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404

    2024 arXiv

  114. [137]

    Zhong, Z.; Ji, Y.; Kong, Z.; Liu, Y.; Wang, J.; Feng, J.; Liu, L.; Wang, X.; Li, Y.; She, Y.; et al. 2025. AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement. arXiv preprint arXiv:2511.23475

    2025

  115. [138]

    Zhu, H.; Wu, W.; Zhu, W.; Jiang, L.; Tang, S.; Zhang, L.; Liu, Z.; and Loy, C. C. 2022. CelebV-HQ : A Large-Scale Video Facial Attributes Dataset. In ECCV

    2022

  116. [139]

    Zhu, H.; Zhao, M.; He, G.; Su, H.; Li, C.; and Zhu, J. 2026. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. arXiv:2602.02214

    2026 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.