Pith. sign in

REVIEW 3 major objections 6 minor 67 references

A fixed sliding window plus cross-modal recurrent memory turns a slow bidirectional audio-video model into a real-time long-form streamer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 20:11 UTC pith:Y7HFCRF5

load-bearing objection Solid systems paper: constant-cost dual-stream AV with a real memory recipe; long-horizon win is real but only half-compared to other streamers. the 3 major comments →

arxiv 2607.26818 v1 pith:Y7HFCRF5 submitted 2026-07-29 cs.CV

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

Yanbo Ding , Zhizhi Guo , Quanyue Song , Yishan He , Zhixiang He , Yongxiang Li , Yali Wang This is my paper
classification cs.CV
keywords streaming audio-video generationcross-modal recurrent memorysliding-window attentiondistribution matching distillationcausal diffusiononline reinforcement learningreal-time generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Offline joint audio-video generators look good but cannot run live: they need the full future context and pay growing cost as sequences lengthen. Prior streaming methods still keep an ever-growing history cache, so latency climbs and long clips break. Ripple claims you can keep quality and synchronization while holding cost constant by pairing a short sliding attention window with small modality-specific memory states that continually compress past audio and video, then briefly exchange information so each modality can condition the other. A three-stage recipe—causal mask adaptation with simulated memory, memory-forcing distillation from a bidirectional teacher, and online dual-stream reinforcement—teaches the student to use that memory under true streaming conditions. The result is roughly 28 frames per second at 480p on one high-end GPU, more than fifteen times faster than the teacher, with coherent generation past the training horizon and stronger scores than existing offline and online joint methods on short and 30-second benchmarks.

Core claim

Ripple shows that fixed-length sliding-window attention, augmented by modality-specific recurrent memory states that are updated from evicted keys and values via EMA and then cross-attended, is enough to preserve long-range temporal coherence and audio-visual sync in causal block-wise joint generation—yielding real-time (~28 FPS at 480P) constant-cost inference over 15× faster than the bidirectional teacher while matching or beating prior offline and streaming systems on short and long benchmarks.

What carries the argument

Cross-modal recurrent memory (construct-then-interact): each modality maintains a fixed set of learnable memory queries that read out from KV pairs leaving the sliding window, update via EMA with L2 normalization, then exchange information through residual cross-modal attention before being prepended (with a first-block sink) as long-term key-value context for every new block.

Load-bearing premise

A small fixed memory bank plus one cross-modal exchange, sitting beside a short window and the first block, really is a good enough summary of all past audio and video that throwing away the full history does not break coherence or sync.

What would settle it

Generate minute-scale clips on diverse open-domain prompts (not only talking heads) with and without the memory bank; if identity, timbre, lip sync, or narrative continuity collapse relative to a full-cache or bidirectional baseline once the window rolls, the memory-sufficiency claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Joint audio-video generation can run interactively at constant per-block cost instead of growing with history length.
  • Long-form (30s to minute-level) streaming AV becomes practical on a single GPU without re-encoding the full past.
  • The three-stage recipe—causal adaptation, memory-forcing distillation, then dual-stream RL—offers a transferable path from bidirectional teachers to streaming students.
  • Prompt switches can reset memory to zero, enabling interactive scene changes without carrying prior context.
  • Existing short-horizon streaming AV systems that rely on unbounded KV caches become less necessary once recurrent cross-modal memory works.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same construct-then-interact memory pattern could transfer to other dual-stream causal generators (e.g., speech-to-gesture or text-to-music-video) where unbounded caches are the bottleneck.
  • The paper’s own note on sink-induced motion anchoring suggests future work should condition identity/timbre explicitly so the first block need not pin later dynamics.
  • If EMA memory is sufficient at α≈0.9, coarser hierarchical memories (seconds vs minutes) might further extend horizon without enlarging the window.
  • Online RL with progressive alignment-then-quality rewards may become standard post-training once streaming rollouts are cheap enough to run in the loop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. Ripple is a streaming joint audio-video generator that replaces an ever-growing KV cache with a fixed-length sliding window plus modality-specific recurrent memory (learnable queries over evicted KV, EMA update with L2 norm, residual cross-modal memory attention) and a first-block sink. A three-stage recipe adapts a bidirectional LTX-2.3 teacher to block-causal attention with simulated memory, then memory-forcing DMD distillation, then GRPO-style online RL with a composite AV/sync/speech/quality reward. The system reports ~28 FPS at 480P on one H100 (~15× faster than the teacher), competitive VerseBench Set-3 quality versus offline and online baselines, and better ID consistency / AV quality than the teacher on a custom 50-prompt 30 s long-video benchmark where prior streamers fail to run.

Significance. Real-time joint AV generation with bounded cost and long-horizon coherence is a genuine systems gap: prior dual-stream self-forcing methods (OmniForcing, Hallo-Live) remain short-horizon and cache-growing. The construct-then-interact memory design, the staged causal adaptation + memory-forcing + dual-stream RL recipe, and the concrete ~28 FPS / 15× latency result are concrete contributions. Strengths include external metrics (DNSMOS, HPS, Synchformer, ASR WER/CER, SyncNet, ImageBind), staged ablations (Tables 3–6), a small user study, and an explicit limitation note on sink-induced motion anchoring. If the long-horizon sufficiency claim holds more broadly, the work is a useful reference architecture for streaming multimodal generation.

major comments (3)
  1. [Table 2, Long-Video Benchmark, Eqs. 1–4] Table 2 and the long-video qualitative panel only compare Ripple to its bidirectional teacher LTX-2.3; OmniForcing and Hallo-Live are stated to fail beyond ~5–15 s. The central claim that fixed-window + EMA memory (Eqs. 1–4, α=0.9, Nv=512, Na=32) + sink is a sufficient long-horizon summary—and is what preserves ID consistency / AV quality when the full KV cache is discarded—is therefore only half-tested. There is no head-to-head evidence that the memory mechanism, rather than simply being the first bounded-cache system that can finish 30 s, drives the long-form gains. A controlled ablation that forces a prior streamer (or a no-memory / no-interaction Ripple variant) to the same 30 s horizon under identical compute, or a longer public benchmark with at least one runnable streaming baseline, is needed to support the long-form superiority claim.
  2. [Limitation and Discussion; Streaming Inference] The paper’s own Limitation section notes that the first-block sink anchors identity/layout at the cost of reduced motion magnitude in long rollouts. This directly qualifies the long-form coherence claim and is not quantified (e.g., motion magnitude, optical-flow energy, or diversity metrics over 30 s). Without such numbers, it is unclear how much of the reported ID-consistency gain is desirable preservation versus undesirable temporal freezing, and whether the memory mechanism mitigates or compounds the sink effect.
  3. [Dataset Preparation; Long-Video Benchmark; Fig. 6] Training data and the long-video benchmark are heavily human-centric (talking-head sources + 50 custom prompts). Open-domain qualitative examples (Fig. 6) are encouraging but unquantified. The sufficiency of modality-specific EMA memory for non-speech, multi-object, or camera-heavy scenes remains an untested load-bearing assumption for the generality of the real-time long-form claim.
minor comments (6)
  1. [Abstract; Introduction] Abstract and intro say “over faster” / “over 15× faster”; keep a single consistent factor and state the exact teacher latency configuration used for the ratio.
  2. [Cross-Modal Memory Interaction] Eqs. (3)–(4) residual cross-modal exchange and the subsequent K/V projections are clear, but the shared dimension dc and whether interaction is applied every block or only on eviction are not stated in the main text.
  3. [Table 1] Table 1 reports three separate Ripple rows under different comparison settings (full offline, vs OmniForcing, vs Hallo-Live) with different latencies (5.9 s / 4.2 s / 3.9 s); clarify whether these are different resolutions, first-frame conditions, or run configurations.
  4. [Dataset Preparation] Stage-1 uses only ~4K ODE trajectories while Stages 2–3 use the full ~3M-clip set; a short note on whether Stage-1 data diversity limits later memory utilization would help.
  5. [Human Evaluation] User study (Table 7) is 25 videos × 30 participants with preference only; reporting breakdowns by quality / sync / speech would strengthen the human evaluation claim.
  6. [Throughout] Minor typos and spacing artifacts (“Audio-videogenerativemodels”, “inferencerollout”, “block-wisecausal”) should be cleaned in a revision.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper with external metrics, frozen teacher distillation, and ablated design choices.

full rationale

Ripple is an engineering/systems paper. Its central claims (constant-cost streaming via sliding window + modality-specific EMA memory + cross-modal interaction; ~28 FPS / ~15× speedup; long-form coherence; superiority on VerseBench and a 30s benchmark) are supported by implementation, external reward/metric models (Synchformer, Whisper ASR WER/CER, HPS, DNSMOS, SyncNet, ImageBind, CLAP), distillation against a frozen bidirectional teacher (LTX-2.3) under DMD, and GRPO-style RL with KL to a reference policy. Equations (1)–(4) define a construct-then-interact memory mechanism; they do not redefine the reported quality or latency numbers. Hyperparameters (α, Nv, Na) are ablated in Table 5 rather than fitted to equal the headline result. Self-citations are ordinary related-work pointers, not load-bearing uniqueness theorems. Evaluation-design concerns (custom 50-prompt long-video set; streaming baselines failing at 30s) affect external validity, not circularity of a derivation. No step reduces a claimed prediction to its own fitted input by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The result rests on standard diffusion/flow-matching and distillation machinery, the assumption that a bidirectional joint AV teacher can be causally adapted, and several design choices (EMA memory, sink tokens, reward heads) introduced for this system. No new physical entities; free parameters are architectural and training knobs selected by ablation rather than fit to hide the claim.

free parameters (5)
  • EMA memory ratio α = 0.9
    Balances retained memory vs new readout; chosen by ablation (best at 0.9). Directly affects long-horizon stability.
  • Video/audio memory token counts Nv, Na = Nv=512, Na=32
    Fixed memory capacity; selected via ablation (512 / 32) under latency constraints.
  • RL reward weights and two-phase schedule = R1: 0.5 rav+0.5 rsp; R2: 0.1(rav+rsp)+0.2 raud+0.6 rvid
    Composite weights (e.g. 0.5/0.5 then 0.1/0.1/0.2/0.6) and 200+200 step split are hand-designed; ablated vs one-stage mix.
  • KL penalty coefficient β and advantage clipping = β=0.1
    Stabilizes GRPO updates; β=0.1, advantages clipped to [-5,5], rewards to [0,1].
  • Sliding window / block structure and four-step student = 4-step student; block-wise window (details in method/supplement)
    Defines the causal student capacity and latency target; inherited from self-forcing style setups and fixed for experiments.
axioms (6)
  • domain assumption Flow-matching / diffusion denoising objectives and RoPE-based DiT dual-stream backbones are valid generative models for joint AV latents.
    Taken from LTX-2.3 and standard DiT literature; foundation of teacher and student.
  • domain assumption Distribution Matching Distillation (DMD) transfers bidirectional teacher sampling behavior into a few-step causal student.
    Stage 2 rests on Yin et al. DMD; paper does not re-prove DMD correctness.
  • ad hoc to paper Evicted-block KV summaries via learnable queries + EMA + residual cross-modal memory attention preserve enough long-range AV information for coherence under a fixed window.
    Core modeling hypothesis of §Cross-Modal Recurrent Memory; supported by ablations but not derived.
  • ad hoc to paper First-block sink KV cache is a stable global identity/layout anchor across long rollouts.
    Used in attention context composition and streaming inference; paper later notes it limits motion diversity.
  • domain assumption External reward models (Synchformer, Whisper ASR WER/CER, HPSv2, DNSMOS) are adequate proxies for perceptual AV quality, sync, and speech alignment in RL.
    Stage 3 optimizes these scores; validity is standard but imperfect proxy assumption.
  • domain assumption Training primarily on filtered human-centric talking-head video transfers sufficiently to the reported benchmarks (and claimed open-domain generalization).
    Dataset section; open-domain figures are qualitative only.
invented entities (2)
  • Cross-modal recurrent memory (modality-specific memory states + construct-then-interact cross-modal memory attention) no independent evidence
    purpose: Compress history beyond the sliding window and exchange audio/video summaries for long-range coherence and sync at constant cost.
    Central architectural invention; evidenced only by this paper’s ablations and benchmarks, not by independent external measurement.
  • Memory-forcing distillation stage no independent evidence
    purpose: End-to-end train memory construction/interaction under DMD while rolling memory during student generation.
    Named training procedure specific to this work; effectiveness shown only in their stage ablations.

pith-pipeline@v1.2.0-daily-grok45 · 25337 in / 4024 out tokens · 78772 ms · 2026-07-30T20:11:03.637064+00:00 · methodology

0 comments
read the original abstract

Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.

Figures

Figures reproduced from arXiv: 2607.26818 by Quanyue Song, Yali Wang, Yanbo Ding, Yishan He, Yongxiang Li, Zhixiang He, Zhizhi Guo.

Figure 1
Figure 1. Figure 1: Motivation. Existing streaming audio-video meth￾ods rely on an ever-growing KV cache, leading to increasing inference cost. Our Ripple combines a fixed-length sliding window with cross-modal recurrent memory for constant￾cost inference and long-range coherence. Additionally, we introduce reinforcement post-training for joint generation, a key training difference from prior streaming AV approaches. methods … view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of Cross-modal Recurrent Memory Mechanism. (a) Modality-specific recurrent memory construction: audio and video memory states are independently updated to summarize historical context. (b) Cross-modal memory interaction: the two memory streams exchange information before being injected as key-value pairs into transformer layers. (iii) the previous-block KV cache, which captures the most recent… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of our Three-Stage Training Recipe. (1) Memory-augmented block-causal mask adaptation: the bidi￾rectional teacher backbone is adapted to causal, block-by-block attention with simulated memory derived from inference rollout, allowing the model to learn the new causal attention pattern. (2) Cross-modal memory-forcing distillation: the complete model is trained end-to-end under a distillation obj… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Comparison.Compared with existing offline and online streaming audio-video generation methods, Ripple achieves high-quality generation on short videos while enabling real-time streaming inference. On long videos, Ripple achieves better temporal consistency than the teacher, demonstrating the effectiveness of our memory mechanism. In the transcribed speech, red indicates mispronounced/incorrect … view at source ↗
Figure 5
Figure 5. Figure 5: Examples from our Long-Video Benchmark. Each example consists of a prompt and its corresponding reference image, where the prompt includes the subject name and speech content for 30-second joint audio-video generation. Model Preference Rate (%) OmniForcing (Su et al. 2026) 20.36 Hallo-Live (Li et al. 2026) 30.53 Ripple (Ours) 49.11 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison on Open-Domain Scenes. Although trained primarily on human-centered data, Ripple general￾izes well to diverse scenarios, achieving generation quality comparable to the teacher while running significantly faster. With Cross-Modal Memory (Ours) Without Cross-Modal Memory 0s 10s 20s 0s 10s 20s Amplitude Amplitude Time Time [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative Comparison With and Without Memory Interaction. The memory interaction module slightly improves audio-visual synchronization, leading to more accurate lip articulation and phoneme pronunciation [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Audio Waveform Comparison With and With￾out Cross-Modal Memory. RMS energy envelopes of gen￾erated audio for a drumming scene. With audio memory, the rhythmic energy patterns remain stable and regular through￾out the sequence. Without audio memory, noticeable timbre drift and irregular energy fluctuations appear in the later segments (e.g., after 10 s), indicating that the memory mech￾anism effectively pre… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative Comparison.Compared with existing offline and online streaming audio-video generation methods, Ripple achieves high-quality generation on short videos while enabling real-time streaming inference. On long videos, Ripple achieves better temporal consistency than the teacher, demonstrating the effectiveness of our memory mechanism. In the transcribed speech, red indicates mispronounced/incorrect … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 24 linked inside Pith

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    arXiv preprint arXiv:2503.20314 , year=

    Wan: Open and advanced large-scale video generative models , author=. arXiv preprint arXiv:2503.20314 , year=

    Pith/arXiv arXiv

  13. [13]

    International Conference on Learning Representations , volume=

    Cogvideox: Text-to-video diffusion models with an expert transformer , author=. International Conference on Learning Representations , volume=

  14. [14]

    arXiv preprint arXiv:2511.18870 , year=

    Hunyuanvideo 1.5 technical report , author=. arXiv preprint arXiv:2511.18870 , year=

    Pith/arXiv arXiv

  15. [15]

    Advances in Neural Information Processing Systems , volume=

    Self forcing: Bridging the train-test gap in autoregressive video diffusion , author=. Advances in Neural Information Processing Systems , volume=

  16. [16]

    Journal of quality technology , volume=

    The exponentially weighted moving average , author=. Journal of quality technology , volume=. 1986 , publisher=

    1986

  17. [17]

    arXiv preprint arXiv:2505.07818 , year=

    Dancegrpo: Unleashing grpo on visual generation , author=. arXiv preprint arXiv:2505.07818 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2509.06155 , year=

    UniVerse-1: Unified Audio-Video Generation via Stitching of Experts , author=. arXiv preprint arXiv:2509.06155 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vbench: Comprehensive benchmark suite for video generative models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  20. [20]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2010.02502 , year=

    Denoising diffusion implicit models , author=. arXiv preprint arXiv:2010.02502 , year=

    Pith/arXiv arXiv 2010

  22. [22]

    arXiv preprint arXiv:2311.15127 , year=

    Stable video diffusion: Scaling latent video diffusion models to large datasets , author=. arXiv preprint arXiv:2311.15127 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2604.14148 , year=

    Seedance 2.0: Advancing video generation for world complexity , author=. arXiv preprint arXiv:2604.14148 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2509.20328 , year=

    Video models are zero-shot learners and reasoners , author=. arXiv preprint arXiv:2509.20328 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2501.00103 , year=

    Ltx-video: Realtime video latent diffusion , author=. arXiv preprint arXiv:2501.00103 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2601.03233 , year=

    LTX-2: Efficient Joint Audio-Visual Foundation Model , author=. arXiv preprint arXiv:2601.03233 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2602.08794 , year=

    Mova: Towards scalable and synchronized video-audio generation , author=. arXiv preprint arXiv:2602.08794 , year=

    arXiv

  28. [28]

    arXiv preprint arXiv:2503.23377 , year=

    Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization , author=. arXiv preprint arXiv:2503.23377 , year=

    arXiv

  29. [29]

    Proceedings of the 40th International Conference on Machine Learning , pages=

    Consistency models , author=. Proceedings of the 40th International Conference on Machine Learning , pages=

  30. [30]

    Advances in neural information processing systems , volume=

    Improved distribution matching distillation for fast image synthesis , author=. Advances in neural information processing systems , volume=

  31. [31]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    From slow bidirectional to fast autoregressive video diffusion models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  32. [32]

    arXiv preprint arXiv:2602.02214 , year=

    Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation , author=. arXiv preprint arXiv:2602.02214 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2603.11647 , year=

    Omniforcing: Unleashing real-time joint audio-visual generation , author=. arXiv preprint arXiv:2603.11647 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2602.19163 , year=

    Javisdit++: Unified modeling and optimization for joint audio-video generation , author=. arXiv preprint arXiv:2602.19163 , year=

    arXiv

  35. [35]

    ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Synchformer: Efficient synchronization from sparse cues , author=. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2024 , organization=

    2024

  36. [36]

    arXiv preprint arXiv:2306.09341 , year=

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis , author=. arXiv preprint arXiv:2306.09341 , year=

    Pith/arXiv arXiv

  37. [37]

    ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors , author=. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2021 , organization=

    2021

  38. [38]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  39. [39]

    Advances in Neural Information Processing Systems , volume=

    Diffusion forcing: Next-token prediction meets full-sequence diffusion , author=. Advances in Neural Information Processing Systems , volume=

  40. [40]

    International conference on machine learning , pages=

    Robust speech recognition via large-scale weak supervision , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  41. [41]

    arXiv preprint arXiv:2510.01284 , year=

    Ovi: Twin backbone cross-modal fusion for audio-video generation , author=. arXiv preprint arXiv:2510.01284 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2603.21986 , year=

    Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model , author=. arXiv preprint arXiv:2603.21986 , year=

    arXiv

  43. [43]

    arXiv preprint arXiv:2604.23632 , year=

    Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation , author=. arXiv preprint arXiv:2604.23632 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  46. [46]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  47. [47]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Scalable diffusion models with transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  48. [48]

    arXiv preprint arXiv:2510.02283 , year=

    Self-forcing++: Towards minute-scale high-quality video generation , author=. arXiv preprint arXiv:2510.02283 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2509.22622 , year=

    Longlive: Real-time interactive long video generation , author=. arXiv preprint arXiv:2509.22622 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2210.02747 , year=

    Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

    Pith/arXiv arXiv

  51. [51]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Imagebind: One embedding space to bind them all , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  52. [52]

    Advances in neural information processing systems , volume=

    Improved techniques for training gans , author=. Advances in neural information processing systems , volume=

  53. [53]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  54. [54]

    arXiv e-prints , pages=

    Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation , author=. arXiv e-prints , pages=

  55. [55]

    Proceedings of the 28th ACM international conference on multimedia , pages=

    A lip sync expert is all you need for speech to lip generation in the wild , author=. Proceedings of the 28th ACM international conference on multimedia , pages=

  56. [56]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Rethinking the inception architecture for computer vision , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  57. [57]

    arXiv preprint arXiv:2304.07193 , year=

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    Pith/arXiv arXiv

  58. [58]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=. 2020 , publisher=

    2020

  59. [59]

    International Conference on Medical image computing and computer-assisted intervention , pages=

    U-net: Convolutional networks for biomedical image segmentation , author=. International Conference on Medical image computing and computer-assisted intervention , pages=. 2015 , organization=

    2015

  60. [60]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  61. [61]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vfhq: A high-quality dataset and benchmark for video face super-resolution , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  62. [62]

    arXiv preprint arXiv:1806.05622 , year=

    Voxceleb2: Deep speaker recognition , author=. arXiv preprint arXiv:1806.05622 , year=

    Pith/arXiv arXiv

  63. [63]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Celebv-text: A large-scale facial text-video dataset , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  64. [64]

    arXiv preprint arXiv:1804.03619 , year=

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation , author=. arXiv preprint arXiv:1804.03619 , year=

    Pith/arXiv arXiv

  65. [65]

    arXiv preprint arXiv:2412.00115 , year=

    OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation , author=. arXiv preprint arXiv:2412.00115 , year=

    Pith/arXiv arXiv

  66. [66]

    Asian conference on computer vision , pages=

    Out of time: automated lip sync in the wild , author=. Asian conference on computer vision , pages=. 2016 , organization=

    2016

  67. [67]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Clap learning audio concepts from natural language supervision , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023