Pith. sign in

REVIEW 3 major objections 4 minor 49 references

UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single audio-video diffusion transformer can stream both face and voice replacement for talking videos, with better lip-sync than cascaded systems.

desk verdict UniSwap is a well-engineered streaming audio-visual identity-swap system, but its headline sync advantage over cascades is not proven until a lip-sync re-anchored cascade baseline is tested. read the letter →

arxiv 2608.11752 v2 pith:QZBPYOQ6 submitted 2026-08-12 cs.CV cs.SD

Yuxuan Zhang , Haozhong Xiong , Jiayi Song , Jinpeng Yu , Yang Shi , Jiaming Liu , Ruihua Huang , Liwei Wang This is my paper · ORCID
classification cs.CVcs.SD
keywords audio-visualidentityswappingtalkingvideogenerationstreamingdiffusiontransformervoiceconversionswap-and-reconstructtrainingdistributionmatchingdistillationpositionalencodingforlong-form
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to establish that a single streaming model, rather than a cascade of separately optimized modules, can replace both the face and the voice of a person in a talking video while preserving what the person says and how they move. If true, it matters because dubbing, localization, and interactive avatars need converted speech and lip motion to stay synchronized, and cascading visual replacement with voice conversion leaves the two modalities uncoordinated. To get training data for a task with no natural paired examples, the authors turn each ordinary video into its own aligned pair by stripping out the original identity and training the model to put it back. The reported payoff is better lip-sync than any evaluated cascade, comparable identity preservation, roughly ten times faster generation than the fastest evaluated baseline, and stable identity over minute-long outputs.

What carries the argument

The load-bearing object is a single audio-video diffusion transformer whose video and audio tokens share a physical-time axis and interact through cross-modal attention, so neither modality is generated without seeing the other. Three training adaptations carry the argument. In-context pretraining concatenates reference, source, and noisy target latents into one attention context with condition positional offsets. Conditional streaming adaptation then applies a Decoupled Streaming Conditioning Mask that gives each token region its inference-time receptive field, turning the bidirectional model into a block-causal KV-cached generator. Efficient self-forcing distribution-matching distillation rolls out the student's own predictions and distills 30 denoising steps down to 3 per block, using switchable low-rank adapters (LoRA) so teacher, generator, and critic share one frozen backbone. For long-form inference, Feature-RoPE Decomposition stores unrotated cached keys and re-applies bounded rotary positions, with a fixed sink block and re-anchored reference, keeping every position inside the training range.

What would settle it

A concrete timing-shift probe would settle the synchronization claim: train or fine-tune on swap-and-reconstruct pairs in which the converted source audio is delayed by roughly 100 ms relative to the video, then evaluate on normally timed test clips; if Sync-C does not degrade, the model is reproducing reference appearance and timbre while ignoring cross-modal timing, contradicting the claim that joint generation is what drives the lip-sync gain.

Watch

Extended reading notes

Core claim

UniSwap's central claim is that talking-video character replacement is best posed as one conditional audio-video generation problem rather than a cascade of a visual replacer and a voice converter. Given a source video with audio, a reference image, and a reference voice clip, a single audio-video diffusion transformer outputs synchronized video and audio carrying the reference appearance and vocal timbre while preserving source motion, background, and linguistic content. Because aligned cross-identity pairs do not exist at scale, every real clip becomes its own training pair: the visual identity is removed by replacing the person with a pose proxy on the masked background, the vocal timbre is randomized by voice conversion, and the model is trained to reconstruct the original clip from those identity-swapped conditions plus the original identity as references. The pipeline then converts a bidirectional backbone into a block-causal three-step generator through in-context pretraining, streaming adaptation with a decoupled conditioning mask, and self-forcing distribution-matching distillation. On the paper's benchmark, the joint model achieves the best audio-visual synchronization (Sync-C 3.633 versus 3.289 for the strongest cascade) with nearly identical identity preservation (DINO-S 0.629 versus 0.630), at 13.6 FPS on one H100, while holding identity roughly flat across one-minute videos.

Load-bearing premise

The load-bearing premise is that the swap-and-reconstruct pipeline's identity-removed source—a pose proxy composited onto the background plus machine-converted voice—preserves the original clip's motion, scene, speech content, and timing faithfully enough that reconstructing the original clip teaches real identity transfer; if either proxy is unfaithful, the model is trained on corrupted targets and the measured transfer is partly artifact.

Editorial extensions

If this is right

  • A cascade-free identity-swap pipeline becomes viable: one transformer produces converted speech and matching lip motion from a single reference image and voice clip, so no separate visual replacer and voice converter need to be reconciled.
  • Blockwise KV-cached inference makes per-block cost independent of total generated duration, so the same architecture can stream a short clip or an hour-long clip without reprocessing the full sequence at each step.
  • Three-step distillation lowers the per-block cost to 1.76 seconds for 24 frames (13.6 FPS on one H100), about 10 times faster than the fastest cascade the paper evaluates, though still below 25 FPS playback.
  • Long-form identity stability improves: DINO-S stays at 0.590–0.596 across three 20-second segments of one-minute videos, while a strong cascade baseline declines from 0.566 to 0.517.
  • Joint generation preserves speech quality close to the dedicated voice converter (SIG 3.486 versus 3.489), showing the synchronization gain is not purchased by degrading the converted voice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because swap-and-reconstruct turns any talking-video corpus into training data, the bottleneck shifts to the fidelity of the pose proxy and the voice converter; a testable extension is that raising proxy fidelity should improve identity-transfer accuracy more than adding model capacity.
  • The joint model's sync advantage over cascades suggests cross-modal information is doing real work; ablating audio conditioning from the video stream during inference would isolate how much of the Sync-C gain comes from seeing the converted speech.
  • Window-bounded positional remapping is a general cure for streaming diffusion drift; the same Feature-RoPE idea could be reused in other autoregressive multimodal diffusion models trained with absolute positions.
  • A deployment consequence the paper leaves implicit: at 13.6 FPS on one H100, real-time playback still needs roughly a 2x speedup, but the per-block cost profile suggests this gap is engineering rather than conceptual.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces UniSwap, a streaming framework for joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers appearance and vocal timbre in a single audio-visual diffusion transformer while preserving source motion, scene, and content. The authors address the lack of aligned cross-identity training pairs with a swap-and-reconstruct pipeline that creates identity-altered sources from real clips and uses the originals as targets. Starting from the LTX-2.3 backbone, they progressively adapt it through in-context pretraining, conditional streaming adaptation with a block-causal mask, and self-forcing DMD with three-step sampling. Feature-RoPE Decomposition and adaptive sink blocks are introduced for long-form streaming inference. Experiments report a Sync-C advantage over cascaded baselines, competitive identity preservation, and 13.6 FPS on one H100, while acknowledging that this is below the 25 FPS playback rate and therefore streaming rather than real-time.

Significance. If the empirical claims are established, UniSwap would be a notable engineering contribution: it demonstrates a scalable way to construct aligned supervision for joint identity replacement and shows that a bidirectional audio-visual diffusion transformer can be converted into an efficient streaming generator with bounded cache and few-step sampling. The systematic ablations of the three training stages and of Feature-RoPE components are valuable. The central claim, however, is that joint generation yields superior audio-visual synchronization compared with cascades; this claim is not yet convincingly supported because the cascade comparison is not a like-for-like cascade, and the long-form claim is extrapolated from one-minute evaluations.

major comments (3)
  1. [Sec. 4.1, Table 1] The comparison protocol undermines the paper's central claim. Every video-replacement baseline is paired with the same Seed-VC audio backend, but the video model never conditions on the Seed-VC output; Sync-C and Sync-D are then measured between the baseline video, whose lip movements were generated for the original source speech, and the converted audio track. This charges the cascade for a mismatch that a natural cascade with a lip-sync re-anchoring or dubbing stage would correct, and it may account for a substantial portion of the reported advantage (Sync-C 3.633 vs. 3.289). The authors should compare against a re-anchored cascade, for example a lip-sync module conditioned on the converted waveform, before attributing the synchronization gain to joint modeling.
  2. [Sec. 4.2, Fig. 1, Table 2] The abstract and Fig. 1 claim stable hour-scale long-form generation, yet the long-form benchmark consists only of one-minute videos, with metrics computed independently on three 20-second segments. No experiment measures behavior beyond one minute, and the training clips are only about 9.6 seconds. The hour-scale claim is therefore unsupported. Please either extend the benchmark to a much longer duration (ideally hour-scale, or at least an order of magnitude beyond the training length) or remove the hour-scale claim from the abstract and Fig. 1.
  3. [Sec. 3.1] The swap-and-reconstruct pipeline is the sole source of training supervision, but its two components are not quantitatively validated. The pose-proxy video is assumed to preserve source motion, timing, background, and scene exactly, and Seed-VC is assumed to preserve speech content and temporal alignment; no metric is reported for either assumption. If the visual proxy or the voice conversion introduces misalignment or artifacts, the model is trained to reconstruct a target from corrupted sources, which directly weakens the identity-transfer and synchronization claims. The authors should report quantitative fidelity checks for the synthesized sources, such as pose alignment error, temporal sync between the converted audio and the source video, and content preservation of the converted speech.
minor comments (4)
  1. [Fig. 1, Sec. 4.3] The 'Real Time' label in Fig. 1 is misleading given that 13.6 FPS is below the 25 FPS playback rate; the text correctly states that the implementation supports streaming generation but not real-time playback. The figure label should be qualified accordingly.
  2. [Sec. 2.3 and Sec. 4.1] Given that OmniForcing is described as a streaming audio-visual diffusion model with block-causal alignment and self-forcing distillation, the abstract's 'first framework for streaming joint audio-visual identity replacement' should either explicitly distinguish UniSwap from OmniForcing in terms of identity replacement or justify why OmniForcing is not a valid baseline. As written, the novelty claim is not fully contextualized.
  3. [Table 3] The efficiency table mixes per-block timings for UniSwap with whole-clip timings for baselines. While the footnote explains this, the FPS row is the only directly comparable measure; presenting per-step times in the same table invites apples-to-oranges comparisons. Consider reporting only wall-clock FPS and end-to-end latency in the main table.
  4. [Supplementary Sec. 8] The user study inherits the same cascade-protocol issue as Table 1, since the baselines are again paired with Seed-VC without re-anchoring. In addition, no significance testing is reported for the difference between UniSwap and SCAIL-2+Seed-VC on lip synchronization (4.11 vs. 3.67).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: training is supervised reconstruction with external benchmarks, and the Seed-VC backend is a controlled normalization, not a fitted prediction.

full rationale

UniSwap is an empirical systems paper rather than a derivation from first principles, and no claimed prediction or result reduces to its own inputs by construction. The swap-and-reconstruct pipeline synthesizes training pairs by removing identity from a real clip and using the original clip as the reconstruction target; this is ordinary supervised self-reconstruction, and the model is evaluated on held-out speakers and reference identities, so the identity-transfer claim is not definitionally forced. The headline audio-visual synchronization claim rests on SyncNet scores, an external pretrained metric that is never used as a training loss; Sync-C, Sync-D, DINO-S, and the voice-quality scores are measured outputs, not fitted parameters renamed as predictions. Using Seed-VC both to synthesize the training source audio and as the common backend for cascade baselines is a deliberate experimental normalization: it does not make UniSwap's sync advantage true by construction, because UniSwap generates its own video and audio jointly while the video-replacement baselines are not audio-conditioned and are simply scored against the Seed-VC waveform. The cited OmniForcing work is external prior work and is not invoked as a load-bearing uniqueness theorem, and no author-overlapping citation is used to forbid alternative designs. The paper's stated limitations (single-speaker videos, no expression control) and its explicit admission that 13.6 FPS is below the 25 FPS playback rate are consistency checks rather than circular steps. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The system is built entirely on existing models and tooling; no new physical entities or conservation laws are postulated. The only invented elements are algorithmic (Feature-RoPE Decomposition, Decoupled Streaming Conditioning Mask, swap-and-reconstruct), which are methods rather than entities. The central claim rests on the fidelity of the swap-and-reconstruct data, the validity of the streaming adapters and distillation, and the extrapolation from 1-minute benchmarks to the claimed hour-scale stability.

free parameters (6)
  • CFG guidance scale for video teacher score (gamma_video) = 3.0
    Hand-chosen in Eq. (5) to balance conditional and unconditional teacher predictions in the DMD loss; not fitted to the benchmark.
  • CFG guidance scale for audio teacher score (gamma_audio) = 5.0
    Hand-chosen in Eq. (5) for the audio stream; not fitted to the benchmark.
  • Denoising noise levels for 3-step student = [0.999, 0.757, 0.522]
    Fixed schedule used in the self-forcing DMD rollout (Sec. 3.4.1); chosen to match DMD practice.
  • Target block size K = 3 video latent frames, with an initial 4-frame block
    Sets the autoregressive block granularity in Sec. 3.3; directly affects exposure bias and cache size.
  • Rolling cache window size W = 4 slots (1 sink, 2 rolling, 1 current)
    Feature-RoPE Decomposition window in Sec. 3.4.2; bounds positional indices and memory.
  • Reference audio crop length = 30% of target audio duration
    Random segment used as the reference voice in Sec. 3.1; affects identity conditioning strength.
assumptions (5)
  • domain assumption The frozen LTX-2.3 backbone provides a high-quality bidirectional audio-video diffusion transformer with native cross-modal attention and a causal video VAE.
    The entire method builds on LTX-2.3 (Sec. 3) without questioning its temporal alignment or generation quality limitations.
  • domain assumption Seed-VC preserves speech content and prosody while changing timbre, and pose-proxy compositing preserves source motion and scene exactly.
    The swap-and-reconstruct pipeline in Sec. 3.1 relies on these properties to create aligned training pairs; if either conversion corrupts timing or content, the reconstruction target is misaligned.
  • domain assumption DMD with separate teacher/generator/critic LoRAs on a shared frozen backbone gives the same distillation signal as full separate models.
    Efficient Multi-LoRA Switching in Sec. 3.4 assumes the LoRA adapters can decouple the three roles without interfering; the paper reports memory usage but no distribution-level comparison to full separate models.
  • ad hoc to paper A 1-minute benchmark extrapolates to hour-scale generation stability.
    The paper claims hour-scale generation (Fig. 1) but only evaluates 20-60 second segments (Table 2), so the stability guarantee is assumed.
  • standard math SyncNet, Q-Align, DINO-S, DNSMOS, and speaker-encoder cosine similarity are valid proxies for audio-visual sync, quality, identity, and voice similarity.
    Standard metrics from prior work, but their limitations (e.g., SyncNet can be fooled by static mouth patterns) are not discussed in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos." pith.science (2026). https://pith.science/paper/QZBPYOQ6

@misc{pith2026260811752,
  author       = {Pith},
  title        = {Pith review of: UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZBPYOQ6}},
  note         = {Machine review of arXiv:2608.11752}
}
read the original abstract

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.

Figures

Figures reproduced from arXiv: 2608.11752 by the authors.

Figure 1
Figure 1. UniSwap performs streaming joint audio-visual identity replacement. Given a source talking video and a reference identity (an image and voice clip), UniSwap jointly transfers appearance and vocal timbre while preserving the source motion, background, and linguistic content. Blockwise autoregressive generation with 3-step sampling runs at 13.6 FPS on one NVIDIA H100 and supports stable hour-scale long-form generation… view at source ↗
Figure 2
Figure 2. Overview of UniSwap. The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which re￾duces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment. 3.1. Swap-and-Reconstruct… view at source ↗
Figure 3
Figure 3. Swap-and-reconstruct paired data synthesis. Every real talking video serves as its own reconstruction target. ① The real clip provides the target video Vt and audio At. ② The visual identity is swapped out by replacing the person with a pose proxy composited onto the masked background plate, the vocal timbre is randomized towards a sampled speaker, and the reference identity is formed by a portrait frame Ir and a ra… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on the short-video benchmark. Given the reference image/audio and the source video/audio (top), video replacement methods (upper rows) transfer the appearance but keep the original voice, and voice conversion methods (middle rows) modify only the…
Figure 5
Figure 5. Figure 5: Qualitative comparison on the long-video benchmark. Frames are sampled every 10 seconds from 1-minute generations. The baselines exhibit identity drift and visual artifacts as generation progresses (highlighted in red), while UniSwap preserves the reference identity th…
Figure 6
Figure 6. Figure 6: Qualitative ablation on Feature-RoPE Decomposition. Frames are sampled every 10 seconds from one-minute generations. Removing any component leads to visible identity drift and artifacts in later segments, whereas the full model remains more consistent throughout the se…
Figure 7
Figure 7. Figure 7: Additional qualitative results on short videos (Part I). For each example, the left column provides the reference image and reference voice, while the right column shows sampled frames and the waveform of the source video/audio followed by the UniSwap output. Red wavef…
Figure 8
Figure 8. Figure 8: Additional qualitative results on short videos (Part II). The examples follow the layout of [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Additional qualitative results on one-minute videos. Frames are sampled every 10 seconds from 0 to 60 seconds. Each row group shows the reference image and voice, the source video/audio (red waveform), and the UniSwap output (blue waveform). Across all three sequences,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 31 canonical work pages

  1. [1]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 8

    work page 2021

  2. [2]

    Diffusion forcing: Next-token prediction meets full-sequence diffu- sion

    Boyuan Chen, Diego Mart ´ı Mons ´o, Yilun Du, Max Sim- chowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffu- sion. In NeurIPS, 2024. 3

    work page 2024

  3. [3]

    Simswap: An efficient framework for high fidelity face swapping

    Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. Simswap: An efficient framework for high fidelity face swapping. In ACM MM, 2020. 3

    work page 2020

  4. [4]

    Wan-animate: Unified character animation and replacement with holistic replica- tion

    Gang Cheng, Xin Gao, Li Hu, et al. Wan-animate: Unified character animation and replacement with holistic replica- tion. In arXiv preprint arXiv:2509.14055, 2025. 2, 3, 7, 8, 9

    arXiv 2025

  5. [5]

    Out of time: Auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: Auto- mated lip sync in the wild. In ACCV Workshop, 2016. 8

    work page 2016

  6. [6]

    High fidelity neural audio compression

    Alexandre D ´efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. In Transactions on Machine Learning Research, 2023. 2

    work page 2023

  7. [7]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic to- kens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Haizhou Zheng, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic to- kens. In arXiv preprint arXiv:2407.05407, 2024. 2, 3, 7, 8

    arXiv 2024

  8. [8]

    Freeman, and Michael Rubinstein

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech sepa- ration. In ACM TOG, 2018. 6

    work page 2018

Show all 49 references
  1. [9]

    Video diffusion transform- ers are in-context learners

    Zhengcong Fei, Di Qiu, Changqian Yu, Debang Li, Mingyuan Fan, and Xiang Wen. Video diffusion transform- ers are in-context learners. arXiv preprint arXiv:2412.10783,

    arXiv

  2. [10]

    Infoswap: Information bottleneck disentanglement for identity swapping

    Gege Gao, Huaibo Huang, Chaoyou Fu, Zhaoyang Li, and Ran He. Infoswap: Information bottleneck disentanglement for identity swapping. In CVPR, 2021. 3

    2021

  3. [11]

    Ltx- 2: Efficient joint audio-visual foundation model

    Yoav HaCohen, Benny Brazowski, Nisan Chiprut, et al. Ltx- 2: Efficient joint audio-visual foundation model. In arXiv preprint arXiv:2601.03233, 2026. 2, 3

    2026 arXiv

  4. [12]

    FullDiT2: Efficient in-context con- ditioning for video diffusion transformers

    Xuanhua He, Quande Liu, Zixuan Ye, Weicai Ye, Qi- ulin Wang, Xintao Wang, Qifeng Chen, Pengfei Wan, Di Zhang, and Kun Gai. FullDiT2: Efficient in-context con- ditioning for video diffusion transformers. arXiv preprint arXiv:2506.04213, 2025. 3

    2025 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 6

    2022

  6. [14]

    Hunyuancus- tom: A multimodal-driven architecture for customized video generation

    Teng Hu, Zhentao Yu, Zhengguang Zhou, et al. Hunyuancus- tom: A multimodal-driven architecture for customized video generation. In arXiv preprint arXiv:2505.04512, 2025. 2, 3, 7, 8, 9

    2025 arXiv

  7. [15]

    In-context lora for diffusion transformers

    Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Tong Feng, Gan Liu, and Jun- wei Yang. In-context lora for diffusion transformers. In arXiv preprint arXiv:2410.23775, 2024. 3, 4

    2024 arXiv

  8. [16]

    Self forcing: Bridging the train-test gap in autoregressive video diffusion

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. In arXiv preprint arXiv:2506.08009, 2025. 3

    2025 arXiv

  9. [17]

    Ref-vc: Robust, expressive and fast zero-shot voice con- version with diffusion transformers

    Yuepeng Jiang, Ziqian Ning, Shuai Wang, Chengjia Wang, Mengxiao Bi, Pengcheng Zhu, Zhonghua Fu, and Lei Xie. Ref-vc: Robust, expressive and fast zero-shot voice con- version with diffusion transformers. In arXiv preprint arXiv:2508.04996, 2025. 2, 3

    2025 arXiv

  10. [18]

    Vace: All-in-one video creation and editing

    Zeyinzi Jiang, Zhen Zhu, Chaojie Qi, Xin Jia, et al. Vace: All-in-one video creation and editing. In arXiv preprint arXiv:2503.07598, 2025. 2, 3, 7, 8, 9

    2025 arXiv

  11. [19]

    Faceshifter: Towards high fidelity and occlusion aware face swapping

    Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Faceshifter: Towards high fidelity and occlusion aware face swapping. In CVPR, 2020. 3

    2020

  12. [20]

    Rolling forcing: Autoregressive long video diffusion in real time

    Kunhao Liu, Wenbo Hu, Jiale Xu, Ying Shan, and Shijian Lu. Rolling forcing: Autoregressive long video diffusion in real time. In ICLR, 2026. 3

    2026

  13. [21]

    JavisDiT: Joint audio-video diffusion trans- former with hierarchical spatio-temporal prior synchroniza- tion

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Jiebo Luo, Ziwei Liu, Hao Fei, and Tat-Seng Chua. JavisDiT: Joint audio-video diffusion trans- former with hierarchical spatio-temporal prior synchroniza- tion. In ICLR, 2026. 3

    2026

  14. [22]

    Zero-shot voice conversion with diffusion transformers

    Songting Liu. Zero-shot voice conversion with diffusion transformers. In arXiv preprint arXiv:2411.09943, 2024. 2, 3, 4, 7, 8

    2024 arXiv

  15. [23]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 3

    2023

  16. [24]

    Open- voice: Versatile instant voice cloning

    Zengyi Qin, Wenliang Zhao, Xumin Yu, and Xin Sun. Open- voice: Versatile instant voice cloning. In arXiv preprint arXiv:2312.01479, 2023. 7, 8

    2023 arXiv

  17. [25]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, et al. Sam 2: Segment anything in images and videos. In ICLR, 2025. 4

    2025

  18. [26]

    Chandan K. A. Reddy, Vishak Gopal, and Ross Cutler. Dns- mos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP, 2022. 8

    2022

  19. [27]

    MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation. In CVPR, 2023. 3

    2023

  20. [28]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023. 3

    2023

  21. [29]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. In Neurocomputing, 2024. 3, 5

    2024

  22. [30]

    Omniforcing: Unleashing real-time joint audio- visual generation

    Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, and Nan Duan. Omniforcing: Unleashing real-time joint audio- visual generation. arXiv preprint arXiv:2603.11647, 2026. 3

    2026

  23. [31]

    Wan: Open and advanced large-scale video gen- erative models

    Wan Team. Wan: Open and advanced large-scale video gen- erative models. In arXiv preprint arXiv:2503.20314, 2025. 2

    2025 arXiv

  24. [32]

    Generalized end-to-end loss for speaker verification

    Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In ICASSP, 2018. 8

    2018

  25. [33]

    Bovik, Hamid R

    Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibil- ity to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. 8

    2004

  26. [34]

    Williams and David Zipser

    Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. In Neural Computation, pages 270–280, 1989. 5

    1989

  27. [35]

    Q-Align: Teaching LMMs for vi- sual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guang- tao Zhai, and Weisi Lin. Q-Align: Teaching LMMs for vi- sual scoring via discrete text-defined levels. In Proceedings ...

    2024

  28. [36]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024. 6

    2024

  29. [37]

    Vit- pose: Simple vision transformer baselines for human pose estimation

    Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vit- pose: Simple vision transformer baselines for human pose estimation. In NeurIPS, 2022. 4

    2022

  30. [38]

    Mocha: End-to-end video character re- placement without structural guidance

    Zhengbo Xu, Jie Ma, Ziheng Wang, Zhan Peng, Jun Liang, and Jing Li. Mocha: End-to-end video character re- placement without structural guidance. In arXiv preprint arXiv:2601.08587, 2026. 2, 3, 7, 8, 9

    2026

  31. [39]

    Scail-2: Unifying controlled character animation with end-to-end in-context conditioning

    Wenhao Yan, Fengjia Guo, Zhuoyi Yang, and Jie Tang. Scail-2: Unifying controlled character animation with end-to-end in-context conditioning. In arXiv preprint arXiv:2606.10804, 2026. 7, 8, 9

    2026 arXiv

  32. [40]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Chen, Xiaotao Dai, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In arXiv preprint arXiv:2408.06072,

    arXiv

  33. [41]

    Tianwei Yin, Micha ¨el Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fr´edo Durand, and William T. Freeman. Im- proved distribution matching distillation for fast image syn- thesis. In NeurIPS, 2024. 3

    2024

  34. [42]

    Freeman, and Taesung Park

    Tianwei Yin, Micha ¨el Gharbi, Richard Zhang, Eli Shecht- man, Fr´edo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024. 2, 3

    2024

  35. [43]

    Free- man, Fr´edo Durand, Eli Shechtman, and Xun Huang

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Free- man, Fr´edo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion mod- els. In CVPR, 2025. 3 UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos Supplement...

    2025

  36. [44]

    The ablated variants exhibit increasing identity drift and vi- sual artifacts in later segments, whereas the full model re- mains more consistent

    Qualitative Ablation Figure 6 complements the quantitative ablation in the main paper by visualizing the effect of each component of Feature-RoPE Decomposition over one-minute generations. The ablated variants exhibit increasing identity drift and vi- sual artifacts in later s...

  37. [45]

    The reference cache persists throughout generation, source keys and values are tem- porary, and completed target blocks are committed to the clean-history cache

    KV-Cached Streaming Inference Algorithm 1 details the blockwise inference procedure used by Stages 2 and 3. The reference cache persists throughout generation, source keys and values are tem- porary, and completed target blocks are committed to the clean-history cache. Algorit...

  38. [46]

    The study compared UniSwap with four video-replacement baselines, each paired with Seed-VC following the cascade protocol in Table 1

    User Study We conducted a blinded user study with 30 participants. The study compared UniSwap with four video-replacement baselines, each paired with Seed-VC following the cascade protocol in Table 1. Each participant evaluated anonymized outputs from all five methods on four ...

  39. [47]

    In all figures, each example con- tains a reference image and reference voice clip, a source video and its audio, and the joint audio-video output pro- duced by UniSwap

    Additional Qualitative Results We provide additional qualitative results for both short and long source videos. In all figures, each example con- tains a reference image and reference voice clip, a source video and its audio, and the joint audio-video output pro- duced by UniS...

  40. [48]

    Limitations UniSwap currently targets single-speaker talking videos; multi-speaker scenes, occlusions, and complex interactions remain challenging. Facial expressions are driven automat- ically by the audio condition rather than controlled explic- itly, so the current model do...

  41. [49]

    Deployment should require consent and provenance mechanisms, visible disclosure where ap- propriate, access controls, and compatibility with forensic detection tools

    Broader Impact Audio-video character replacement can support film- making, localization, and accessibility, but it also in- creases the risk of impersonation, non-consensual media, and misinformation. Deployment should require consent and provenance mechanisms, visible disclos...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.