Pith. sign in

REVIEW 3 cited by

AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.04001 v2 pith:VDCL3KMK submitted 2023-05-06 cs.CV cs.AI

Seungwoo Lee , Chaerin Kong , Donghyeon Jeon , Nojun Kwak This is my paper · ORCID
classification cs.CVcs.AI
keywords diffusionsynthesistemporalaudio-aligneddynamicsemploymodelsvideo
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in diffusion models have showcased promising results in the text-to-video (T2V) synthesis task. However, as these T2V models solely employ text as the guidance, they tend to struggle in modeling detailed temporal dynamics. In this paper, we introduce a novel T2V framework that additionally employ audio signals to control the temporal dynamics, empowering an off-the-shelf T2I diffusion to generate audio-aligned videos. We propose audio-based regional editing and signal smoothing to strike a good balance between the two contradicting desiderata of video synthesis, i.e., temporal flexibility and coherence. We empirically demonstrate the effectiveness of our method through experiments, and further present practical applications for contents creation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

    cs.CV 2024-12 conditional novelty 7.0 of 10

    AV-Link unifies video-to-audio and audio-to-video generation by aligning frozen diffusion-model activations with temporally matched rotary position embeddings in a shared Fusion Block.

  2. LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LeviTor controls 3D object trajectories in generated videos by feeding K-means clustered mask points with estimated depth into a video diffusion model.

  3. Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generation

    cs.MM 2024-12 conditional novelty 5.0 of 10

    An automatic music-visualizer pipeline that uses genre-guided image generation and audio-energy-controlled frame interpolation beats linear interpolation on a new synchrony metric.

Pith tools