REVIEW 2 cited by
The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/
Forward citations
Cited by 2 Pith papers
-
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
RLHF failures are classifiable training transitions: aggressive PPO yields localized reward hacking that row-level diagnostics catch and pre-transition features partially predict.
-
Secure & Personalized Music-to-Video Generation via CHARCHA
An automated pipeline generates personalized music videos from audio alone, using a CAPTCHA-style liveness check (CHARCHA) to collect and protect the user's facial identity.
Discussion (0). Continue with ORCID to comment.