Pith. sign in

REVIEW 2 cited by

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.04509 v1 pith:6MDF2NZI submitted 2023-09-08 cs.SD cs.CVcs.GReess.AS

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

Yujin Jeong , Wonjeong Ryoo , Seunghyun Lee , Dabin Seo , Wonmin Byeon , Sangpil Kim , Jinkyu Kim This is my paper
classification cs.SD cs.CVcs.GReess.AS
keywords audiotposgenerationvideoaudio-to-videodiffusionmagnitudemodel
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

    cs.LG 2026-06 conditional novelty 6.0

    RLHF failures are classifiable training transitions: aggressive PPO yields localized reward hacking that row-level diagnostics catch and pre-transition features partially predict.

  2. When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

    cs.LG 2026-06 unverdicted novelty 5.0

    An empirical study of RLHF pipelines classifies failure modes such as reward hacking by analyzing directions of change in learned reward and judge scores across training checkpoints and shows they can be localized and...