Pith. sign in

REVIEW 3 cited by

Temporally Aligned Audio for Video with Autoregression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.13689 v1 pith:2HQFOKEX submitted 2024-09-20 cs.CV cs.MMcs.SDeess.AS

Ilpo Viertola , Vladimir Iashin , Esa Rahtu This is my paper · ORCID
classification cs.CVcs.MMcs.SDeess.AS
keywords v-auraalignmentrelevancesamplestemporalvisualvisualsoundaligned
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. V-AURA uses a high-framerate visual feature extractor and a cross-modal audio-visual feature fusion strategy to capture fine-grained visual motion events and ensure precise temporal alignment. Additionally, we propose VisualSound, a benchmark dataset with high audio-visual relevance. VisualSound is based on VGGSound, a video dataset consisting of in-the-wild samples extracted from YouTube. During the curation, we remove samples where auditory events are not aligned with the visual ones. V-AURA outperforms current state-of-the-art models in temporal alignment and semantic relevance while maintaining comparable audio quality. Code, samples, VisualSound and models are available at https://v-aura.notion.site

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper

    cs.CV 2025-09 reject novelty 5.0 of 10

    A GPT-2 mapper over dual visual encoders claims 16% training cost and better alignment, but test-time use of true class labels makes the comparison invalid for V2A.

  2. Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fine-tuning a video encoder with self-distillation on cropped and shifted clips makes video-to-audio generation robust to partially visible Foley targets.

  3. Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks

    cs.SD 2025-05 reject novelty 4.0 of 10

    The paper proposes an open-source 10-hour video-to-piano benchmark with four-level Chain-of-Perform annotations, but supplies only preliminary, incomplete baseline results.

Pith tools