Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VinTAGe's claim: to hear a scene completely — on-screen foley and off-screen ambience — audio generation must be grounded in both the video and a text prompt.

desk verdict Useful joint video-text audio model and a transparent benchmark, but the 'necessity' claim is weaker than the paper's own Table 5 suggests. read the letter →

arxiv 2412.10768 v1 pith:HTF4WOLN submitted 2024-12-14 cs.CV cs.MMcs.SDeess.AS

Saksham Singh Kushwaha , Yapeng Tian This is my paper · ORCID
classification cs.CVcs.MMcs.SDeess.AS
keywords holisticaudiogenerationjointtext-videoconditioningvideo-to-audiotext-to-audioflowmatchingmodalitybiasaudio-visualsynchronizationVinTAGe-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses a post-production problem: a silent video usually needs both on-screen sounds (a dog barking in frame, timed to its movements) and off-screen sounds (a vacuum cleaner humming off camera), and existing tools only produce one of the two. Text-to-audio models generate the complete soundscape but cannot lock it to what is shown; video-to-audio models sync to the image but drop what is not visible. VinTAGe claims that conditioning on video and text together, with the two modalities kept in balance, lets a single model generate both layers in one audio track that is synchronized with the video and consistent with the caption. If the claim holds, a sound designer could go from silent footage plus a written description straight to a finished mix, and the new VinTAGe-Bench (636 video-text-audio pairs of mixed on-screen and off-screen sounds) makes the task measurable for the first time. The paper further reports the best results among the compared methods on the standard VGGSound video-to-audio benchmark.

What carries the argument

The engine is a flow-matching transformer: following flow matching, the model learns a velocity field $v_\theta(x_t, t, c_v, c_t)$ that linearly interpolates Gaussian noise into a compressed audio latent, and at inference an ODE solver integrates the field backwards. The backbone is SiT, the Scalable Interpolant Transformer, extended into a Joint VT-SiT whose blocks interleave self-attention with two cross-attention layers, one over text tokens and one over video tokens, fused by a learned per-layer weight $\omega_l$. The conditioning signal comes from the Visual-Text Encoder, which cross-attends CLIP frame embeddings (concatenated with sinusoidal encodings of mean optical-flow energy and frame indices) to FLAN-T5 text embeddings through zero-initialized tanh gates, and pools both modalities into a global vector applied by adaptive layer normalization. The decisive training mechanism is teacher-student alignment: batches alternate joint, text-only, and video-only inputs, and on the single-modality entries the joint model must reproduce the velocity predicted by frozen uni-modal teachers through the losses $L_t$ and $L_v$, which stops either modality from dominating. At inference, two classifier-free guidance scales $s_{\text{vis}}$ and $s_{\text{txt}}$ let the operator fade the output between visual and textual priorities.

What would settle it

Take uncurated real-world clips with natural multi-source soundscapes — several off-screen sounds at once, overlapping frequency bands, and multiple moving on-screen objects, the regime the paper's own limitations section flags — and compare VinTAGe against a strong text-to-audio model fed a detailed caption of the video. If the text-only model matches the joint model on semantic completeness and the temporal-synchronization advantage disappears on these clips, the claim that joint video-text conditioning is necessary for holistic audio would be falsified; running the same test on the benchmark's excluded hard cases (similar-frequency or busy mixtures) would show whether the filtering drove the result.

Watch

Extended reading notes

Core claim

The central claim is that joint text and visual interaction is necessary for holistic audio generation, and that a flow-based transformer can carry it out without drifting into one modality. The paper argues that each single-condition family is structurally incomplete: text-to-audio has no access to timing, and video-to-audio has no access to what lies off-screen, so neither can produce the full sound of a scene. Prior joint models assume the text and video describe the same content, which breeds a visual bias that suppresses off-screen sounds. VinTAGe counters this with three coordinated moves: a Visual-Text Encoder whose gated cross-attention lets video and text inform each other without forcing them to agree; a Joint VT-SiT transformer in which each block learns a weight $\omega_l$ balancing the two modalities' cross-attention; and a teacher-student scheme in which pretrained text-only and video-only generators supply velocity targets for single-modality inputs, so the joint model learns to match both teachers. On VinTAGe-Bench the model posts the best generation quality (FAD 3.05, FID 16.43) and the best mean audio-text/audio-visual alignment (16.06), and it is the joint model whose on-screen and off-screen concept accuracies stay closest together (57.7% and 43.6%), where the strongest joint rival splits 64.9% against 21.7%. The same model posts the best FID (6.65) and MKL (4.12) on VGGSound among the compared methods.

Load-bearing premise

The load-bearing premise is that VinTAGe-Bench's recipes — one on-screen sound mixed with one randomly chosen off-screen sound, with hard cases filtered out — stand in for real holistic audio, so the paper's conclusion that joint conditioning is necessary inherits everything from that curated two-source mix; if genuine off-screen soundscapes are messier than the benchmark, the necessity claim is not established.

Editorial extensions

If this is right

  • Post-production sound for a silent clip reduces to a single pass: feed the video and a written description, and receive one mixed track containing both foley and off-screen ambience, with no manual mixing of separate stems.
  • Joint conditioning beats both uni-modal families: on VinTAGe-Bench, VinTAGe's FAD (3.05) and FID (16.43) are the best in the table, and its mean audio-text/audio-visual alignment (16.06) is the highest.
  • Modality bias is measurable and curable: joint baselines that assume aligned text and video (ReWaS, FoleyCrafter) generate on-screen sounds well but drop off-screen sounds, while teacher-guided training keeps VinTAGe's two accuracies close (57.7% versus 43.6%).
  • The two guidance scales make the text-video trade-off controllable at inference: raising $s_{\text{vis}}$ shifts the output toward visual-only sounds, raising $s_{\text{txt}}$ adds off-screen content, and balanced values produce both.
  • Because the same model is evaluated directly on VGGSound, the architecture doubles as a video-to-audio system, posting the best FID (6.65) and MKL (4.12) among the compared methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The necessity claim is only as strong as the benchmark: VinTAGe-Bench mixes exactly one on-screen source with one off-screen source drawn from VGGSound and deletes hard cases (spectrally overlapping sounds, busy on-screen audio, poorly visible objects), so a real-world soundscape with several overlapping off-screen sources is an untested regime in which the conclusion could weaken.
  • The paper's own limitations section (Appendix F) concedes that mean optical-flow magnitude is a coarse temporal guide that may fail when several moving objects each carry their own sound; an object-wise temporal-energy variant is the natural next test of whether fine-grained synchronization survives in complex scenes.
  • The two guidance scales suggest an interactive application the authors do not build: a two-knob foley-versus-ambience fader that a sound editor could adjust per shot, exploiting the classifier-free guidance the model already learns.
  • Because on-screen temporal alignment is measured only by subjective listening (existing metrics assume single-source audio), scoring the model's own output with a neural audio separator would give an objective, scalable test of the synchronization claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. VinTAGe addresses holistic audio generation: given a silent video and a text prompt, it aims to synthesize both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with the text. The proposed model comprises a Visual-Text Encoder (CLIP visual features, optical-flow energy, frame indices, and FLAN-T5 text embeddings with gated cross-attention), a flow-matching SiT transformer with per-layer text/video cross-attention weights, and teacher-student alignment losses from pretrained text-only and video-only audio generators. The paper also introduces VinTAGe-Bench, a 636-pair benchmark built from 212 VGGSound test videos by mixing each original audio track with one randomly selected offscreen sound and filtering out mixtures that are hard to distinguish. Experiments compare against V2A, T2A, and joint text-video baselines on VinTAGe-Bench and on VGGSound, reporting state-of-the-art FAD/FID numbers and a small subjective study. The central claim is that joint text and visual interaction is necessary for holistic audio generation.

Significance. The paper is useful in several respects. It formulates a timely task, contributes a new benchmark (VinTAGe-Bench) that the community can reuse, and proposes a sensible architecture with teacher-guided modality balancing; the comparison set is broad, and the limitations section is candid about the difficulty of temporal alignment metrics for mixed audio. The paper also ships a demo and promises code and pretrained models. However, the central conceptual claim—that joint conditioning is necessary for holistic audio generation—is not supported by the evidence. The benchmark construction makes single-modality models fail by design on one of the two subtasks, and the paper's own controlled comparison (Table 5) shows text-only conditioning achieving higher concept accuracy than the joint model. The VGGSound state-of-the-art claim also needs qualification because VinTAGe receives text captions that the pure V2A baselines do not. The contribution would remain valuable if the claims were reframed around what is actually demonstrated: joint conditioning improves distribution-level quality and alignment on curated two-source mixtures and mitigates modality bias.

major comments (4)
  1. [Abstract, §4, Table 5] The claim that 'joint text and visual interaction is necessary for holistic audio generation' is not established by the experiments. VinTAGe-Bench is constructed so that the offscreen sound is invisible in the video and appears only in the text caption, while mixtures with similar frequencies or 'busy' onscreen sounds are removed (§4, Dataset paragraph). Single-modality models are therefore guaranteed to fail on one of the two subtasks by construction, and the ambiguous cases where joint disambiguation would be most informative are filtered out. More importantly, the paper's own controlled comparison in Table 5 shows that text-only conditioning (svis=0.0, stxt=5.0) achieves higher mean concept accuracy than the joint model (54.32 vs. 50.66), and the visual-only teacher is competitive in mean alignment. The data support a weaker conclusion: joint conditioning improves FID/FAD and mean alignment on this curated benchmark. The necessity claim should be removed or substantially qualified.
  2. [§4, Table 2] The VGGSound state-of-the-art claim is not supported as stated. VinTAGe is conditioned on Auto-ACD text captions, which include the sound class label, while SpecVQGAN, Im2Wav, and Diff-Foley receive no text. Since VGGSound audio is largely determined by the class label, giving the model this text is an information advantage unrelated to joint video-text modeling. The authors update only the text+video baselines (FoleyCrafter and ReWaS) with Auto-ACD captions; the V2A baselines are not given an analogous captioning channel. The claim should be limited to joint text-video models with equalized inputs, or the comparison should include text-only and video-text variants of all baselines.
  3. [§4, Subjective Evaluation; Appendix D] The subjective evaluation is too small and too poorly characterized to support the temporal-alignment and faithfulness claims. Ten listeners rated 35 samples on VinTAGe-Bench, with no confidence intervals, significance tests, or inter-rater agreement reported. The MOS gaps (for example, MOS-T 3.36 vs. 2.96 for FoleyCrafter) may be meaningful, but with this sample size and no statistics they are not load-bearing. This matters because the paper explicitly declines to use objective temporal metrics (Appendix F) and relies on subjective scores for the temporal-synchronization claim. The authors should report per-item variance, confidence intervals, and a statistical test, or weaken the corresponding conclusions.
  4. [§3.3, Appendix B.3] The classifier-free guidance scales svis and stxt are tuned empirically to 2.5 and used for all main results, but no validation protocol is described. Appendix B.3 sweeps one scale while fixing the other, yet the paper does not state whether these sweeps were performed on the test benchmark or a held-out set. With only 636 test pairs and no error bars, the reported improvements could reflect guidance-scale selection rather than the model's joint-conditioning design. The authors should describe the scale-selection procedure and, ideally, report results across a range of scales.
minor comments (5)
  1. [§3.3, Eq. (7)] The sentence introducing Eq. (7) says 'stxt and svis represent the weights of visual and text guidance,' but stxt is the text weight and svis is the visual weight; the assignment is swapped.
  2. [§4, Table 1] The caption states 'Top-2 results are highlighted,' but in the typeset table only one row appears highlighted; please check the formatting so the intended highlighting is visible.
  3. [Appendix C] The phrase 'for each video, we selected two off-screen categories and one no off-screen scenario from the offscreen categories' is confusing; clarify that the no-offscreen scenario is a caption-only condition without a mixed offscreen audio track.
  4. [§3.2, Gated Cross-Attention] The dimensions of cv and ct are not fully specified: ct has length L (token count), cv has length N (frame count), and the final MLP projection size is given only in Appendix A.1. Stating these dimensions in the main text would improve readability.
  5. [References] References [9] and [10] appear to be the same Lumina-T2X paper, and the author list in [10] contains an empty entry ('Chris Liu, ,'); these should be merged and corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

The claim that joint text and video conditioning is 'necessary' is built into the construction of VinTAGe-Bench, though the VGGSound results are independent.

  1. self definitional [Abstract; Section 4, VinTAGe-Bench construction paragraph]
    "Offscreen sounds were randomly selected from the VGGSound test set and mixed with the original audio, adjusting gains for balanced pressure levels [11]. ... Text captions were generated for each scenario, with the order of on-screen and off-screen sound descriptions randomly shuffled to prevent bias. ... Our comprehensive experiments on VinTAGe-Bench demonstrate that joint text and visual interaction is necessary for holistic audio generation."

    VinTAGe-Bench defines holistic audio as a mixture of an onscreen sound and a randomly added offscreen sound. By construction, the offscreen clip is chosen from VGGSound test audio and mixed into the track, so it is not present in the video; its identity is available only through the caption, which names both sounds. The onscreen sound's temporal envelope, meanwhile, is available only through the video. Thus a video-only model cannot produce the offscreen component and a text-only model cannot lock the onscreen timing: the failure of single-modality models on this benchmark is guaranteed by dataset assembly rather than discovered by experiment.

full rationale

VinTAGe is an empirical system, and most of its engineering machinery — flow matching, the VT-Encoder, teacher-guided losses — is not circular: the VGGSound test set is an external benchmark, and the reported FAD/FID/MKL and MOS improvements there are independent evidence. The construction-bound step is the paper's headline conclusion that joint conditioning is 'necessary' for holistic audio generation. That conclusion is inferred from VinTAGe-Bench, which is built by taking VGGSound test videos, mixing each with one randomly chosen offscreen VGGSound test audio, and writing captions that name both sounds. The offscreen audio is therefore absent from the video by construction and present only in the text, while precise onscreen timing lives only in the video, so single-modality baselines fail on the corresponding sub-task for that reason, not because the experiments revealed an emergent necessity. The claim is further weakened by the paper's own Table 5, where text-only VinTAGe achieves higher Mean-Acc (54.32) than joint VinTAGe (50.66), so the 'necessary' claim is not even supported by the paper's controlled comparison. Because the VGGSound SOTA result and the architecture ablations rest on external or internally consistent evidence, the circularity is partial rather than total.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on a small number of fitted constants; it is an empirical ML system. The only explicit hand-tuned values are the two classifier-free guidance scales and the teacher-loss weights. The benchmark's construction is the main domain assumption, and the flow matching and frozen encoders are standard background.

free parameters (2)
  • Classifier-free guidance scales svis and stxt = 2.5, 2.5
    Chosen empirically at inference (Sec. 3.3) with no reported validation split; they may have been tuned on the evaluation benchmarks themselves.
  • Teacher alignment loss weights lambda_v, lambda_t = 1, 1
    Set equal in the appendix (A.1); no sensitivity analysis is reported.
assumptions (4)
  • standard math Flow matching linearly interpolates between noise and data with a constant velocity field
    Adopted from Lipman et al. and SiT (Sec. 3.1); standard in the literature.
  • domain assumption CLIP visual features and FLAN-T5 text features provide sufficiently aligned semantic representations of video and text
    Used throughout the VT-Encoder (Sec. 3.2) and for Wav2CLIP evaluation; standard practice but not verified in this paper.
  • domain assumption Pretrained T2A and V2A teacher models provide reliable velocity targets for student distillation
    Teacher-student alignment losses (Sec. 3.3) assume the teachers' velocities are better targets than raw noise-to-data interpolation for single-modality conditions.
  • ad hoc to paper The VinTAGe-Bench mixture of one onscreen audio and one randomly selected offscreen audio, after filtering hard cases, is representative of holistic audio scenes
    The benchmark is constructed by the authors (Sec. 4) and the filtering criteria are not justified by external evidence; this is the paper's own premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation." pith.science (2026). https://pith.science/paper/HTF4WOLN

@misc{pith2026241210768,
  author       = {Pith},
  title        = {Pith review of: VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTF4WOLN}},
  note         = {Machine review of arXiv:2412.10768}
}
read the original abstract

Recent advances in audio generation have focused on text-to-audio (T2A) and video-to-audio (V2A) tasks. However, T2A or V2A methods cannot generate holistic sounds (onscreen and off-screen). This is because T2A cannot generate sounds aligning with onscreen objects, while V2A cannot generate semantically complete (offscreen sounds missing). In this work, we address the task of holistic audio generation: given a video and a text prompt, we aim to generate both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with text and video. Previous approaches for joint text and video-to-audio generation often suffer from modality bias, favoring one modality over the other. To overcome this limitation, we introduce VinTAGe, a flow-based transformer model that jointly considers text and video to guide audio generation. Our framework comprises two key components: a Visual-Text Encoder and a Joint VT-SiT model. To reduce modality bias and improve generation quality, we employ pretrained uni-modal text-to-audio and video-to-audio generation models for additional guidance. Due to the lack of appropriate benchmarks, we also introduce VinTAGe-Bench, a dataset of 636 video-text-audio pairs containing both onscreen and offscreen sounds. Our comprehensive experiments on VinTAGe-Bench demonstrate that joint text and visual interaction is necessary for holistic audio generation. Furthermore, VinTAGe achieves state-of-the-art results on the VGGSound benchmark. Our source code and pre-trained models will be released. Demo is available at: https://www.youtube.com/watch?v=QmqWhUjPkJI.

Figures

Figures reproduced from arXiv: 2412.10768 by the authors.

Figure 1
Figure 1. Our VinTAGe model can generate visually aligned and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (A) VT-Encoder, (B) Joint VT-SiT block, (C) Overall training pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Input and target construction. During training, an audio [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: The statistics and examples of our VinTAGe-Bench. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Examples of generated audio from VinTAGe-Bench. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation on the alignment loss. “Birds chirp and a toilet flushes.” “Sheep bleats and fire crackles.” flush starts ~0.5s flush starts ~3.5s bleat bleat bleat bleat “Wind blows.” Ground-truth (On-screen) Text Prompt : Silent Video : Ground-truth Foleycrafter Tango2 Diff…
Figure 8
Figure 8. Figure 8: Effect of classifier free guidance. racy) is mitigated by guidance loss and augmentation. Fur￾thermore, in Tab. 4, we observe that mean-flow and frame￾index features help to improve the generation quality. Effect of svis and stxt. We further analyse the effect of class…
Figure 9
Figure 9. Figure 9: To achieve temporal alignment with the video, we detect [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Teacher models: (A) SiT block for text-only (B) SiT [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Generation examples with different text permutation. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: , we present the metrics when stxt is fixed at 2.5, and svis is incrementally increased from 0.0 to 7.5 in steps of 2.5. We observe a degradation in the FID, indicating a divergence from the real test distribution, alongside an improvement in audio-visual (AV) alignme…
Figure 13
Figure 13. Figure 13: svis = 2.5 and stxt ∈ [0.0, 2.5, 5.0, 7.5] model. Lower text weights indicate a greater contribu￾tion from visual cross-attention (because visual weight is 1-wl). This suggests that the model relies more on visual￾semantic and temporal embeddings to effectively guide …
Figure 14
Figure 14. Figure 14: Distribution of onscreen and offscreen sounds. For each onscreen class, we plot the frequency distribution of off-screen sounds. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Bar-plot for layer-wise cross-attention weights [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 18
Figure 18. Figure 18: VintAGe generated audio (spectrograms) for SORA [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 16
Figure 16. Figure 16: More examples of VinTAGe-Bench [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Interface for subjective study. bird chirps wind blows “Birds chirp and wind blows” “Wind blows and birds chirp” “Birds chirp while wind blows” “Birds chirp as the wind blows” “Amidst the blowing wind, birds sing” “Wind blows” Text Prompt : Video “Birds chirp alongsid…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

    cs.CV 2024-12 conditional novelty 7.0 of 10

    AV-Link unifies video-to-audio and audio-to-video generation by aligning frozen diffusion-model activations with temporally matched rotary position embeddings in a shared Fusion Block.

  2. Sound Scene Synthesis at the DCASE 2024 Challenge

    cs.AI 2025-01 conditional novelty 4.0 of 10

    Four text-to-audio systems were evaluated against a human reference in the DCASE 2024 Task 7 challenge, with a 36% quality gap and strong but small-sample FAD-to-human correlation.

Reference graph

Works this paper leans on

56 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [1]

    The Foley grail: The art of perform- ing sound for film, games, and animation

    Vanessa Theme Ament. The Foley grail: The art of perform- ing sound for film, games, and animation . Routledge, 2014. 1

    work page 2014

  2. [2]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 6

    work page 2023

  3. [3]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  4. [4]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. 2, 6, 8, 12

    work page 2020

  5. [5]

    Gentron: Diffusion trans- formers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion trans- formers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6441–6451, 2024. 6

    work page 2024

  6. [6]

    Audio-vision: sound on screen

    Michel Chion. Audio-vision: sound on screen . Columbia University Press, 2019. 1

    work page 2019

  7. [7]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 2, 11

    work page 2024

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 11

    arXiv 2010

Show all 56 references
  1. [10]

    Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers

    Peng Gao, Le Zhuo, Chris Liu, , Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945, 2024. 3

    2024 arXiv

  2. [11]

    Text-to-audio generation using instruc- tion tuned llm and latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruc- tion tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023. 1, 2, 3, 5, 6, 12, 14

    2023 arXiv

  3. [12]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. 3

    2023

  4. [13]

    Determining op- tical flow

    Berthold KP Horn and Brian G Schunck. Determining op- tical flow. Artificial intelligence, 17(1-3):185–203, 1981. 3, 11

    1981

  5. [14]

    Make-an-audio: Text-to-audio gen- eration with prompt-enhanced diffusion models

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio gen- eration with prompt-enhanced diffusion models. In Inter- national Conference on Machine Learning , pages 13916– 13932...

    2023

  6. [15]

    Captivating sound

    Sander Huiberts. Captivating sound. Utrecht School of the Arts, 2010. 1

    2010

  7. [16]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In British Machine Vision Conference (BMVC), 2021. 2, 6, 7, 8

    2021

  8. [17]

    Mixing audio: concepts, practices, and tools

    Roey Izhaki. Mixing audio: concepts, practices, and tools . Routledge, 2017. 1

    2017

  9. [18]

    Read, watch and scream! sound generation from text and video

    Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. 2024. 1, 2, 3, 4, 6, 7, 8, 12

    2024

  10. [19]

    Fr \’echet audio distance: A metric for evaluating music enhancement algorithms

    Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr \’echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018. 6

    2018 arXiv

  11. [20]

    Hifi-gan: Generative adversarial networks for efficient and high fi- delity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fi- delity speech synthesis. Advances in neural information pro- cessing systems, 33:17022–17033, 2020. 3, 12

    2020

  12. [21]

    Audiogen: Textually guided audio gen- eration

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D ´efossez, Jade Copet, Devi Parikh, Yaniv Taig- man, and Yossi Adi. Audiogen: Textually guided audio gen- eration. arXiv preprint arXiv:2209.15352, 2022. 2

    2022 arXiv

  13. [22]

    Diff-sage: End-to-end spa- tial audio generation using diffusion models

    Saksham Singh Kushwaha, Jianbo Ma, Mark RP Thomas, Yapeng Tian, and Avery Bruni. Diff-sage: End-to-end spa- tial audio generation using diffusion models. arXiv preprint arXiv:2410.11299, 2024. 3

    2024 arXiv

  14. [23]

    V oice- box: Text-guided multilingual universal speech generation at scale, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. V oice- box: Text-guided multilingual universal speech generation at scale, 2023. 3

    2023

  15. [24]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  16. [25]

    Audi- oLDM: Text-to-audio generation with latent diffusion mod- els

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audi- oLDM: Text-to-audio generation with latent diffusion mod- els. Proceedings of the International Conference on Machine Learning, pages 21450–21474, 2023. 1, 2, 3, 12

    2023

  17. [26]

    Flashaudio: Rectified flows for fast and high-fidelity text-to-audio generation

    Huadai Liu, Jialei Wang, Rongjie Huang, Yang Liu, Heng Lu, Wei Xue, and Zhou Zhao. Flashaudio: Rectified flows for fast and high-fidelity text-to-audio generation. arXiv preprint arXiv:2410.12266, 2024. 3

    2024 arXiv

  18. [27]

    Plumbley

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm 2: Learning holistic au- dio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing , ...

    2024

  19. [28]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 3 9

    2022 arXiv

  20. [29]

    Separate anything you describe

    Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D Plumbley, and Wenwu Wang. Separate anything you describe. arXiv preprint arXiv:2308.05037, 2023. 6, 15

    2023 arXiv

  21. [30]

    Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models, 2023

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models, 2023. 1, 2, 4, 6, 7, 8, 14

    2023

  22. [31]

    Albergo, Nicholas M

    Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. 2024. 2, 3, 4, 12

    2024

  23. [32]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization, 2024

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization, 2024. 1, 2, 6, 7, 13

    2024

  24. [33]

    Samplernn: An unconditional end- to-end neural audio generation model

    Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end- to-end neural audio generation model. arXiv preprint arXiv:1612.07837, 2016. 2

    2016 arXiv

  25. [34]

    Text-to- audio generation synchronized with videos

    Shentong Mo, Jing Shi, and Yapeng Tian. Text-to- audio generation synchronized with videos. arXiv preprint arXiv:2403.07938, 2024. 3

    2024 arXiv

  26. [35]

    Bal- ancing act: Distribution-guided debiasing in diffusion mod- els

    Rishubh Parihar, Abhijnya Bhat, Abhipsa Basu, Saswat Mallick, Jogendra Nath Kundu, and R Venkatesh Babu. Bal- ancing act: Distribution-guided debiasing in diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 6668–6678,

  27. [36]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 3, 4, 12

    2023

  28. [37]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI con- ference on artificial intelligence, 2018. 4

    2018

  29. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  31. [40]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  32. [41]

    I hear your true colors: Image guided audio generation, 2022

    Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation, 2022. 2, 6, 8

    2022

  33. [42]

    Auto- acd: A large-scale dataset for audio-language representation learning

    Luoyi Sun, Xuenan Xu, Mengyue Wu, and Weidi Xie. Auto- acd: A large-scale dataset for audio-language representation learning. In ACM Multimedia, 2024. 6, 8

    2024

  34. [43]

    Learning from between-class examples for deep sound recognition

    Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Learning from between-class examples for deep sound recognition. arXiv preprint arXiv:1711.10282 , 2017. 12, 14

    2017 arXiv

  35. [44]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 4

    2017

  36. [45]

    Audiobox: Unified audio generation with natural language prompts, 2023

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, ...

    2023

  37. [46]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foun- dation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foun- dation models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. 1, 3, 6

    2024

  38. [47]

    One-peace: Exploring one general representa- tion model toward unlimited modalities

    Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xi- aohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representa- tion model toward unlimited modalities. arXiv preprint arXiv:2305.11172, 2023. 6

    2023 arXiv

  39. [48]

    Wav2clip: Learning robust audio repre- sentations from clip

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio repre- sentations from clip. In ICASSP 2022 - 2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2022. 6

    2022

  40. [49]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Si...

    2023

  41. [50]

    Son- icvisionlm: Playing sound with vision language models

    Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Son- icvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 26866–26875, 2024. 2, 3, 4

    2024

  42. [51]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. In CVPR,

  43. [52]

    Diffsound: Discrete diffusion model for text-to-sound generation

    Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 31:1720–1733, 2023. 2

    2023

  44. [53]

    Diverse and aligned audio-to-video genera- tion via text-to-video model adaptation

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video genera- tion via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6639– 6647, 2024. 3, 7, 11, 14

    2024

  45. [54]

    Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds, 2024

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds, 2024. 1, 2, 3, 4, 6, 7, 8 10

    2024

  46. [55]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 6, 7, 12

    2024

  47. [56]

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  48. [57]

    Wind blows

    Le Zhuo, Ruoyi Du, Xiao Han, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit. arXiv preprint arXiv:2406.18583, 2024. 3 Appendix A. Implementation details Video energy Video energy sampled at 0.25 s F...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.