Pith. sign in

REVIEW 4 major objections 4 minor 48 references

SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single ControlNet branch can steer a pretrained text-to-audio transformer into competitive video-synchronized foley synthesis.

desk verdict Solid ControlNet-on-pretrained-TTA paper that deserves review, but the DeSync metric may be circular with the Synchformer conditioning and needs an independent check. read the letter →

arxiv 2505.16195 v2 pith:67OR7X4N submitted 2025-05-22 cs.SD cs.AIcs.LGeess.ASeess.IV

Zhi Zhong , Akira Takahashi , Shuyang Cui , Keisuke Toyama , Shusuke Takahashi , Yuki Mitsufuji This is my paper
classification cs.SDcs.AIcs.LGeess.ASeess.IV
keywords foleysynthesisvideo-to-audioControlNetSpecMaskGITmaskedgenerativetransformerfrequency-awaretemporalfeaturealigneraudio-visualsynchronizationdeepvideofeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpecMaskFoley claims that a pretrained text-to-audio model, SpecMaskGIT, can be turned into a video-synchronized foley synthesizer without retraining the audio generator from scratch. The method adds one trainable ControlNet branch, initialized from half of the backbone, that consumes deep video features through a frequency-aware temporal feature aligner. On the VGGSound benchmark the paper reports that this single-branch design outperforms earlier ControlNet-based foley systems and is competitive with, sometimes better than, from-scratch systems. If the claim holds, foley synthesis becomes an adaptation problem rather than a large-scale generative-model training problem, requiring only about 126 million trainable parameters and single-GPU training.

What carries the argument

The load-bearing object is the frequency-aware temporal feature aligner (FT-Aligner) coupled to a zero-initialized Transformer ControlNet branch. SpecMaskGIT represents a 10-second clip as a $[F=5, T=53, D=768]$ discrete token map; the FT-Aligner takes 1-D video features of shape $[t,d]$, projects them with a 1-D convolution and adaptive average pooling to $[1,T,D]$, then repeats them along the frequency axis to $[F,T,D]$. This identical-across-frequency injection preserves the temporal position of video cues inside a 2-D time-frequency grid. The ControlNet branch is initialized with the first 12 Transformer blocks of the frozen backbone, connected through zero-initialized linear layers, and trained by standard cross-entropy on the discrete tokens; a two-condition classifier-free guidance interpolates between unconditional, video-only, and text-plus-video logits. The paper states that without the FT-Aligner's exact alignment, training SpecMaskFoley would not be possible.

What would settle it

Train SpecMaskFoley again with the FT-Aligner replaced by a learned per-frequency linear projection of the same temporal features, or with features broadcast only through attention; if DeSync and FAD do not worsen, the claimed necessity of the frequency-repeat alignment is disproved. A second test uses clips whose sound timing differs by frequency band, such as a high-frequency click followed by a low-frequency rumble, because the model injects identical features across all frequency rows and its sync scores on such clips would reveal whether that inductive bias is harmful.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the time-frequency structure of a pretrained masked generative audio model does not block video control: a single ControlNet branch can inject high-dimensional deep video features into SpecMaskGIT once those features are aligned to the 2-D token map. The FT-Aligner downsamples 1-D temporal features from Synchformer and CLIP to the model's 53 temporal positions and repeats them across the 5 frequency rows, so every token at the same time step receives the same video information. With only cross-entropy loss and a two-condition classifier-free guidance, SpecMaskFoley reaches a DeSync of 0.65 seconds, FAD of 1.03, and 0.47-second inference for a 10-second clip, beating previous ControlNet methods and matching strong from-scratch baselines. The paper concludes that this narrows the gap between adapted pretrained models and from-scratch foley models.

Load-bearing premise

The load-bearing premise is that copying the same downsampled video feature onto every frequency row of the audio token grid preserves enough temporal information for synchronization; if that alignment is wrong, the ControlNet branch has no other route to tell the generator when a sound should happen.

Editorial extensions

If this is right

  • If the central claim holds, pretrained text-to-audio checkpoints can be reused for foley, cutting trainable parameters to 126M and enabling single-GPU training instead of from-scratch training of much larger models.
  • ControlNet-based foley becomes competitive with from-scratch generation on both audio quality and audio-video synchronization, so deep video features no longer require a purpose-built audio generator.
  • SpecMaskFoley's 12-step inference (0.47 seconds for a 10-second clip) suggests near-real-time foley for editing workflows without distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is that the FT-Aligner recipe should apply to other 2-D time-frequency masked generative audio models, because the aligner only requires matching the temporal dimension and feature dimension of the target latent.
  • The paper hypothesizes that CLIP features are mostly semantic and averages them globally; a testable variant would condition on per-frame CLIP features to see whether scene changes within a clip benefit from time-varying semantic information.
  • The paper acknowledges its 2-D latent has only 53 temporal frames, which may cap synchronization on very fast events; transplanting the method to a 1-D MaskGIT latent, the stated future work, would test that ceiling.
  • A controlled comparison swapping the FT-Aligner for per-frequency learned projections would test how much of the gain comes from this specific alignment, since the paper's claim that training fails without it is not accompanied by ablative evidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SpecMaskFoley, a method that equips the pretrained text-to-audio model SpecMaskGIT with a transformer-based ControlNet branch, enabling video-conditioned foley synthesis without training the audio backbone from scratch. To reconcile 1-D temporal video features (Synchformer and globally averaged CLIP features) with the 2-D time-frequency token map of SpecMaskGIT, the authors introduce a frequency-aware temporal feature aligner (FT-Aligner) that downsamples the video features, repeats them along the frequency axis, and injects them into the ControlNet branch. A multi-condition classifier-free guidance scheme is also introduced. On the VGGSound benchmark, the method reports competitive FAD, KL, ImageBind similarity, and DeSync scores, and the authors claim it can outperform several from-scratch baselines while training only a 126M-parameter branch on a single GPU.

Significance. If the results are taken at face value, the paper makes a useful contribution by showing that a simple ControlNet branch with a single feature-alignment mechanism can steer a pretrained masked generative transformer into a competitive video-to-audio foley model. The core idea is plausible and clearly presented: the zero-initialized ControlNet formulation is adapted to a discrete masked generative model, and the FT-Aligner is a natural way to broadcast temporal features across frequency positions. The authors provide a public benchmark evaluation, an open demo page, ablations of the multi-CFG and inference-step settings, and a careful specification of hyperparameters, all of which are strengths. The main weakness is that the only temporal-synchronization metric (DeSync) is computed with the same family of Synchformer encoders that provide the model's video conditioning, which threatens the headline 'synchronized foley' claim; the paper also lacks confidence intervals and quantitative FT-Aligner ablations.

major comments (4)
  1. [§3.2, §4.3, Table 1] The DeSync metric is load-bearing for the central claim of temporal synchronization, but the paper does not establish that the metric is independent of the conditioning features. Section 3.2 states that the model conditions on 25-Hz Synchformer deep features, and Section 4.3 defines DeSync using the metric attributed to MMAudio [6] and Synchformer [12]. If the Synchformer encoder used to compute DeSync is the same model (or shares weights) with the encoder that provides the conditional features, then the reported DeSync advantage over from-scratch baselines (e.g., 0.65 s vs. Frieren's 0.85 s in Table 1) may reflect the model being trained to satisfy that specific evaluator rather than genuine audiovisual synchronization. The authors should either use an independent synchronization evaluator (e.g., a differently trained audio-visual sync model or a human listening study) or, at minimum, explicitly disclose whether the DeSync encoder weights differ from the conditioning encoder and justify why leakage is not a concern. As written, the paper's only temporal-alignment evidence is potentially circular.
  2. [§3.3] The paper asserts that without the FT-Aligner's careful frequency alignment 'we would not be able to train SpecMaskFoley,' but this claim is not supported by any quantitative ablation. No comparison is provided against alternative ways of injecting 1-D temporal features into the 2-D latent, such as learned per-frequency embeddings, cross-attention over the frequency axis, or directly concatnating the 1-D feature before the transformer. Since the FT-Aligner is presented as the key innovation that unlocks a single ControlNet branch, the absence of an ablation makes it impossible to assess whether the proposed inductive bias (identical temporal features across all five frequency rows) is necessary, sufficient, or optimal. Please add an ablation table with at least one alternative alignment strategy, and report the resulting FAD/DeSync values (or a clear failure description if training diverges).
  3. [Table 1, Table 2] All reported benchmark scores are point estimates without confidence intervals or significance tests. The abstract's statement that SpecMaskFoley 'could even outperform strong from-scratch baselines' is supported only by comparisons of single numbers (e.g., FAD 1.03 vs. Frieren 1.34 in Table 1). Given that the main differences are sometimes small, the authors should report error bars computed over multiple evaluation runs or across random seeds, or perform a simple significance test for the headline comparisons. In addition, when citing this result, please note that MMAudio [6] still achieves better scores on all quality and synchronization metrics in Table 1; the current wording 'outperform strong from-scratch baselines' accurately refers to selected baselines but the sentence could be misinterpreted.
  4. [§3.4 / §5.2] The description of the multi-CFG scheme in Eq. (1) contains an unstated mismatch that should be clarified. The text says 'We do not use logits gained by conditioning the audio backbone with CLAP text features,' but the training-time formula includes ℓ_text&video, which involves CLAP text features. On reading further, the intended meaning is that no text-only CFG term is used, but the current wording is confusing. In addition, Section 5.2's ablation of multi-CFG does not actually isolate the effect of text prompts alone: the two listed ablations remove either ℓ_text&video or ℓ_video, but there is no condition that removes only the effect of CLAP text conditioning while keeping video conditioning. The paper's statement that 'text prompts alone are less important' (based on the 90% random masking) is therefore not directly verified by the reported ablation.
minor comments (4)
  1. [Abstract and throughout] There are several typographical errors, e.g., 'feautures' in Section 3.4, 'syntheis' in Section 5.1, and 'F oley' in the abstract. The paper would benefit from a careful proofreading pass.
  2. [§4.3] The sentence 'We use the av-benchmark 1 to evaluate...' is missing a period after the footnote reference. More importantly, the footnote 'PANN has been reported as not being robust in some scenarios' would be more useful if it cited the specific evidence or stated whether PANN was excluded before or after computing the reported scores.
  3. [Fig. 2 and §3.3] The caption of Fig. 2 mentions 'a CLAP embedding is treated as a conditional mask C' and the figure uses fire/ice icons for trainable/frozen modules. Readers would benefit from a caption that also labels the FT-Aligner and the ControlNet branch explicitly, since these are the paper's main contributions.
  4. [§5.2, 'Impact of inference steps'] The text says 'the optimal number of inference steps was 16' for SpecMaskGIT and 'scores saturate after 12 steps' for SpecMaskFoley. The FAD-vs-iterations plot in Fig. 3 shows a non-monotone curve; please state whether the 12-step choice was made by early stopping on the test set, and whether the same conclusion holds for the FD and KL metrics (which are not shown in Fig. 3).

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: benchmark scores are measured outputs; Synchformer overlap in conditioning and DeSync metric is a metric-leakage concern, not a formal reduction.

full rationale

The paper's central claim is empirical: a frozen SpecMaskGIT plus a trainable ControlNet branch with FT-Aligner produces competitive foley on VGGSound. The reported FAD, KL, IB, and DeSync numbers are measured outputs, not quantities derived from fitted constants. The multi-CFG formula (Eq. 1) is a standard extension of CFG and contains no hidden dependency on the result. The FT-Aligner downsampling-and-repeat operation is an architectural choice; the statement 'Without this careful feature alignment, we would not be able to train SpecMaskFoley' (Sec. 3.3) lacks an ablation but is not circular. Self-citations to SpecMaskGIT and MMAudio are normal and not load-bearing for the architectural claim. The one notable overlap—Synchformer [12] is used both to extract the deep video features that condition the model (Sec. 3.2) and as the basis of the DeSync synchronization metric (Sec. 4.3)—is a genuine external-validity risk: the model is given access to the same representation family that defines the sync metric, so DeSync differences (e.g., vs. Frieren) may partly reflect this conditioning rather than general audiovisual synchrony. However, the paper does not define its output in terms of DeSync, does not train to optimize DeSync, and the audio is generated by cross-entropy token prediction; therefore no formal circular reduction can be exhibited. Score 2 reflects minor self-citation overlap rather than circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the transferability of a frozen pretrained TTA model, the adequacy of Synchformer and CLIP features, and an empirically asserted alignment strategy. No new physical or conceptual entities are introduced; the FT-Aligner is an architectural component rather than a new entity. Several design choices, such as CFG scale, temperature, and unconditional mask rate, are hand-set and affect the final scores.

free parameters (5)
  • Multi-CFG scale schedule = linear increase from 0 to 3 over 12 inference steps
    Chosen by hand for evaluation; ablations show it improves all metrics, so the benchmark numbers depend on this schedule.
  • Gumbel temperature = 9.0
    Standard SpecMaskGIT sampling temperature, selected for evaluation; affects token sampling stochasticity and reported scores.
  • Unconditional CLAP mask rate = 90%
    Training choice in Section 3.4; the paper states text prompts alone are less important, but the exact rate is hand-set.
  • Base learning rate and scaling = 1e-3, scaled by batch size / 256
    Training hyperparameter in Section 4.2; required to reproduce the ControlNet training recipe.
  • ControlNet branch depth = first 12 of 24 Transformer blocks
    Architecture choice in Section 4.2; the number of copied blocks affects capacity and is not derived from first principles.
assumptions (5)
  • domain assumption SpecMaskGIT's pretrained SpecVQGAN and MaskGIT backbone can be adapted without joint retraining of the full model.
    Section 3.1 uses the official checkpoint as a frozen backbone; the low-cost claim depends on this transfer working.
  • domain assumption Synchformer (25 Hz) and CLIP (8 Hz) video features, with CLIP globally averaged, provide sufficient synchronization and semantic information for foley.
    Section 3.2 adopts these features from MMAudio; if they are insufficient, the ControlNet branch cannot recover temporal alignment.
  • ad hoc to paper Injecting identical temporal features across all frequency tokens preserves temporal alignment in a 2-D latent.
    Section 3.3 states empirically that without this alignment training fails, but no ablative evidence is provided.
  • domain assumption av-benchmark metrics (PaSST FD/KL, VGGish FAD, ImageBind IB similarity, Synchformer DeSync) are valid proxies for foley quality and synchronization.
    Section 4.3 defines the evaluation; excluding PANN changes the metric set relative to some prior work.
  • domain assumption VGGSound concatenated tags serve as text prompts for evaluation.
    Section 4.1 follows common practice in ReWaS, VATT, FoleyCrafter, and MMAudio; test-set tags are available in the benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet." pith.science (2026). https://pith.science/paper/67OR7X4N

@misc{pith2026250516195,
  author       = {Pith},
  title        = {Pith review of: SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67OR7X4N}},
  note         = {Machine review of arXiv:2505.16195}
}
read the original abstract

Foley synthesis aims to synthesize high-quality audio that is both semantically and temporally aligned with video frames. Given its broad application in creative industries, the task has gained increasing attention in the research community. To avoid the non-trivial task of training audio generative models from scratch, adapting pretrained audio generative models for video-synchronized foley synthesis presents an attractive direction. ControlNet, a method for adding fine-grained controls to pretrained generative models, has been applied to foley synthesis, but its use has been limited to handcrafted human-readable temporal conditions. In contrast, from-scratch models achieved success by leveraging high-dimensional deep features extracted using pretrained video encoders. We have observed a performance gap between ControlNet-based and from-scratch foley models. To narrow this gap, we propose SpecMaskFoley, a method that steers the pretrained SpecMaskGIT model toward video-synchronized foley synthesis via ControlNet. To unlock the potential of a single ControlNet branch, we resolve the discrepancy between the temporal video features and the time-frequency nature of the pretrained SpecMaskGIT via a frequency-aware temporal feature aligner, eliminating the need for complicated conditioning mechanisms widely used in prior arts. Evaluations on a common foley synthesis benchmark demonstrate that SpecMaskFoley could even outperform strong from-scratch baselines, substantially advancing the development of ControlNet-based foley synthesis models. Demo page: https://zzaudio.github.io/SpecMaskFoley_Demo/

Figures

Figures reproduced from arXiv: 2505.16195 by the authors.

Figure 1
Figure 1. Audio synthesis quality (FAD [11]) and audio-video temporal alignment [12] of different methods. The proposed SpecMaskFoley achieves competitive scores in both axes without from-scratch training. [21], from-scratch models like Frieren [19] and MMAudio [6] have brought foley synthesis to a new level. As shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of SpecMaskFoley. Ice icons: frozen modules. Fire icons: trainable modules. A CLAP embedding is treated as a conditional [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Left: FAD vs. Number of iterations. Right: DeSync vs. Number [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 28 canonical work pages

  1. [6]

    Taming multimodal joint training for high-quality video- to-audio synthesis,

    H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y . Mitsufuji, “Taming multimodal joint training for high-quality video- to-audio synthesis,” arXiv preprint arXiv:2412.15322 , 2024

    arXiv 2024

  2. [12]

    Synchformer: Efficient synchronization from sparse cues,

    V . Iashin, W. Xie, E. Rahtu, and A. Zisserman, “Synchformer: Efficient synchronization from sparse cues,” in Proc. ICASSP 2024 . IEEE, 2024, pp. 5325–5329

    work page 2024

  3. [1]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio generation,” arXiv preprint arXiv:2209.15352 , 2022

    arXiv 2022

  4. [2]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation,

    J. Huang, Y . Ren, R. Huang, D. Yang, Z. Ye, C. Zhang, J. Liu, X. Yin, Z. Ma, and Z. Zhao, “Make-an-audio 2: Temporal-enhanced text-to-audio generation,” arXiv preprint arXiv:2305.18474 , 2023

    arXiv 2023

  5. [3]

    Specmaskgit: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond,

    M. Comunit `a, Z. Zhong, A. Takahashi, S. Yang, M. Zhao, K. Saito, Y . Ikemiya, T. Shibuya, S. Takahashi, and Y . Mitsufuji, “Specmaskgit: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond,” arXiv preprint arXiv:2406.17672 , 2024

    arXiv 2024

  6. [4]

    Soundctm: Uniting score-based and consistency models for text-to-sound generation,

    K. Saito, D. Kim, T. Shibuya, C.-H. Lai, Z. Zhong, Y . Takida, and Y . Mitsufuji, “Soundctm: Uniting score-based and consistency models for text-to-sound generation,” arXiv preprint arXiv:2405.18503 , 2024

    arXiv 2024

  7. [5]

    Stable audio open,

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” in Proc. ICASSP 2025 . IEEE, 2025, pp. 1–5

    work page 2025

  8. [7]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,

    L. Ruan, Y . Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo, “Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,” in Proc. CVPR 2023 , 2023, pp. 10 219–10 228

    work page 2023

Show all 48 references
  1. [8]

    Visual echoes: A simple unified transformer for audio- visual generation,

    S. Yang, Z. Zhong, M. Zhao, S. Takahashi, M. Ishii, T. Shibuya, and Y . Mitsufuji, “Visual echoes: A simple unified transformer for audio- visual generation,” arXiv preprint arXiv:2405.14598 , 2024

    2024 arXiv

  2. [9]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,

    H. Wang, J. Ma, S. Pascual, R. Cartwright, and W. Cai, “V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,” in Proc. AAAI 2024 , 2024, pp. 15 492–15 501

    2024

  3. [10]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,

    Y . Xing, Y . He, Z. Tian, X. Wang, and Q. Chen, “Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,” in Proc. CVPR 2024 , 2024, pp. 7151–7161

    2024

  4. [11]

    Fr \’echet audio distance: A metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr \’echet audio distance: A metric for evaluating music enhancement algorithms,” arXiv preprint arXiv:1812.08466, 2018

    2018 arXiv

  5. [13]

    Adding conditional control to text- to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text- to-image diffusion models,” in Proc. CVPR 2023 , 2023, pp. 3836–3847

    2023

  6. [14]

    Uni-controlnet: All-in-one control to text-to-image diffusion models,

    S. Zhao, D. Chen, Y .-C. Chen, J. Bao, S. Hao, L. Yuan, and K.-Y . K. Wong, “Uni-controlnet: All-in-one control to text-to-image diffusion models,” Proc. NeurIPS 2023 , vol. 36, pp. 11 127–11 150, 2023

    2023

  7. [15]

    Read, watch and scream! sound generation from text and video,

    Y . Jeong, Y . Kim, S. Chun, and J. Lee, “Read, watch and scream! sound generation from text and video,” arXiv preprint arXiv:2407.05551 , 2024

    2024 arXiv

  8. [16]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,

    Y . Zhang, Y . Gu, Y . Zeng, Z. Xing, Y . Wang, Z. Wu, and K. Chen, “Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,” arXiv preprint arXiv:2407.01494 , 2024

    2024 arXiv

  9. [17]

    Tell what you hear from what you see– video to audio generation through text,

    X. Liu, K. Su, and E. Shlizerman, “Tell what you hear from what you see– video to audio generation through text,” arXiv preprint arXiv:2411.05679 , 2024

    2024 arXiv

  10. [18]

    Temporally aligned audio for video with autoregression,

    I. Viertola, V . Iashin, and E. Rahtu, “Temporally aligned audio for video with autoregression,” in Proc. ICASSP 2025 . IEEE, 2025, pp. 1–5

    2025

  11. [19]

    Frieren: Efficient video-to-audio generation network with rectified flow matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation network with rectified flow matching,” Proc. NeurIPS 2024 , vol. 37, pp. 128 118– 128 138, 2024

    2024

  12. [20]

    Mavil: Masked audio-video learners,

    P.-Y . Huang, V . Sharma, H. Xu, C. Ryali, Y . Li, S.-W. Li, G. Ghosh, J. Malik, C. Feichtenhofer et al. , “Mavil: Masked audio-video learners,” Proc. NeurIPS 2023 , vol. 36, pp. 20 371–20 393, 2023

    2023

  13. [21]

    Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models,

    S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models,” Proc. NeurIPS 2023 , vol. 36, pp. 48 855–48 876, 2023

    2023

  14. [22]

    Stable-v2a: Synthesis of synchronized sound effects with temporal and semantic controls,

    R. F. Gramaccioni, C. Marinoni, E. Postolache, M. Comunit `a, L. Cosmo, J. D. Reiss, and D. Comminiello, “Stable-v2a: Synthesis of synchronized sound effects with temporal and semantic controls,” arXiv preprint arXiv:2412.15023, 2024

    2024 arXiv

  15. [23]

    Smooth-foley: Creating continuous sound for video-to-audio generation under semantic guidance,

    Y . Zhang, X. Xu, and M. Wu, “Smooth-foley: Creating continuous sound for video-to-audio generation under semantic guidance,” in Proc. ICASSP

  16. [24]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Proc. ICML 2021 . PmLR, 2021, pp. 8748–8763

    2021

  17. [25]

    Maskgit: Masked generative image transformer,

    H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “Maskgit: Masked generative image transformer,” in Proc. CVPR 2022 , 2022, pp. 11 315–11 325

    2022

  18. [26]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,” arXiv preprint arXiv:2303.15389 , 2023

    2023 arXiv

  19. [27]

    Music foundation model as generic booster for music downstream tasks,

    W. Liao, Y . Takida, Y . Ikemiya, Z. Zhong, C.-H. Lai, G. Fabbro, K. Shimada, K. Toyama, K. Cheuk, M. A. Mart ´ınez-Ram´ırez et al. , “Music foundation model as generic booster for music downstream tasks,” arXiv preprint arXiv:2411.01135 , 2024

    2024 arXiv

  20. [28]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  21. [29]

    High- fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- fidelity audio compression with improved rvqgan,” Proc. NeurIPS 2023 , vol. 36, pp. 27 980–27 993, 2023

    2023

  22. [30]

    Taming visually guided sound generation,

    V . Iashin and E. Rahtu, “Taming visually guided sound generation,”arXiv preprint arXiv:2110.08791, 2021

    2021 arXiv

  23. [31]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in CVPR 2022 , 2022, pp. 16 000–16 009

    2022

  24. [32]

    Extending audio masked autoencoders toward audio restoration,

    Z. Zhong, H. Shi, M. Hirano, K. Shimada, K. Tateishi, T. Shibuya, S. Takahashi, and Y . Mitsufuji, “Extending audio masked autoencoders toward audio restoration,” in IEEE WASPAA 2023 , 2023, pp. 1–5

    2023

  25. [33]

    Mage: Masked generative encoder to unify representation learning and image synthesis,

    T. Li, H. Chang, S. Mishra, H. Zhang, D. Katabi, and D. Krishnan, “Mage: Masked generative encoder to unify representation learning and image synthesis,” in Proc. CVPR 2023 , 2023, pp. 2142–2152

    2023

  26. [34]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXiv preprint arXiv:2301.12503 , 2023

    2023 arXiv

  27. [35]

    Pixart- {\delta}: Fast and controllable image generation with latent consistency models,

    J. Chen, Y . Wu, S. Luo, E. Xie, S. Paul, P. Luo, H. Zhao, and Z. Li, “Pixart- {\delta}: Fast and controllable image generation with latent consistency models,” arXiv preprint arXiv:2401.05252 , 2024

    2024 arXiv

  28. [36]

    Cocola: Coherence-oriented contrastive learning of musical audio representations,

    R. Ciranni, G. Mariani, M. Mancusi, E. Postolache, G. Fabbro, E. Rodol `a, and L. Cosmo, “Cocola: Coherence-oriented contrastive learning of musical audio representations,” arXiv preprint arXiv:2404.16969 , 2024

    2024 arXiv

  29. [37]

    Editing music with melody and text: Using controlnet for diffusion transformer,

    S. Hou, S. Liu, R. Yuan, W. Xue, Y . Shan, M. Zhao, and C. Zhang, “Editing music with melody and text: Using controlnet for diffusion transformer,” in Proc. ICASSP 2025 . IEEE, 2025, pp. 1–5

    2025

  30. [38]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  31. [39]

    Muse: Text-to- image generation via masked generative transformers,

    H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein et al. , “Muse: Text-to- image generation via masked generative transformers,” arXiv preprint arXiv:2301.00704, 2023

    2023 arXiv

  32. [40]

    Stemgen: A music generation model that listens,

    J. D. Parker, J. Spijkervet, K. Kosta, F. Yesiler, B. Kuznetsov, J.-C. Wang, M. Avent, J. Chen, and D. Le, “Stemgen: A music generation model that listens,” in Proc. ICASSP 2024 . IEEE, 2024, pp. 1116–1120

    2024

  33. [41]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proc. CVPR 2023 , 2023, pp. 15 180–15 190

    2023

  34. [42]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc. ICASSP 2023 , 2023, pp. 1–5

    2023

  35. [43]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in Proc. ICASSP 2017 . IEEE, 2017, pp. 776–780

    2017

  36. [44]

    Effi- cient training of audio transformers with patchout,

    K. Koutini, J. Schl ¨uter, H. Eghbal-Zadeh, and G. Widmer, “Effi- cient training of audio transformers with patchout,” arXiv preprint arXiv:2110.05069, 2021

    2021 arXiv

  37. [45]

    Vggsound: A large- scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large- scale audio-visual dataset,” in Proc. ICASSP 2020 . IEEE, 2020, pp. 721–725

    2020

  38. [46]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” NeurIPS, vol. 33, pp. 17 022–17 033, 2020

    2020

  39. [47]

    Cnn architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold et al. , “Cnn architectures for large-scale audio classification,” in Proc. ICASSP 2017 . IEEE, 2017, pp. 131–135

    2017

  40. [48]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM TASLP, vol. 28, pp. 2880–2894, 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.