REVIEW 13 cited by
Efficient Training of Audio Transformers with Patchout
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transformers can outperform Convolutional Neural Networks (CNNs) on vision and audio tasks. However, one of the main shortcomings of transformer models, compared to the well-established CNNs, is the computational complexity. In transformers, the compute and memory complexity is known to grow quadratically with the input length. Therefore, there has been extensive work on optimizing transformers, but often at the cost of degrading predictive performance. In this work, we propose a novel method to optimize and regularize transformers on audio spectrograms. Our proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU. Furthermore, we propose a transformer model that outperforms CNNs in terms of both performance and training speed. Source code: https://github.com/kkoutini/PaSST
Forward citations
Cited by 13 Pith papers
-
Conditional Flow Matching for Visually-Guided Acoustic Highlighting
Conditional flow matching with a rollout loss and early audio-visual fusion achieves state-of-the-art results on visually-guided acoustic highlighting.
-
UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
A unified audio-video generator built by stitching pre-trained video and music diffusion models, trained on 7,600 hours of data, with a new evaluation benchmark.
-
MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
A lightweight adapter that adds rotary position embeddings to decoupled cross-attention enables efficient time-varying style control and audio inpainting/outpainting for text-to-music diffusion Transformers.
-
Video-Guided Text-to-Music Generation Using Public Domain Movie Collections
OSSL is the first self-hosted, mood-annotated video-music dataset, and a video adapter on MusicGen-Medium improves film music generation over text-only baselines.
-
SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
SSLAM pre-trains audio transformers on partially mixed audio clips with a source retention loss, improving polyphonic sound tagging while keeping monophonic benchmark scores.
-
SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
A ControlNet branch plus a frequency-aware feature aligner lets a pretrained masked generative TTA model produce video-synchronized foley, beating several from-scratch models on VGGSound.
-
FdAudio: MeanFlow-Anchored Fr\'echet-Distance Post-Training for One-Step Text-to-Audio Generation
MeanFlow-anchored multi-representation FD post-training improves one-step text-to-audio quality without collapsing multi-step sampling.
-
JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing
A dual-branch diffusion transformer with joint video-audio self-attention and a keypoint-based mouth-area loss reports top lip-sync and speech metrics on two benchmarks.
-
AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation
A single multimodal diffusion transformer generates video-synchronized general audio, speech, and song from flexible combinations of video, text, and lyrics inputs.
-
Adaptive Knowledge Distillation using a Device-Aware Teacher for Low-Complexity Acoustic Scene Classification
A low-complexity scene classifier trained by two-teacher knowledge distillation and device-specific fine-tuning reaches 57.93% accuracy on the DCASE 2025 development set.
-
Joint Feature and Output Distillation for Low-complexity Acoustic Scene Classification
A 61k-parameter acoustic scene classifier trained with joint output-level and feature-level knowledge distillation reaches 59.30% accuracy on TAU Urban Acoustic Scenes 2022 Mobile.
-
Enhancing Stereo Sound Event Detection with BiMamba and Pretrained PSELDnet
Replacing the Conformer decoder in pretrained PSELDnet with a bidirectional Mamba plus asymmetric convolution reports 39.6% versus 38.2% stereo SELD F20 on the DCASE2025 development set, using 76M versus 210M parameters.
-
ASAudio: A Survey of Advanced Spatial Audio Research
A comprehensive survey that systematically categorizes spatial audio research by representation, task, dataset, and evaluation.
Discussion (0). Continue with ORCID to comment.