Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Can Sound Replace Vision in LLaVA With Token Substitution?

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes an encoder-dependent trade-off: aligning audio tokens to the CLIP visual manifold improves cross-modal retrieval but degrades text generation, and the encoder's pretraining paradigm decides which side wins.

desk verdict A broad, clearly written empirical study with a genuinely useful new alignment dataset, but the central trade-off claim is weakened by a raw-vs-projected comparison that conflates alignment with a learned 1.9M-parameter bottleneck. read the letter →

arxiv 2506.10416 v2 pith:B6R6DHLB submitted 2025-06-12 cs.MM cs.SDeess.AS

Ali Vosoughi , Jing Bi , Pinxin Liu , Yunlong Tang , Chenliang Xu This is my paper · ORCID
classification cs.MMcs.SDeess.AS
keywords audio-visualalignmentcross-modalretrievaltextgenerationCLIPembeddingspacetokensubstitutionLLaVAaudioencodersWhisper
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a single audio token can substitute for visual tokens in LLaVA, and what happens when audio-visual alignment is pushed to an extreme. To answer it, the authors built AV E-2, a dataset of roughly 580,000 clips with continuous five-dimensional alignment scores, and used the most aligned pairs to train a small projection from five audio encoders into the fixed CLIP visual space. Their central finding is an encoder-dependent trade-off: projecting audio into CLIP space produces large gains in cross-modal retrieval, but consistently lowers the quality of generated captions because the projection sacrifices audio-specific information. Encoders pretrained with direct audio-language supervision (CLAP, Whisper) retain more of that information and balance the two objectives, while image-centric encoders (ImageBind, Wav2CLIP, AudioCLIP) favor retrieval at a larger cost to generation. The paper also introduces WhisperCLIP, a layer-averaged Whisper token, as a midpoint that preserves both acoustic detail and visual grounding.

What carries the argument

The load-bearing identity is $H(A)=I(A;V)+H(A|V)$, read as a conservation law: every bit of mutual information that projection adds between audio and the CLIP visual manifold must come out of $H(A|V)$, the audio-specific content that supports fluent, detailed captions. The operational mechanism is token substitution in LLaVA: the audio token $a$ replaces the visual [CLS] token, while $k$ visual patch tokens ($k=15$ or $150$) are kept, chosen either uniformly at random or by cosine similarity to the audio token, and the language model generates under $P(w|a,v_{i_1},...,v_{i_k})$. Alignment is imposed either by the 1.9M-parameter MLP that maps the audio embedding into CLIP space, or by raw padding that preserves the embedding's native geometry. WhisperCLIP refines the audio token by averaging hidden states across all Whisper layers, which retains speech semantics, prosody, and environmental context before alignment.

What would settle it

Train an information-preserving map from raw audio embeddings to CLIP space, such as a normalizing flow or an invertible projection, and measure retrieval and caption quality under the same token-substitution setup. If the caption drop disappears while retrieval gains persist, the trade-off is an artifact of the projection; if the drop persists, alignment itself is the cause.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that alignment with the CLIP visual manifold is a double-edged sword for audio-conditioned vision-language models. Replacing LLaVA's visual [CLS] token with an audio token works only if a minimal set of patch tokens remains, and the form of that audio token determines performance: a learned 1.9M-parameter MLP projection into CLIP space markedly improves audio-to-video and video-to-audio retrieval, while raw padded embeddings produce richer descriptions but weaker retrieval. The paper explains the pattern with the entropy decomposition $H(A)=I(A;V)+H(A|V)$: a deterministic projection can increase shared information only by cutting the audio-unique information that generation needs. It then shows that the encoder's pretraining paradigm decides how severe the cut is, with language-supervised encoders degrading less, and proposes WhisperCLIP, which averages Whisper's hidden layers to keep more of that unique information while still aligning to CLIP.

Load-bearing premise

The central claim treats the 1.9M-parameter projection as a faithful probe of 'alignment with the visual manifold,' so if the caption-quality drop comes from the projection's bottleneck or from the distribution shift of the projected tokens rather than from alignment itself, the trade-off is not established.

Editorial extensions

If this is right

  • System builders can select an audio encoder by task, using image-centric encoders for cross-modal retrieval and text-centric encoders for open-ended generation with audio-substituted vision tokens.
  • A small projection into CLIP space is a cheap way to convert raw audio embeddings into much stronger retrieval features without fine-tuning the language model, with ImageBind's Top-1 audio-to-video retrieval on AudioCaps rising from 0.8% to 45.2%.
  • Increasing the visual token budget from $k=15$ to $k=150$ helps all models, but raw embeddings remain ahead of CLIP-aligned ones, so the generation loss cannot be recovered merely by supplying more visual context.
  • Language exposure during pretraining is the main predictor of robustness to forced alignment: CLAP and Whisper lose less generation quality than AudioCLIP, Wav2CLIP, and ImageBind.
  • WhisperCLIP's all-layer averaging yields the best raw generation scores among the tested encoders and a favorable retrieval-generation balance, making it the paper's recommended operating point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If $H(A)=I(A;V)+H(A|V)$ is the right explanation, the same trade-off should appear when other modality pairs are pushed onto a fixed CLIP manifold (depth, thermal, or tactile input), a cheap extension the paper does not run.
  • The fitted linear relation between retrieval gain and generation loss (0.163% generation loss per retrieval percentage point plus an 11.9% baseline) rests on only five encoders; testing more encoders would show whether it is a general exchange rate or a small-sample pattern.
  • WhisperCLIP's layer-averaging trick could in principle be applied to CLAP or ImageBind to move them toward the same balanced operating point; the paper does not test those variants.
  • Because the attention maps show raw embeddings attending to context beyond the sound source, a focused evaluation on AV E-2 clips with invisible sound sources would reveal whether preserving $H(A|V)$ matters most precisely when the sound source cannot be seen.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper asks whether replacing LLaVA's visual [CLS] token with an audio token, while retaining a small set of visual patch tokens, can support both cross-modal retrieval and text generation. The authors introduce AVE-2, a large dataset of 580K clips with continuous five-dimensional audio-visual alignment scores; use these scores to train 'superaligned' projections; and compare two audio-token strategies: a 1.9M-parameter MLP that projects audio embeddings into CLIP's visual space, and a raw padded/truncated embedding. Across five encoders (Whisper, CLAP, AudioCLIP, Wav2CLIP, ImageBind) and four datasets, they find that projection dramatically improves retrieval but degrades caption quality, while raw embeddings preserve richer descriptions but weaker retrieval. They further propose WhisperCLIP, a layer-averaged Whisper variant, and report a retrieval-generation trade-off curve. The central claim is that image-centric encoders favor retrieval and text-centric encoders favor generation, with the trade-off governed by the preservation or loss of audio-specific information under alignment.

Significance. If the trade-off claim is valid, the paper provides useful practical guidance for audio-conditioned vision-language systems: image-centric encoders for retrieval, text-centric encoders for generation. The work's strengths include a large new continuous-alignment dataset, a systematic comparison of five encoders on four datasets, a simple and reproducible token-substitution framework, and several careful ablations of the WhisperCLIP projection (loss functions, pooling, backbone size). The paper also makes an explicit falsifiable prediction about encoder families. However, the central identification is weakened by a confound between the projection mechanism and alignment itself, and the information-theoretic framing is internally inconsistent. The external datasets provide independent grounding, but the quantitative trade-off regression is based on only five points. These issues are load-bearing for the main claim and require additional experiments or a more cautious framing.

major comments (4)
  1. [Methodology, Alignment Strategies; Experiments] The central raw-vs-projected comparison is confounded. The projected token is produced by a 1.9M-parameter MLP trained with InfoNCE on AVE-2 'superaligned' pairs, while the raw token is obtained by padding or truncating the original embedding. These conditions differ not only in alignment to the CLIP visual manifold but also in being a learned nonlinear transformation, an information bottleneck of 1.9M parameters, having training/test distribution overlap with AVE-2, and producing a token that lies near the CLIP distribution expected by LLaVA's visual projector, whereas the raw token is out-of-distribution. The observed caption-quality drop could therefore be caused by MLP compression or distribution shift rather than by alignment per se. Please add control conditions, such as a random or untrained MLP of the same size, a linear projection, or a projection trained on shuffled pairs, to attribute the change to alignment.
  2. [Information–Theoretic Perspective, Eq. (3)] The text first states that 'Any deterministic projection f(A) ... cannot increase I(A;V)' and then repeatedly attributes retrieval gains to 'maximising I(A;V) via projection.' These statements are inconsistent: by the data processing inequality, I(f(A);V) ≤ I(A;V), so a projection cannot increase mutual information with V relative to the raw audio embedding. The retrieval improvements in Table 3 must be explained by a change of metric space or by the contrastive training objective, not by an increase in I(A;V). Please reconcile the theoretical framing with the experiments, either by redefining the claimed mechanism or by measuring mutual information estimates.
  3. [Retrieval–Generation Trade-off Analysis, Fig. 4] The claimed positive correlation (r≈0.45) and the regression line y=0.163x+11.867 are computed from only five encoder-level points. With n=5, r=0.45 is not statistically distinguishable from zero, and no confidence intervals or p-values are reported. The subsequent statement that Wav2CLIP has the most favorable trade-off ratio (0.64% generation loss per retrieval point) and CLAP the highest cost (3.57% per point) is not derivable from the presented regression. Please either report a proper uncertainty analysis with per-encoder error bars and significance testing, or present the trade-off as a qualitative observation.
  4. [AudioVisual Event Evaluation (AVE-2) Dataset; Experimental Setup] AVE-2 is the primary benchmark, yet the projection MLP is trained on AVE-2 'superaligned' pairs, making the AVE-2 retrieval results partially in-distribution. The external datasets (AudioCaps, TVSum, Ego4D) mitigate this, but AudioCaps is itself derived from AudioSet, from which AVE-2 is also drawn. Please report the overlap between the AVE-2 training pairs and the AudioCaps evaluation clips, and show whether the raw→projected retrieval and generation trends hold on the external datasets alone, without AVE-2.
minor comments (5)
  1. [Tables 4 and 7] Whisper appears in Table 7 but not in Table 4; please clarify whether Whisper was evaluated in the generation experiments and why it is omitted from Table 4.
  2. [Experiments, Generation Quality] The text says BLEU scores fall by roughly 20–25% on average; from Table 4 the declines are approximately 11–24%, averaging about 18%, so the stated range should be corrected.
  3. [Table 3 caption] The caption mentions red and green values, but the typeset table uses arrows (raw→projected); please make the color/arrow convention consistent and accessible.
  4. [Throughout] There are several formatting artifacts, including 'LLaV A' in the title and abstract and 'V osoughi' in the author byline and references; these should be fixed.
  5. [AudioVisual Event Evaluation (AVE-2) Dataset] The AVVA-curation annotation pipeline is referenced only via a URL and a separate paper; the manuscript should describe the annotation protocol in enough detail to be self-contained, including inter-annotator agreement or validation of the five-dimensional scores.

Circularity Check

2 steps flagged · score 6.0 of 10

Retrieval-side 'prediction' is built into the projection loss; generation-side results and external datasets keep the trade-off claim partially independent.

  1. fitted input called prediction [Alignment Strategies / Cross-Modal Retrieval Experiments (Table 3, Table 8)]
    "We train a small multilayer perceptron M to map ha into the CLIP visual embedding space. ... Across all datasets, projecting the audio features into CLIP’s space yields large improvements in retrieval accuracy."

    The projection is trained with an InfoNCE loss whose objective is similarity to CLIP visual features (Table 8: 'InfoNCE' gives the lowest mean retrieval rank), and retrieval is evaluated as Top-1/3/10 cosine-similarity accuracy in that same CLIP space (Table 3). The 'dramatic improvement' from raw to projected is therefore the training objective itself, renamed as an experimental finding, not an independent prediction. Eq. (3) cannot license this: for a deterministic f, I(f(A);V) ≤ I(A;V), so the paper's claim that projection 'maximises I(A;V)' contradicts its own information-theoretic inequality. The generation drop is additionally confounded by the 1.9M-parameter bottleneck, so the retrieval-generation trade-off is only partially identified.

  2. self citation load bearing [AudioVisual Event Evaluation (AVE-2) Dataset; Experimental Setup]
    "we leveraged a five-dimensional scoring system that gave each video segment a detailed quality profile as described in [https://avva-curation.github.io/AVVA-curation/] (Vosoughi, Emmanouilidou, and Gamper 2025) ... AVE-2, our primary benchmark containing 580,145 audio–visual clips with fine-grained alignment annotations."

    The dataset's five-dimensional alignment scores, which define both the 'superaligned' training pairs and the primary benchmark, are inherited from the authors' own AVVA-curation pipeline (first author overlaps with this paper). The manuscript provides descriptive statistics but no independent human or external validation of these scores, so the AVE-2-based evidence for the alignment hypothesis rests on a self-citation chain. This is mitigated by AudioCaps, TVSum, and Ego4D evaluations; it is a structural dependence rather than a fully external check.

full rationale

The entropy decomposition in Eq. (3) is a standard identity and is not itself circular; the empirical measurements of caption quality (Tables 4-7) and the external benchmarks provide independent content. However, the central retrieval-side result is structurally forced: the projection MLP is trained with InfoNCE/distribution-matching against CLIP features, and the retrieval tables measure similarity in that same CLIP space. Calling the resulting gain a 'previously unknown trade-off' overstates what is, on the retrieval side, a direct evaluation of the training loss. The AVE-2 scores are also produced by an overlapping-author pipeline, making the primary benchmark partially in-distribution. Because the generation-side degradation and the external-dataset results are genuine, independent measurements, the paper is partially circular, not wholly so. Score 6.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests less on derived mathematics than on the experimental setup: the entropy chain rule is standard, but the mapping from that identity to the measured trade-off requires treating retrieval as mutual information, treating AVE-2 scores as truth, and treating the MLP projection as a pure alignment operation.

free parameters (2)
  • visual token budget k = 15 and 150
    Chosen based on empirical study to obtain the minimum critical number of tokens; the headline generation results depend on this hand-picked budget.
  • trade-off regression slope and intercept = 0.163 and 11.867
    Least-squares line through five encoder points with r=0.45, used to quantify generation loss per retrieval point.
assumptions (5)
  • standard math Chain rule for entropy H(A)=I(A;V)+H(A|V)
    Used in the Information-Theoretic Perspective to motivate the trade-off between shared and modality-specific information.
  • standard math Data processing inequality: a deterministic projection cannot increase I(A;V)
    Implicit in the claim that projection cannot increase shared information between audio and vision.
  • domain assumption Retrieval accuracy is a valid proxy for I(A;V) in CLIP space
    The projection is trained with InfoNCE and retrieval is used as the measure of alignment, conflating the learned objective with mutual information.
  • domain assumption AVE-2 five-dimensional scores are trustworthy ground truth for alignment quality
    Scores come from three LLM-based models in AVVA-curation; no human validation or inter-annotator agreement is reported.
  • domain assumption Replacing LLaVA's [CLS] token with an audio token and keeping k visual patches is a valid conditioning setup
    This framework is introduced ad hoc for the paper, and k is chosen empirically to avoid model collapse.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Sound Replace Vision in LLaVA With Token Substitution?." pith.science (2026). https://pith.science/paper/B6R6DHLB

@misc{pith2026250610416,
  author       = {Pith},
  title        = {Pith review of: Can Sound Replace Vision in LLaVA With Token Substitution?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6R6DHLB}},
  note         = {Machine review of arXiv:2506.10416}
}
read the original abstract

What happens when we push audio-visual alignment to its absolute limits? To systematically investigate this question, we needed datasets with granular alignment quality annotations, but existing datasets treat alignment as binary, either synchronized or not. To address this limitation, we developed a comprehensive dataset featuring detailed alignment scores that reveal the hidden spectrum of audio-visual perceptual correspondence. Using these precise scores, we create "superaligned" representations by training exclusively on the most perfectly matched audio-visual pairs, then conduct our systematic investigation into how this extreme alignment transforms perceptual model behavior across retrieval and generation tasks. The encoders under study fall into two main groups consisting of image-centric encoders that were pretrained using visual modalities as intermediary hubs for connecting modalities, and text-centric encoders that were pretrained with direct audio-language alignment. We first measure the baseline performance of these encoders on two key tasks, namely cross-modal retrieval and text description generation in vision-language models. Subsequently, we realign all encoders with the CLIP space using highly coherent audio-visual data and observe the performance changes. Our findings reveal that the initial architectural type of the encoder determines how it responds to the alignment process. Image-centric encoders, which are inherently designed for alignment, demonstrate exceptional performance in cross-modal retrieval, but this intensive alignment causes compression of unique linguistic information and reduces the quality of their text description generation in vision-language models. In contrast, text-centric encoders, which possess stronger linguistic authenticity, are able to maintain a better balance between the two objectives.

Figures

Figures reproduced from arXiv: 2506.10416 by the authors.

Figure 1
Figure 1. Audio encoders fall into two distinct camps: image [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SoundCLIP Architecture. The pipeline consists of: (1) encoding audio with a frozen encoder, (2) optionally projecting this embedding into CLIP’s visual space with a lightweight MLP (1.9M parameters) or preserving the raw embedding, and (3) replacing the visual [CLS] token with the audio token while selecting k visual tokens according to cosine similarity. Alignment Strategies Our framework considers two ways of cons… view at source ↗
Figure 3
Figure 3. Comparison of audio-guided patch selection using [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The fundamental retrieval-generation trade-off. Positive correlation (r ≈ 0.45) between retrieval improve￾ment and generation quality loss across all encoders. ment with the CLIP visual space enhances retrieval capabil￾ities, it compromises the quality of generation. L…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 53 canonical work pages

  1. [1]

    Deep variational information bottleneck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016

    arXiv 2016

  2. [2]

    Arandjelovic and P

    R. Arandjelovic and P. Favaro. Object-sounds self-supervision for unlabeled video data. In IEEE Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  3. [3]

    Eagle: Egocentric aggregated language-video engine

    Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, et al. Eagle: Egocentric aggregated language-video engine. In ACM Int. Con. Multimedia

    work page

  4. [4]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In Int. Con. Acoustics, Speech, and Sig. Process., pages 721--725. IEEE, 2020

    work page 2020

  5. [5]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In Int. Con. Acoustics, Speech, and Sig. Process., pages 1--5. IEEE, 2023

    work page 2023

  6. [6]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In IEEE Con. Comput. Vis. Pattern Recog., pages 15180--15190, 2023

    work page 2023

  7. [7]

    Audioset, 2017

    Google Research . Audioset, 2017. Available online: https://research.google.com/audioset/download.html [Accessed: ]

    work page 2017

  8. [8]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    work page 2022

Show all 59 references
  1. [9]

    Audioclip: Extending clip to image, text and audio

    Andrey Guzhov, Federico Raue, J \"o rn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In Int. Con. Acoustics, Speech, and Sig. Process., pages 976--980. IEEE, 2022

    2022

  2. [10]

    chirp" from the

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. In IEEE Con. Comput. Vis. Pattern Recog., pages 13117--13127, 2024

    2024

  3. [11]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  4. [12]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proc. Annual Con. North American Chapter Assoc. for Comput. Linguistics, pages 119--132, 2019

    2019

  5. [13]

    Align before fuse: Vision and language representation learning with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 0 9694--9705, 2021

    2021

  6. [14]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. pages 19730--19742. PMLR, 2023

    2023

  7. [15]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  8. [16]

    Oscar: Object state captioning and state change representation

    Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, et al. Oscar: Object state captioning and state change representation. In Proc. Annual Con. North American Chapter Assoc. for Comput. Linguistics, pages 3565--3576, 2024

    2024

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al. Learning transferable visual models from natural language supervision. pages 8748--8763. PMLR, 2021

    2021

  10. [18]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, et al. Robust speech recognition via large-scale weak supervision. pages 28492--28518. PMLR, 2023

    2023

  11. [19]

    Tvsum: Summarizing web videos using titles

    Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5179--5187, 2015

    2015

  12. [20]

    From vision to audio and beyond: A unified model for audio-visual representation and generation

    Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to audio and beyond: A unified model for audio-visual representation and generation. 2024

    2024

  13. [21]

    Deep learning and the information bottleneck principle

    Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1--5. Ieee, 2015

    2015

  14. [22]

    Learning audio concepts from counterfactual natural language

    Ali Vosoughi, Luca Bondi, Ho-Hsiang Wu, and Chenliang Xu. Learning audio concepts from counterfactual natural language. In Int. Con. Acoustics, Speech, and Sig. Process., pages 366--370. IEEE, 2024

    2024

  15. [23]

    Quality over quantity? LLM -based curation for a data-efficient audio-video foundation model

    Ali Vosoughi, Dimitra Emmanouilidou, and Hannes Gamper. Quality over quantity? LLM -based curation for a data-efficient audio-video foundation model. In Proceedings of the 33rd European Signal Processing Conference (EUSIPCO), 2025. To appear

    2025

  16. [24]

    Wav2clip: Learning robust audio representations from clip

    Ho-Hsiang Wu, Jiaming Xiao, Zekun Xu, Andy Huang, Zhiyao Xie, Eric Lee, and Yuandong Zhuang. Wav2clip: Learning robust audio representations from clip. In ICASSP, 2022

    2022

  17. [25]

    Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving

    Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, and Renaud Marlet. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2023

  18. [26]

    Square Attack: A Query-efficient Black-box Adversarial Attack via Random Search

    Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square Attack: A Query-efficient Black-box Adversarial Attack via Random Search . In Proceedings of the European Conference on Computer Vision, pages 484--501, 2020

    2020

  19. [27]

    Adversarial example games

    Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien, and Will Hamilton. Adversarial example games. Advances in neural information processing systems, 33: 0 8921--8934, 2020

    2020

  20. [28]

    Towards Evaluating the Robustness of Neural Networks

    Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks . In IEEE Symposium on Security and Privacy, pages 39--57, 2017

    2017

  21. [29]

    Boosting Decision-based Black-box Adversarial Attacks with Random Sign Flip

    Weilun Chen, Zhaoxiang Zhang, Xiaolin Hu, and Baoyuan Wu. Boosting Decision-based Black-box Adversarial Attacks with Random Sign Flip . In Proceedings of the European Conference on Computer Vision, pages 276--293, 2020

    2020

  22. [30]

    Boosting Adversarial Attacks with Momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9185--9193, 2018

    2018

  23. [31]

    Evading Defenses to Transferable Adversarial Examples by Translation-invariant Attacks

    Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading Defenses to Transferable Adversarial Examples by Translation-invariant Attacks . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4312--4321, 2019

    2019

  24. [32]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale . In Proceedings of t...

    2020

  25. [33]

    Patch-wise Attack for Fooling Deep Neural Network

    Lianli Gao, Qilong Zhang, Jingkuan Song, Xianglong Liu, and Heng Tao Shen. Patch-wise Attack for Fooling Deep Neural Network . In Proceedings of the European Conference on Computer Vision, pages 307--322, 2020

    2020

  26. [34]

    Explaining and Harnessing Adversarial Examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples . In Proceedings of the International Conference on Learning Representations, 2015

    2015

  27. [35]

    Lgv: Boosting Adversarial Example Transferability from Large Geometric Vicinity

    Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting Adversarial Example Transferability from Large Geometric Vicinity . In Proceedings of the European Conference on Computer Vision, pages 603--618, 2022

    2022

  28. [36]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770--778, 2016

    2016

  29. [37]

    Rethinking spatial dimensions of vision transformers

    Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11936--11945, 2021

    2021

  30. [38]

    Densely Connected Convolutional Networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4700--4708, 2017

    2017

  31. [39]

    Adversarial Examples in the Physical World

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial Examples in the Physical World . In Artificial Intelligence Safety and Security, pages 99--112, 2018

    2018

  32. [40]

    Decision-based Adversarial Attack with Frequency Mixup

    Xiu-Chuan Li, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Decision-based Adversarial Attack with Frequency Mixup . IEEE Transactions on Information Forensics and Security, 17: 0 1038--1052, 2022

    2022

  33. [41]

    Learning Transferable Adversarial Examples via Ghost Networks

    Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning Transferable Adversarial Examples via Ghost Networks . In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11458--11465, 2020

    2020

  34. [42]

    Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks

    Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks . In Proceedings of the International Conference on Learning Representations, 2020

    2020

  35. [43]

    Delving into Transferable Adversarial Examples and Black-box Attacks

    Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks . In Proceedings of the International Conference on Learning Representations, 2017

    2017

  36. [44]

    Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows . In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012--10022, 2021

    2021

  37. [45]

    Frequency Domain Model Augmentation for Adversarial Attack

    Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency Domain Model Augmentation for Adversarial Attack . In Proceedings of the European Conference on Computer Vision, pages 549--566, 2022

    2022

  38. [46]

    Hierarchical vision transformers for disease progression detection in chest x-ray images

    Amarachi B Mbakwe, Lyuyang Wang, Mehdi Moradi, and Ismini Lourentzou. Hierarchical vision transformers for disease progression detection in chest x-ray images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 685--695. Springer, 2023

    2023

  39. [47]

    Deepfool: A Simple and Accurate Method to Fool Deep Neural Networks

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A Simple and Accurate Method to Fool Deep Neural Networks . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2574--2582, 2016

    2016

  40. [48]

    Intriguing Properties of Neural Networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks . arXiv:1312.6199, 2013

    2013 arXiv

  41. [49]

    Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization

    Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang, Lingyi Hong, Yan Wang, and Wenqiang Zhang. Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization . arXiv:2211.11236, 2022 a

    2022 arXiv

  42. [50]

    Enhancing the Transferability of Adversarial Attacks through Variance Tuning

    Xiaosen Wang and Kun He. Enhancing the Transferability of Adversarial Attacks through Variance Tuning . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1924--1933, 2021

    1924

  43. [51]

    Admix: Enhancing the Transferability of Adversarial Attacks

    Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the Transferability of Adversarial Attacks . In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16158--16167, 2021 a

    2021

  44. [52]

    Boosting Adversarial Transferability through Enhanced Momentum

    Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting Adversarial Transferability through Enhanced Momentum . In The British Machine Vision Conference, 2021 b

    2021

  45. [53]

    Triangle Attack: A Query-efficient Decision-based Adversarial Attack

    Xiaosen Wang, Zeliang Zhang, Kangheng Tong, Dihong Gong, Kun He, Zhifeng Li, and Wei Liu. Triangle Attack: A Query-efficient Decision-based Adversarial Attack . In Proceedings of the European Conference on Computer Vision, pages 156--174, 2022 b

    2022

  46. [54]

    Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving Transferability of Adversarial Examples With Input Diversity . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2730--2739, 2019

    2019

  47. [55]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Girshick Ross, Dollár Piotr, Tu Zhuowen, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1492--1500, 2017

    2017

  48. [56]

    Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability

    Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14983--14992, 2022

    2022

  49. [57]

    Meta-learning the Search Distribution of Black-box Random Search Based Adversarial Attacks

    Maksym Yatsura, Jan Metzen, and Matthias Hein. Meta-learning the Search Distribution of Black-box Random Search Based Adversarial Attacks . In Proceedings of the Advances in Neural Information Processing Systems, pages 30181--30195, 2021

    2021

  50. [58]

    Learning to transform dynamically for better adversarial transferability

    Rongyi Zhu, Zeliang Zhang, Susan Liang, Zhuo Liu, and Chenliang Xu. Learning to transform dynamically for better adversarial transferability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24273--24283, 2024

    2024

  51. [59]

    Uia-vit: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection

    Wanyi Zhuang, Qi Chu, Zhentao Tan, Qiankun Liu, Haojie Yuan, Changtao Miao, Zixiang Luo, and Nenghai Yu. Uia-vit: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection. In European conference on computer vision, pages 391--407. Springer, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.