arxiv: 2604.08147 · v1 · submitted 2026-04-09 · 💻 cs.SD · cs.CV

Recognition: 2 theorem links

· Lean Theorem

Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

Linge Wang , Yingying Chen , Bingke Zhu , Lu Zhou , Jinqiao Wang

Authors on Pith no claims yet

Pith reviewed 2026-05-10 17:42 UTC · model grok-4.3

classification 💻 cs.SD cs.CV

keywords audio-visual representation learningsemantic noise reductionmasked reconstructioncontrastive alignmentteacher-guided learningzero-shot retrievaldual-path frameworkmultimodal pretraining

0 comments

The pith

Decoupling contrastive alignment from masked reconstruction into separate paths with teacher guidance reduces semantic noise in audio-visual learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Recent audio-visual models combine contrastive alignment with masked reconstruction but suffer interference when both run in one pass. The contrastive branch ends up using visibility patterns meant for reconstruction, which adds semantic noise and hurts cross-modal alignment. TG-DP fixes this by running the two objectives on separate paths with different masking and by letting a teacher model guide how visible tokens are organized in the alignment path. A sympathetic reader would care because this shows a simple architectural change can boost zero-shot retrieval without sacrificing other performance metrics, pointing to objective decoupling as a general lever for better multimodal learning.

Core claim

The central discovery is that jointly optimizing contrastive and reconstruction objectives in a single pass introduces semantic noise into the contrastive branch due to mismatched visibility patterns, and that disentangling these into a Teacher-Guided Dual-Path framework, where the contrastive path uses alignment-suited masking guided by a teacher on visible tokens, yields improved cross-modal representations as evidenced by higher zero-shot retrieval scores.

What carries the argument

Teacher-Guided Dual-Path framework that decouples the masking regimes for reconstruction and contrastive branches and provides auxiliary guidance from a teacher model on visible tokens for alignment.

Load-bearing premise

The assumption that semantic noise primarily stems from the contrastive branch inheriting reconstruction-oriented random patches, and that disentangling masking plus teacher guidance on visible tokens will reduce this without causing new instabilities.

What would settle it

Training a single-path baseline that applies the same teacher guidance on visible tokens but retains shared masking across objectives, then comparing its zero-shot R@1 retrieval scores on AudioSet to those of the dual-path model, would test whether path decoupling is required.

Figures

Figures reproduced from arXiv: 2604.08147 by Bingke Zhu, Jinqiao Wang, Linge Wang, Lu Zhou, Yingying Chen.

**Figure 2.** Figure 2: Overall pipeline of our proposed framework. The model consists of two objective-specific forward passes: (1) the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

read the original abstract

Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2\% to 37.4\% for video-to-audio retrieval and from 27.9\% to 37.1\% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

TG-DP decouples masking in audio-visual SSL via dual paths and teacher guidance, yielding modest retrieval gains whose mechanism needs tighter isolation.

read the letter

The main thing here is that TG-DP splits reconstruction and contrastive alignment into separate paths with different masking regimes, then adds a teacher to organize visible tokens in the contrastive branch so it does not inherit reconstruction-style random patches. This directly targets the interference the authors describe in joint optimization of the two objectives. The design choice is new in its specific combination of dual-path decoupling plus teacher-guided token structure for the alignment side. The paper does a reasonable job spelling out the problem and showing how the split plus guidance can reduce semantic noise without major new instabilities. The numbers on AudioSet are concrete: video-to-audio R@1 moves from 35.2% to 37.4% and audio-to-video from 27.9% to 37.1%, with SOTA linear-probe results on AS20K and VGGSound. Code release helps anyone who wants to check or extend it. The soft spot is attribution. The dual-path architecture adds capacity and compute, so the reported lifts could stem from the extra path rather than the teacher-guided disentanglement specifically. The abstract gives no ablations that remove the teacher while keeping separate masking, or that match total resources in a single-path baseline, which leaves the load-bearing claim plausible but not fully pinned down. The stress-test note is fair on this point. This work is for researchers already doing audio-visual self-supervised pretraining who hit objective conflicts. A reader looking for an engineering tweak to combine reconstruction and alignment would find usable ideas. It deserves serious peer review because the idea is clear, the benchmarks are standard, and the empirical claims are modest rather than overstated. I would send it to referees and ask for the missing controls on the teacher's isolated contribution.

Referee Report

2 major / 2 minor

Summary. The paper proposes the Teacher-Guided Dual-Path (TG-DP) framework for audio-visual representation learning. It decouples masked reconstruction and contrastive alignment into separate optimization paths with disentangled masking regimes, using a teacher model to provide auxiliary guidance on visible tokens in the contrastive branch to reduce semantic noise and interference. The work reports state-of-the-art zero-shot retrieval on AudioSet (R@1 improved from 35.2% to 37.4% video-to-audio and 27.9% to 37.1% audio-to-video) along with SOTA linear-probe accuracy on AS20K and VGGSound, with code released.

Significance. If the gains are shown to stem specifically from the disentangled masking plus teacher guidance rather than added capacity, the approach would provide a concrete way to reduce optimization conflicts between reconstruction and alignment objectives in large-scale multimodal pretraining. The explicit code release is a positive for reproducibility.

major comments (2)

[Experimental results] Experimental results section: No ablation is reported that retains the dual-path architecture and separate masking regimes but removes the teacher guidance on visible tokens. This is load-bearing for the central claim that teacher guidance specifically reduces semantic noise in the contrastive branch; without it, the R@1 gains on AudioSet could be explained by increased parameters alone.
[Method] Method section (TG-DP framework description): The paper does not quantify or bound the additional optimization instabilities or overfitting risk introduced by the teacher guidance signals, leaving the assumption that disentangling masking plus teacher guidance will reliably stabilize cross-modal alignment untested against the skeptic concern.

minor comments (2)

[Abstract] Abstract: The baseline numbers (35.2%, 27.9%) are given without naming the prior method or citing its paper, making it harder to assess the magnitude of improvement.
[Experimental results] The linear-probe results on AS20K and VGGSound are stated as SOTA but without reporting the exact accuracy numbers or the competing methods in the same table as the retrieval results.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback and the recommendation for major revision. The points raised help clarify the contribution of the teacher guidance component. We address each major comment point by point below, with planned revisions to the manuscript.

read point-by-point responses

Referee: [Experimental results] Experimental results section: No ablation is reported that retains the dual-path architecture and separate masking regimes but removes the teacher guidance on visible tokens. This is load-bearing for the central claim that teacher guidance specifically reduces semantic noise in the contrastive branch; without it, the R@1 gains on AudioSet could be explained by increased parameters alone.

Authors: We agree that isolating the contribution of the teacher guidance is essential to support the central claim. The current experiments compare against baselines but do not include a controlled ablation that keeps the dual-path structure and disentangled masking while removing only the teacher guidance on visible tokens. We will add this ablation to the experimental results section, reporting zero-shot retrieval and linear-probe metrics for the variant without teacher guidance. This will allow direct assessment of whether the observed gains (e.g., R@1 improvements on AudioSet) stem specifically from the teacher signals rather than added capacity or the dual-path design alone. revision: yes
Referee: [Method] Method section (TG-DP framework description): The paper does not quantify or bound the additional optimization instabilities or overfitting risk introduced by the teacher guidance signals, leaving the assumption that disentangling masking plus teacher guidance will reliably stabilize cross-modal alignment untested against the skeptic concern.

Authors: We acknowledge that the manuscript provides no explicit quantification, theoretical bound, or formal analysis of potential optimization instabilities or overfitting risks arising from the teacher guidance signals. In practice, our training runs across multiple seeds exhibited stable convergence and consistent performance gains without signs of divergence or overfitting, which empirically supports the stabilizing effect under the reported hyperparameters. To address the concern, we will expand the method section with a discussion of observed training dynamics, including references to loss curves and hyperparameter sensitivity where feasible. A rigorous theoretical characterization of the optimization landscape remains outside the scope of this work. revision: partial

standing simulated objections not resolved

A formal theoretical quantification or bound on the additional optimization instabilities or overfitting risks introduced by the teacher guidance signals

Circularity Check

0 steps flagged

No circularity; empirical claims rest on benchmark results

full rationale

The paper introduces TG-DP, a dual-path architecture that decouples masking regimes and adds teacher guidance for audio-visual pretraining. All central claims are supported by reported empirical metrics (R@1 improvements on AudioSet zero-shot retrieval, linear-probe SOTA on AS20K/VGGSound) rather than any derivation, equation, or self-citation chain that reduces to its own inputs by construction. No self-definitional steps, fitted inputs renamed as predictions, or load-bearing self-citations appear in the provided text. The work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 1 invented entities

The central claim depends on the design choice of separate masking regimes and teacher guidance, which function as free parameters, plus the domain assumption that mismatched visibility is the dominant source of semantic noise.

free parameters (2)

masking visibility patterns per path
Different hiding strategies are chosen for the reconstruction and alignment branches; these are design decisions that must be selected or tuned.
teacher guidance mechanism details
How the teacher organizes visible tokens is a new component whose exact implementation and hyperparameters are not specified in the abstract.

axioms (1)

domain assumption Joint optimization of reconstruction and contrastive objectives introduces semantic noise because reconstruction masking is suboptimal for alignment
This premise is stated directly in the abstract as the motivation for decoupling.

invented entities (1)

Teacher-Guided Dual-Path (TG-DP) framework no independent evidence
purpose: To decouple reconstruction and alignment into separate optimization paths with auxiliary teacher guidance
New architecture introduced in this work; no independent evidence outside the paper's reported results.

pith-pipeline@v0.9.0 · 5558 in / 1482 out tokens · 143551 ms · 2026-05-10T17:42:39.575667+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths... teacher model further provides auxiliary guidance for organizing visible tokens
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

By disentangling the masking regimes of the two branches... lower masking ratio (50%)... teacher-guided masking strategy

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

37 extracted references · 7 canonical work pages · 4 internal anchors

[1]

Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.Advances in neural information processing systems, 34:24206–24221, 2021

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.Advances in neural information processing systems, 34:24206–24221, 2021. 1

2021
[2]

Self- supervised multimodal versatile networks.Advances in neu- ral information processing systems, 33:25–37, 2020

Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovi´c, Jason Ramapuram, Jeffrey De Fauw, Lu- cas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks.Advances in neu- ral information processing systems, 33:25–37, 2020. 1, 3

2020
[3]

Look, listen and learn

Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InProceedings of the IEEE international conference on computer vision, pages 609–617, 2017. 1, 2

2017
[4]

Objects that sound

Relja Arandjelovic and Andrew Zisserman. Objects that sound. InProceedings of the European conference on com- puter vision (ECCV), pages 435–451, 2018. 2

2018
[5]

Cav-mae sync: Improving contrastive audio-visual mask au- toencoders via fine-grained alignment

Edson Araujo, Andrew Rouditchenko, Yuan Gong, Saurab- hchand Bhati, Samuel Thomas, Brian Kingsbury, Leonid Karlinsky, Rogerio Feris, James R Glass, and Hilde Kuehne. Cav-mae sync: Improving contrastive audio-visual mask au- toencoders via fine-grained alignment. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18794–18803, 20...

2025
[6]

Sound- net: Learning sound representations from unlabeled video

Yusuf Aytar, Carl V ondrick, and Antonio Torralba. Sound- net: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29,
[7]

Data2vec: A general frame- work for self-supervised learning in speech, vision and lan- guage

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general frame- work for self-supervised learning in speech, vision and lan- guage. InInternational conference on machine learning, pages 1298–1312. PMLR, 2022. 3

2022
[8]

Multimodal machine learning: A survey and tax- onomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018

Tadas Baltru ˇsaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and tax- onomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018. 1

2018
[9]

Emerg- ing properties in self-supervised vision transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 3

2021
[10]

Vggsound: A large-scale audio-visual dataset

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020. 2, 5

2020
[11]

Localizing visual sounds the hard way

Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Na- grani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16867–16876, 2021. 2

2021
[12]

Exploring simple siamese rep- resentation learning

Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 15750–15758, 2021. 3

2021
[13]

Audio set: An ontology and human- labeled dataset for audio events

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In2017 IEEE interna- tional conference on acoustics, speech and signal processing (ICASSP), pages 776–780. IEEE, 2017. 2, 5

2017
[14]

Audiovisual masked autoencoders

Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. Audiovisual masked autoencoders. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16144–16154, 2023. 1, 2

2023
[15]

Imagebind: One embedding space to bind them all

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15180–15190, 2023. 2, 6

2023
[16]

Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive audio-visual masked autoencoder. InThe Eleventh International Conference on Learning Representa- tions, 2023. 1, 2, 5, 6

2023
[17]

Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020. 3

2020
[18]

Cross- mae: Cross-modality masked autoencoders for region-aware audio-visual pre-training

Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xi- aoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. Cross- mae: Cross-modality masked autoencoders for region-aware audio-visual pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26721–26731, 2024. 1, 2

2024
[19]

Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language

Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 13117–13127, 2024. 2, 6

2024
[20]

Masked autoencoders are scalable vision learners

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 1

2022
[21]

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015. 3

work page internal anchor Pith review Pith/arXiv arXiv 2015
[22]

Mavil: Masked audio-video learners.Advances in Neural Information Processing Sys- tems, 36:20371–20393, 2023

Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al. Mavil: Masked audio-video learners.Advances in Neural Information Processing Sys- tems, 36:20371–20393, 2023. 1, 2, 6

2023
[23]

Equiav: Leveraging equivari- ance for audio-visual contrastive learning.arXiv preprint arXiv:2403.09502, 2024

Jongsuk Kim, Hyeongkeun Lee, Kyeongha Rho, Junmo Kim, and Joon Son Chung. Equiav: Leveraging equivari- ance for audio-visual contrastive learning.arXiv preprint arXiv:2403.09502, 2024. 2

work page arXiv 2024
[24]

Siamese vision transform- ers are scalable audio-visual learners

Yan-Bo Lin and Gedas Bertasius. Siamese vision transform- ers are scalable audio-visual learners. InEuropean Confer- ence on Computer Vision, pages 303–321. Springer, 2024. 2, 6

2024
[25]

Audio- visual instance discrimination with cross-modal agreement

Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio- visual instance discrimination with cross-modal agreement. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12475–12486, 2021. 2

2021
[26]

Multimodal deep learn- ing

Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learn- ing. InICML, pages 689–696, 2011. 1

2011
[27]

Representation Learning with Contrastive Predictive Coding

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 1

work page internal anchor Pith review Pith/arXiv arXiv 2018
[28]

Audio-visual scene analysis with self-supervised multisensory features

Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. InPro- ceedings of the European conference on computer vision (ECCV), pages 631–648, 2018. 1

2018
[29]

Ambient sound provides supervision for visual learning

Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. InEuropean conference on computer vision, pages 801–816. Springer, 2016. 2

2016
[30]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 3

2021
[31]

FitNets: Hints for Thin Deep Nets

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets.CoRR, abs/1412.6550, 2014. 3

work page internal anchor Pith review arXiv 2014
[32]

Multimodal learning with deep boltzmann machines

Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann machines. InAdvances in neu- ral information processing systems, 2012. 1

2012
[33]

From vision to audio and beyond: A unified model for audio-visual repre- sentation and generation.arXiv preprint arXiv:2409.19132,

Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to audio and beyond: A unified model for audio-visual repre- sentation and generation.arXiv preprint arXiv:2409.19132,

work page arXiv
[34]

Multimodal transformer for unaligned multimodal language sequences

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the 57th annual meeting of the association for computational linguistics, pages 6558–6569,
[35]

The sound of pixels

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. InProceedings of the European conference on com- puter vision (ECCV), pages 570–586, 2018. 2

2018
[36]

iBOT: Image BERT Pre-Training with Online Tokenizer

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832,

work page internal anchor Pith review arXiv
[37]

Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretrain- ing to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. 2, 6

work page arXiv 2023