Pith. sign in

REVIEW 2 major objections 5 minor 59 references

SceneBind: Binding What and Where Across Vision, Audio and Language

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SceneBind claims that adding object-centric semantic-spatial slots to a global scene embedding improves cross-modal scene retrieval, spatial retrieval, and zero-shot audio-visual localization over global-semantics-only embeddings.

desk verdict Solid system paper with a genuinely useful object-centric spatial layer, but the headline audio-spatial claims rest on Gemini-derived labels that the paper itself admits are not binaurally grounded. read the letter →

arxiv 2607.15265 v1 pith:5S4CVV7Z submitted 2026-07-16 cs.CV cs.AIcs.MMcs.SD

Mingfei Chen , Zijun Cui , Ruoke Zhang , Hyeonggon Ryu , Eli Shlizerman This is my paper
classification cs.CVcs.AIcs.MMcs.SD
keywords omni-modalrepresentationsemantic-spatialslotsbinauralaudiocross-modalretrievalobjectgroundingaudio-visuallocalizationspatialzero-shottransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SceneBind claims that a scene is best represented not by a single global embedding but by a global embedding together with a small set of object-centric 'semantic-spatial slots,' each carrying what an object is and where it is (azimuth, elevation, distance) with a confidence. The paper shows this representation can be trained on top of frozen pretrained vision, audio, and text encoders with a few lightweight modules, supervised by bipartite matching to object-level text clauses and by contrastive alignment. On its curated real-world binaural benchmark, this what-and-where binding beats global-semantics-only embeddings on cross-modal scene retrieval, spatial retrieval, and object grounding, and transfers zero-shot to egocentric audio-visual localization. The upshot, if true, is that spatial structure is not a separate task but a cheap, learnable layer on existing semantic representations.

What carries the argument

The load-bearing object is the semantic-spatial slot, a tuple (s_k, r_k, c_k) where s_k is a semantic embedding decoded by a cross-attention module, r_k are predicted azimuth, elevation, and distance distributions over discretized bins, and c_k is an activity confidence. The spatial decoder uses each semantic slot as a query to attend to visual patch tokens or fused binaural audio context (semantic tokens plus a four-channel magnitude/phase spatial encoding). At inference, SceneBind Matching gates slots by confidence, matches each query slot to its best candidate slot under a semantic similarity times spatial-distribution-agreement score weighted by geometric-mean confidence, and adds this o

What would settle it

Re-evaluate audio-only object grounding on a held-out set whose spatial labels are produced by a physically calibrated binaural measurement or a head-tracked human listener; if per-attribute audio accuracy collapses to near chance on those labels, the claim that audio slots learn genuine spatial structure is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that object-centric semantic-spatial slots are what carry the 'where' signal that global embeddings miss. The ablation isolates the causal role: adding slot matching raises spatial retrieval from 25.4 to 38.4 and text-conditioned scene retrieval from 34.2 to 39.9, while removing object semantic supervision drops grounding accuracy from 19.6 to 8.8. SceneBind trains these slots by bipartite matching predicted slots to ground-truth object clauses, combining global InfoNCE alignment, Gaussian-smoothed cross-entropy over discretized azimuth/elevation/distance bins, and intra-scene plus cross-scene slot contrastive losses. With about 10–25 slots (audio b

Load-bearing premise

The load-bearing premise is that the spatial labels used to supervise and evaluate the model are reliable; in particular, audio-only events receive direction and distance labels from a vision-language annotator that does not directly perceive binaural acoustic cues, so a systematic bias toward visible or front-center events would inflate the reported audio spatial competence.

Editorial extensions

If this is right

  • Scene retrieval across audio–image, audio–text, and image–text improves when object-level spatial agreement is combined with global semantic similarity, with the largest gains on text-conditioned queries.
  • Semantically identical scenes that differ only in spatial layout (e.g., 360-degree viewpoint variants) become separable, which global embeddings cannot do.
  • Text queries can ground objects to per-attribute spatial predictions from either audio or vision, with audio contributing strong elevation cues and vision stronger azimuth/distance cues.
  • A low-cost spatial layer on frozen pretrained encoders suffices for state-of-the-art retrieval and zero-shot transfer, so this design can sit on top of any semantic encoder.
  • Zero-shot egocentric audio-visual localization benefits directly from the slot representation, outperforming prior finetuned approaches without task-specific training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the slot formulation is right, extending slots along time should let them carry motion trajectories instead of static positions, directly addressing the paper's noted limitation of short windows.
  • A testable next step is physically calibrating the audio spatial labels; the paper's own annotation caveat suggests audio accuracy may currently be vision-inflated, and cleaner labels could change the audio–vision balance.
  • The slot structure with confidence could serve as a grounded query language for spatial LLM reasoning: a language model that consumes global embedding plus slots gets an explicit coordinate system for 'where is X' questions.
  • One stress test implied by the method: measure retrieval performance as a function of object overlap between query and candidate scenes; best-match slot scoring should degrade more gracefully than global-only matching when scenes share no objects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces SceneBind, an omni-modal scene representation that combines a global semantic embedding with K object-centric semantic-spatial slots, each carrying semantic features, predicted azimuth/elevation/distance, and a confidence score. The representation is extracted from frozen SigLIP2 and M2D-CLAP encoders, with lightweight trainable spatial decoders and matching modules. The authors curate a binaural audio-visual dataset with Gemini-generated semantic-spatial annotations, propose a bipartite-matching training objective, and evaluate cross-modal retrieval, spatial retrieval, object grounding, and zero-shot egocentric audio-visual localization. The central claims are that object-centric slots add spatial structure beyond global semantic embeddings and that SceneBind achieves state-of-the-art cross-modal scene and spatial retrieval while transferring zero-shot.

Significance. If the results hold, SceneBind is a useful and relatively inexpensive way to add object-level spatial grounding to existing frozen encoders, and the consistent ablations (Tables 3-5) support the causal role of object slots and the training objectives. The per-attribute grounding accuracies in Table 2 (e.g., 83.2% audio elevation, 66.1% visual distance) are far above the 0.7% all-attributes chance level and suggest the model learns nontrivial spatial associations. The two-stage training recipe and the zero-shot AV-localization results are also positive contributions. However, the audio-specific claims are weakened by the fact that the audio spatial ground-truth labels are generated by Gemini from visual evidence rather than from binaural cues, as the paper itself acknowledges in Appendix G. Because audio spatial performance is a headline result and a large fraction of the benchmark queries are audio-relevant, this data-loop concern must be resolved before the audio component of the contribution can be considered established.

major comments (2)
  1. [Appendix B.2 / B.3 / G; Sec. 3.3] The audio spatial supervision and evaluation labels come from the Gemini annotation pipeline, which Appendix G states 'does not directly perceive binaural spatial cues' and whose 'spatial estimates are inferred mainly from visual evidence.' For audio_only events (Fig. 8) there is by definition no visible object, so their azimuth/elevation/distance labels are LLM guesses conditioned on audio and visual context rather than on inter-channel binaural information. These same labels supervise the audio spatial decoder through the object grounding loss in Sec. 3.3 and define the ground truth for audio-relevant queries: 745 of 1,527 spatial-retrieval queries are audio-relevant (B.4), and Table 2 reports audio elevation accuracy 83.2%. A systematic bias in Gemini's guesses -- e.g., a front/center prior or a tendency to align with visible objects -- would inflate the reported audio spatial compete
  2. [Sec. 3.2 / Sec. 3.3; Table 2] The paper claims that audio spatial understanding comes from binaural cues, and it adds a four-channel time-frequency spatial encoder to the monaural semantic audio tokens (Sec. 3.2). Yet there is no ablation that removes the binaural spatial encoder or replaces the binaural input with mono audio. Given that the training labels are themselves vision-derived (Appendix G), the audio spatial decoder might be learning to reproduce Gemini's semantic/positional priors rather than exploiting inter-channel level and phase differences. The ablations in Tables 3-5 vary matching and losses but never vary the audio input evidence. An experiment comparing SceneBind's audio branch with and without the binaural spatial features is needed to support the specific claim that the representation captures true spatial audio cues.
minor comments (5)
  1. [Table 6] The column layout is hard to parse: the header 'Audio–Text Visual–Text A V VT AT Avg' mixes task names and metric abbreviations. Please separate the three retrieval directions and the metric columns explicitly.
  2. [Sec. 1 / Sec. 3.2] The abstract says 'a few additional tokens,' but the model uses K=50 object queries per modality (Appendix C.1). Clarify that the overhead is small relative to the 256 visual patch tokens, not that the absolute number of tokens is small.
  3. [Appendix B.3] The human review process is described qualitatively, but no inter-reviewer agreement or correction-rate statistics are reported. Even a brief table of reviewer decisions (keep/discard/correct) would help calibrate the reliability of the benchmark labels.
  4. [Sec. 4.4 / Table 1] The sentence 'achieving 65.3 on V↔T (+28% over pretrained, +48% over finetuned)' appears to mix percentage-point and relative-improvement language. Please state the comparison metric explicitly and consistently.
  5. [Appendix G] The main text should cross-reference Appendix G's limitation that Gemini does not perceive binaural cues. As written, the main text presents the audio spatial results without acknowledging this significant caveat.

Circularity Check

1 steps flagged · score 4.0 of 10

Audio spatial claims are entangled with Gemini's vision-derived labels; the core slot representation and matching scheme are otherwise self-contained.

  1. other [Sec. B.2 (Annotation), Sec. 3.3 (spatial supervision), Sec. 4.2/G (evaluation and limitations)]
    "Although Gemini does not directly perceive binaural spatial cues, it provides strong visual recognition... estimate the corresponding camera-coordinate spatial labels from the visual evidence. /// Gemini ... does not directly perceive binaural spatial cues. Its spatial estimates are inferred mainly from visual evidence, which can bias annotations toward visible or front-facing objects and may miss or hallucinate off-screen or rear-field events."

    The audio spatial decoder is trained (Sec. 3.3: L_theta, L_phi, L_d) and evaluated (Table 2, Sec. 4.2) on the same Gemini-generated spatial labels that, by the paper's own admission, were inferred from the visual stream rather than from binaural cues. For audio_only events there is no visible object, so their labels are LLM guesses. Reported audio spatial accuracy (e.g., 83.2% elevation) therefore measures fit to a vision-derived label-generation prior, not independent binaural perception. This is a partial circularity confined to the audio spatial claim; the slot-matching equations and external AVLoc zero-shot evaluation are not circular.

full rationale

The central derivation — global embedding + K semantic-spatial slots (Eq. 1), bipartite matching supervision (Eq. 3), and SceneBind Matching (Eq. 6) — is not a re-derivation of its own output. Tables 3 and 4 are internal ablations, and Table 7 evaluates zero-shot transfer on an external egocentric audio-visual localization benchmark, which provides independent evidence that the representation carries spatial signal. There is no load-bearing uniqueness theorem or ansatz smuggled in via self-citation; the authors' prior work appears only as background context. The one genuine circularity is the audio spatial label loop: the benchmark and training labels for audio events originate from Gemini, which the paper itself states cannot perceive binaural cues and infers spatial estimates mainly from visual evidence, biasing toward visible/front-facing objects. The same labels supervise and score the audio spatial decoder, so the reported audio spatial accuracy is partially a measure of how well the model reproduces the label-generation prior rather than true binaural spatial understanding. Human review of the benchmark does not dissolve this loop because it starts from the same Gemini proposals and reliability for audio_only spatial corrections is not quantified. This warrants a moderate score, not a high one, because the visual/text slot machinery and the external zero-shot evaluation remain independent and non-circular.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claim rests on roughly eight hand-set hyperparameters and four domain assumptions about the reliability of LLM-generated spatial labels and pretrained-encoder alignment; no new physical entities are introduced. The object-centric semantic-spatial slot is a representational construct (DETR-style), not a claimed new constituent of the world.

free parameters (8)
  • Number of object slots K = 50
    Chosen by hand; Fig. 3 shows performance saturates beyond ~10 slots and audio peaks near 5, so 50 is conservative but no sensitivity analysis is reported.
  • Inference weighting λobj = 0.05 (A–V), 0.5 (text-related)
    Test-time weighting of object score vs global score (Sec. C.4), tuned separately per retrieval direction on the benchmark.
  • Bipartite matching coefficients λ, α, β (Eq. 3) = not reported
    Matching score mixes semantic and spatial terms; values are not given in main text or appendix, leaving assignment supervision incompletely specified.
  • Loss weights (Table 9) = 15 weights, e.g., 2.0 for L_av_global and L_conf, 1.0/0.5/0.5 for azimuth/elevation/distance
    Full objective is a weighted sum; no sensitivity analysis or search procedure is reported.
  • Confidence gate and reranking window = 0.05; top-50
    Active-slot threshold and global-prefilter size chosen by hand (Sec. C.4).
  • Spatial smoothing σ = 1.0 azimuth (circular), 0.5 elevation/distance
    Gaussian-smoothed cross-entropy bandwidths chosen without stated justification (Sec. C.2).
  • Verification thresholds and balancing keep rate = ImageBind ≥0.10, CLAP ≥0.05; 50% keep for front-center
    Event-filtering thresholds (Table 8) and spatial-balancing rate (Sec. B.2) shape the dataset and are chosen ad hoc.
  • AV-localization guide-map temperatures = 0.45, 0.60, 0.75; σx=σy=0.13
    Zero-shot localization pipeline parameters (Sec. E) tuned for the downstream cIoU metric.
assumptions (6)
  • domain assumption Binaural waveforms contain recoverable azimuth/elevation/distance cues that a from-scratch conv frontend can extract under frozen mono-trained semantic tokens.
    Sec. 3.2/C.1: spatial encoder trained on 4-channel STFT features; the paper does not quantify binaural-cue SNR in its corpus.
  • domain assumption M2D-CLAP audio patch tokens can be aligned into SigLIP2's vision-text space with a residual MLP and safely frozen afterwards.
    Sec. 3.2/C.1: alignment MLP pretrained on Binaural captions; assumes the two pretrained spaces share sufficient structure.
  • domain assumption Gemini's visual-evidence-based spatial estimates are accurate enough to serve as supervision for both visual and audio spatial attributes.
    Fig. 8 prompt and Sec. B.2; the paper's own Sec. G concedes this is biased toward visible/front-facing objects.
  • domain assumption ImageBind and CLAP verification scores (thresholds ≥0.10/≥0.05) suffice to filter LLM-hallucinated events and correctly downgrade modality types.
    Table 8; no analysis of verification error rates is provided.
  • standard math In-batch InfoNCE negatives provide unbiased cross-modal alignment across heterogeneous sources (Binaural, AudioCaps, MS-COCO).
    Sec. C.2; standard contrastive-learning assumption, not validated for dataset-mixing bias.
  • standard math Hungarian bipartite matching yields the correct supervision assignment despite noisy predicted slots and variable object counts.
    Sec. 3.3, Eq. 3; optimal assignment is standard, but the matching score depends on the untested λ, α, β.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SceneBind: Binding What and Where Across Vision, Audio and Language." pith.science (2026). https://pith.science/paper/5S4CVV7Z

@misc{pith2026260715265,
  author       = {Pith},
  title        = {Pith review of: SceneBind: Binding What and Where Across Vision, Audio and Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5S4CVV7Z}},
  note         = {Machine review of arXiv:2607.15265}
}
read the original abstract

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.

Figures

Figures reproduced from arXiv: 2607.15265 by the authors.

Figure 1
Figure 1. SceneBind models scenes by jointly capturing semantics ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SceneBind overview. (a) SceneBind Encoding Model maps multimodal inputs to a global embedding s global and object-centric slots (sk, rk, ck). (b) SceneBind Representation X represents each scene with a global embedding and K object-centric semantic-spatial slots. (c) Bipartite matching aligns slots with GT object clauses for supervision. (d) SceneBind Matching combines global similarity Sglobal with object slot alig… view at source ↗
Figure 3
Figure 3. Object slot scaling. Per￾formance saturates or drops beyond a moderate number; ∼10 slots suffice [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Qualitative results. SceneBind retrieves semantically and spatially aligned scenes across audio, vision, and text. We show top-3 candidates with global similarity (g), object-centric matching score (o), and grounded slots with confidence and spatial labels. 4.7 Zero-sh…
Figure 5
Figure 5. Figure 5: AV localization visualization. Global semantic attention proposes sounding regions, refined by spatial consistency of SceneBind slots. to model object motion, long-range scene dynamics, and temporally consistent semantic-spatial reasoning. Additional discussion is prov…
Figure 6
Figure 6. Figure 6: Long-tail distributions of scene tags and semantic event clusters in the Binaural dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Binaural data construction funnel. The pipeline filters source videos, selects diverse [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Audio–Visual event and spatial annotation prompt for binaural videos via Gemini. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Azimuth distribution of the primary audio–visual event per clip, before ( [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Human verification interface for reviewing candidate benchmark clips. It allows reviewers [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: A→V beach retrieval. SceneBind ranks the ground truth first by matching both the global beach semantics and the object level spatial cue of waves, while baseline retrievals capture coarse beach semantics but may miss the spatial layout. Case 2: Multiple Slots Capture …
Figure 12
Figure 12. Figure 12: T→V river retrieval. SceneBind matches both the global river road context and object level spatial cues of the riverbed and vehicles, while ImageBind is less sensitive to the detailed text query and M2D-SigLIP2* misses key objects or retrieves weaker layouts [PITH_FU…
Figure 13
Figure 13. Figure 13: A→V moving bus retrieval. SceneBind uses multiple bus engine relevant slots to represent plausible semantic spatial hypotheses for a moving sound source, retrieving scenes with road and vehicle layouts that better match the binaural audio. how SceneBind slots expose o…
Figure 14
Figure 14. Figure 14: T→A jazz retrieval. SceneBind ranks the ground-truth audio first, and its top retrieved samples show similar jazz performance context, spectrogram and waveform patterns, and object level spatial cues for instruments such as saxophone and drums [PITH_FULL_IMAGE:figure…
Figure 15
Figure 15. Figure 15: V→A street retrieval. SceneBind retrieves audio with similar street-scene acoustics and frontal vehicle cues. Although the ground truth is ranked second, the top results remain semantically and spatially close, illustrating the challenge of scene retrieval when audio …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 10 linked inside Pith

  1. [1]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    arXiv 2024

  2. [2]

    SA VVY: Spatial awareness via audio-visual LLMs through seeing and hearing

    Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, and Eli Shlizerman. SA VVY: Spatial awareness via audio-visual LLMs through seeing and hearing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  3. [3]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  4. [4]

    Sigmoid loss for language image pre-training.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023

    2023

  5. [5]

    Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H’enaff, Jeremiah Harmsen, Andreas Steiner, and Xiao-Qi Zhai

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim M. Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H’enaff, Jeremiah Harmsen, Andreas Steiner, and Xiao-Qi Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense feature...

    arXiv 2025

  6. [6]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  7. [7]

    M2d-clap: Masked modeling duo meets clap for learning general-purpose audio-language representation, 2024

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Masahiro Yasuda, Shun- suke Tsubaki, and Keisuke Imoto. M2d-clap: Masked modeling duo meets clap for learning general-purpose audio-language representation, 2024

    2024

  8. [8]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023

    2023

Show all 59 references
  1. [9]

    Soundspaces 2.0: A simulation platform for visual-acoustic learning.Advances in Neural Information Processing Systems, 35:8896– 8911, 2022

    Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning.Advances in Neural Information Processing Systems, 35:8896– 8911, 2022

    2022

  2. [10]

    Spatial-clap: Learning spatially-aware audio–text embeddings for multi-source conditions.arXiv preprint arXiv:2509.14785, 2025

    Kentaro Seki, Yuki Okamoto, Kouei Yamaoka, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. Spatial-clap: Learning spatially-aware audio–text embeddings for multi-source conditions.arXiv preprint arXiv:2509.14785, 2025

    2025

  3. [11]

    Audioclip: Extending clip to image, text and audio, 2021

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio, 2021

    2021

  4. [12]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  5. [13]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  6. [14]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023. 11

    2023

  7. [15]

    Robohop: Segment-based topological map representation for open-world visual navigation

    Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open-world visual navigation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4090...

    2024

  8. [16]

    Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfus...

    2023

  9. [17]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference...

    2024

  10. [18]

    3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    2024

  11. [19]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. InProceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636, 2023

    2023

  12. [20]

    Visual language maps for robot navigation

    Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  13. [21]

    Chatsplat: 3d conversational gaussian splatting

    Hanlin Chen, Fangyin Wei, and Gim Hee Lee. Chatsplat: 3d conversational gaussian splatting. arXiv preprint arXiv:2412.00734, 2024

    2024 arXiv

  14. [22]

    Language embedded 3d gaus- sians for open-vocabulary scene understanding

    Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333–5343, 2024

    2024

  15. [23]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    2024

  16. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.ArXiv, abs/2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.ArXiv, abs/2303.05499, 2023

    2023 arXiv

  17. [25]

    Grounding dino 1.5: Advance the "edge" of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection. ArXiv, abs/2...

    2024 arXiv

  18. [26]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors,Computer Vision - ECCV 2020 - 16th Europea...

    2020

  19. [27]

    Deformable DETR: deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021

    2021

  20. [28]

    Visual scene graphs for audio source separation.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1184–1193, 2021

    Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1184–1193, 2021. 12

    2021

  21. [29]

    Learning audio-visual dynamics using scene graphs for audio source separation

    Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for audio source separation. InAdvances in Neural Information Processing Systems, 2022

    2022

  22. [30]

    Audio-visual grouping network for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10565–10574, 2023

    2023

  23. [31]

    Lavss: Location-guided audio-visual spatial audio separation

    Yuxin Ye, Wenming Yang, and Yapeng Tian. Lavss: Location-guided audio-visual spatial audio separation. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5508–5519, 2024

    2024

  24. [32]

    Audio-visual scene analysis with self-supervised multisen- sory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisen- sory features. InProceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018

    2018

  25. [33]

    Discriminative sounding objects localization via self-supervised audiovisual matching

    Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33:10077–10087, 2020

    2020

  26. [34]

    Cyclic co-learning of sounding object visual grounding and sound separation.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2744–2753, 2021

    Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2744–2753, 2021

    2021

  27. [35]

    Sound event localization and detection of overlapping sources using convolutional recurrent neural networks

    Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34–48, 2018

    2018

  28. [36]

    Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    David Diaz-Guerra, Antonio Miguel, and Jose R Beltran. Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    2020

  29. [37]

    Bat: Learning to reason about spatial sounds with large language models.International conference on machine learning, 2024

    Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learning to reason about spatial sounds with large language models.International conference on machine learning, 2024

    2024

  30. [38]

    Learn- ing spatially-aware language and audio embeddings

    Bhavika Suresh Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, YELENA MENYAYLENKO, Barry-John Theobald, Jonathan Sheaffer, and Miguel Sarabia. Learn- ing spatially-aware language and audio embeddings. InThe Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  31. [39]

    Phasecoder: Microphone geometry-agnostic spatial audio understanding for multimodal llms, 2026

    Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, and Vivek Kumar. Phasecoder: Microphone geometry-agnostic spatial audio understanding for multimodal llms, 2026

    2026

  32. [40]

    Hear you are: Teaching llms spatial reasoning with vision and spatial sound

    Hyeonggon Ryu, Joon Son Chung, and David Harwath. Hear you are: Teaching llms spatial reasoning with vision and spatial sound. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 38606–38615, 2026

    2026

  33. [41]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  34. [42]

    Harold W. Kuhn. The hungarian method for the assignment problem.Naval Research Logistics (NRL), 52, 1955

    1955

  35. [43]

    Audiocaps: Generat- ing captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. InNAACL-HLT, 2019

    2019

  36. [44]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014. 13

    2014

  37. [45]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  38. [46]

    Omniaudio: Generating spatial audio from 360-degree video.ArXiv, abs/2504.14906, 2025

    Huadai Liu, Tianyi Luo, Qikai Jiang, Kaicheng Luo, Peiwen Sun, Jialei Wan, Rongjie Huang, Qian Chen, Wen Wang, Xiangtai Li, Shiliang Zhang, Zhijie Yan, Zhou Zhao, and Wei Xue. Omniaudio: Generating spatial audio from 360-degree video.ArXiv, abs/2504.14906, 2025

    2025 arXiv

  39. [47]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910–22921, 2023

    2023

  40. [48]

    Learning to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018

    2018

  41. [49]

    Space-time memory network for sounding object localization in videos.arXiv preprint arXiv:2111.05526, 2021

    Sizhe Li, Yapeng Tian, and Chenliang Xu. Space-time memory network for sounding object localization in videos.arXiv preprint arXiv:2111.05526, 2021

    2021 arXiv

  42. [50]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867–16876, 2021

    2021

  43. [51]

    Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes

    Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3222–3...

    2022

  44. [52]

    Mix and localize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022

    2022

  45. [53]

    Seeing speech and sound: Distinguishing and locating audio sources in visual scenes

    Hyeonggon Ryu, Seongyu Kim, Joon Son Chung, and Arda Senocak. Seeing speech and sound: Distinguishing and locating audio sources in visual scenes. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13540–13549, 2025

    2025

  46. [54]

    Cnn architectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In2017 ieee international conference on acoustics, speech...

    2017

  47. [55]

    Hrtf measurements of a kemar dummy-head microphone

    Bill Gardner and Keith Martin. Hrtf measurements of a kemar dummy-head microphone. 1994. 14 A Appendix Overview In this appendix, we provide supplementary details and analyses that complement the main paper. • Dataset Curation(Sec. B): We describe the data sources, annotation ...

    1994

  48. [56]

    - visual_only: if visible, but it is silent or not synchronized with any sound

    Event Modality Constraints - audio_only: if you can hear the object, but it is not in the field of view. - visual_only: if visible, but it is silent or not synchronized with any sound. - audio_visual: if visible AND the sound is clearly synchronized with its motion AND there i...

  49. [57]

    - Duration Constraint: Events must be short atomic instances

    Annotation Rules (Strict) - Timestamps: Must be relative to clip start (0s to 10s). - Duration Constraint: Events must be short atomic instances. Duration must be EXACTLY 1 or 2 seconds. - Correct Format: [1,3), or [2,3) relative to clip start; Incorrect Format: [00:01:21-00:0...

  50. [58]

    Spatial Definitions (Camera Coordinates) Azimuth (Horizontal): Center is 0 deg; Negative is left; Positive is right. - Hard Left: [-90, -60) - Left: [-60, -30) - Slight Left: [-30, -15) - Front: [-15, 15] - Slight Right: (15, 30] - Right: (30, 60] - Hard Right: (60, 90] Elevat...

  51. [59]

    - semantic_anno (6 to 8 words): Describe WHAT the object is doing/being

    Field Descriptions - semantic_tag: A concise semantic tag for the event, e.g., ‘car honking’, ‘playing piano’. - semantic_anno (6 to 8 words): Describe WHAT the object is doing/being. Strictly Forbidden: Directional words. - spatial_anno (about 8 words): Describe WHERE the obj...

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.