{"id":"7a1e1cd2-4652-49e9-a977-48f98d634a25","arxiv_id":"2412.01488","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TACO uses semantically constrained co-factorization of frozen CLIP/CLAP features, then prompts FC-CLIP with the identified sounding factor, achieving state-of-the-art unsupervised sound-prompted segmentation without training.","lead":"This paper introduces TACO, a training-free method that segments image regions corresponding to sounds by factorizing frozen CLIP and CLAP features with non-negative matrix factorization. It reports state-of-the-art results among unsupervised sound-prompted segmentation methods on AVSBench, ADE Sound Prompted, and AVSS, using semantic word anchors to match audio and visual concepts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The supplied algorithm does not enforce the non-negativity or [0,1] constraints claimed in Eq. (6) and Sec. 3.2; the reported factors and U^{k*}_I segmentation may come from an unconstrained factorization, so the paper's described mechanism is not what is evaluated.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper's empirical results are plausible but need clarification. However, I identify a different load-bearing weakness than the reader's primary 'weakest_assumption'. The reader emphasized the semantic alignment of CLIP and CLAP word anchors; that is a real modeling assumption, but the paper provides ablations (word bank size, KL vs CE) and the method still works on standard benchmarks. A more immediate and concrete correctness threat is the mismatch between the constrained optimization stated in Eq. (6)/Sec. 3.2 and the unconstrained gradient-descent algorithm in Appendix C. This is not a mere notation issue: if the optimization is exactly as described, the matrices can be negative, the semantic descriptors can be signed, cross-entropy is not properly defined, and the interpretation of U^{k*}_I as a segmentation mask is invalid. The paper's clamping analysis only covers text embeddings, not the decomposed features used in the actual pipeline. Therefore, the published description does not yet establish that the reported 64.04 mask-IoU comes from the 'semantically constrained co-NMF' mechanism that the paper claims. This reinforces the reader's conditional verdict: acceptance should be contingent on either code release or a corrected algorithm description plus an experiment that matches the reported numbers. I would not reject outright because the method could be sound with a simple implementation fix, and the empirical evidence suggests some version of the co-factorization works. The concern is concrete and testable, so UNCHANGED (conditional) is the right call.","tokens_in":15732,"tokens_out":8951,"duration_ms":83674,"concrete_test":"Implement two variants on the AVSBench S4 split with the paper's hyperparameters (K=8, beta_p=125, 1800 steps, lr=0.25): (A) exactly as the Appendix C pseudo-code, i.e., Gaussian initialization and direct gradient updates on U_A, U_I, V_A, V_I with no constraint enforcement; (B) the stated constrained version, with U_A and U_I obtained via sigmoid from freely optimized real matrices and V_A, V_I projected to non-negative values (or parameterized via exponentiation) after each step. Compare the S4 mask-IoU to the reported 64.04±0.25. If (A) reproduces the number, the paper's NMF and soft-mask claims are inaccurate; if (B) matches, the pseudo-code omits a critical step. Either outcome settles whether the central claim is supported by the published algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is a 'semantically constrained co-NMF' whose non-negativity and bounded-activation constraints are asserted in Sec. 3.2 and Eq. (6). But the paper's own implementation details contradict these constraints. Sec. 3.2 states that U_A and U_I are re-parameterized through a sigmoid so that U_A, U_I in [0,1], and Eq. (6) constrains V_A, V_I, U_A, U_I to be non-negative. However, Appendix C's pseudo-code initializes U_A, U_I, V_A, V_I from Gaussians and updates them directly by gradient descent, with no sigmoid, no ReLU, and no projection onto the non-negative orthant. The main text also says 'All the matrices U_A, V_A, U_I and V_I are initialized randomly from Gaussians,' which is inconsistent with a sigmoid re-parameterization. If the optimization is run exactly as described, factors and activations can be negative. Then: (i) the Hadamard products X_I ⊙ U^k_I used for semantic components can be negative, making their cosine similarities to anchors signed and possibly ill-defined; (ii) the cross-entropy penalty CE(D_I^k, D_A^k) is not defined for arbitrary signed descriptor vectors unless an unstated softmax or normalization is applied; and (iii) the claim that U^{k*}_I is a soft segmentation mask in [0,1] is false. The paper's clamping experiment (Appendix A) validates that ReLU on CLIP *text* embeddings is harmless for FC-CLIP, but it does not validate clamping the *decomposed image/audio features* that are actually used. This is load-bearing because the reported state-of-the-art numbers are attributed to a specific constrained optimization; if the actual optimization omits those constraints, the described method is not the method that produced Table 1, and the interpretability and segmentation-map arguments collapse. The concern is resolved only by code release or by a precise statement of the actual optimization (e.g., whether sigmoid/ReLU is applied after each update).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TACO proposes a training-free pipeline for sound-prompted segmentation. It uses frozen CLIP and CLAP encoders to obtain image and audio token matrices, factorizes each matrix into a small number of factors and activations via a soft co-NMF objective, and aligns one audio factor with one image factor using cosine similarities to a shared word bank. The selected image activation U^{k*}_I is used directly as a mask, and the corresponding image factor V^{k*}_I prompts FC-CLIP for a refined open-vocabulary segmentation. The paper reports state-of-the-art results on AVSBench S4/MS3 and ADE Sound Prompted, introduces a semantic variant of the ADE Sound Prompted dataset, and supports its design with ablations over the penalty, semantic components, word bank size, K, and beta_p.","tokens_in":16120,"tokens_out":11647,"duration_ms":112880,"significance":"If the method is implemented as described, this is a significant contribution: it would show that frozen CLIP/CLAP features contain enough audio-visual correspondence for sound-prompted localization without any training, and that NMF-style factor decomposition yields interpretable prompts that improve an open-vocabulary segmenter. The paper is transparent in important ways: it reports standard deviations over three runs, ablates the main design choices, and gives a detailed pseudo-code. The significance is currently tempered by the mismatch between the stated non-negativity/bounded constraints and the unconstrained updates in Algorithm 1, and by the partly self-referential selection of the sounding factor.","major_comments":[{"comment":"The optimization actually described by the manuscript is inconsistent with the claims. Section 3.2 says UA and UI are re-parameterized through the sigmoid so that they lie in [0,1], and Eq. (6) imposes non-negativity on VA, VI, UA, and UI. However, Algorithm 1 initializes UA, UI, VA, and VI from Gaussians and updates them directly by gradient descent, with no sigmoid, clipping, or projection, and Section 4 repeats that the matrices are initialized from Gaussians. Since Appendix C states that the pseudo-code details the exact computations, either the pseudo-code is not the implemented method, or the evaluated method does not satisfy the constraints on which the interpretability and soft-mask claims rest. This is load-bearing: U^{k*}_I is used as a segmentation mask, and negative entries in U affect the Hadamard products in Eq. (4), the cosine descriptors in Eq. (5), and the cross-entropy penalty in Eq. (6). Please state the exact optimization actually used, correct the pseudo-code, and re-run or confirm the experiments under the stated constraints.","section":"Section 3.2 / Eq. (6) / Algorithm 1 (Appendix C)"},{"comment":"The clamping experiment validates clamping only the CLIP text embedding before it is fed to FC-CLIP (Table 4). The inputs that actually need to be non-negative for the NMF objective are the CLIP image token features and CLAP audio features, and the semantic components C^k_I and C^k_A are averages of those features. The paper does not measure the effect of clipping those features to zero, nor does it explain how negative entries in the semantic components are handled. The claim that clipping negative values preserves representation power for 'our tasks' is therefore not supported by the provided experiment. Please validate clamping on the actual NMF inputs or specify a different mechanism for satisfying the non-negativity condition.","section":"Section 4 (Preliminary experiments) and Appendix A"},{"comment":"The penalty CE(D_I^k, D_A^k) is not well defined as written. The descriptors D are vectors of cosine similarities, which can take negative values and are not normalized to be probability distributions. Cross-entropy requires two probability distributions (or a distribution and logits, with the direction specified), so an additional normalization such as softmax must be specified. Without this, the objective in Eq. (6) is ambiguous, and different choices of normalization will change the gradient and the final k*. Please give the exact definition of the cross-entropy used, including whether the descriptors are passed through softmax and which descriptor is treated as the target.","section":"Eq. (5) and Eq. (6)"},{"comment":"The selection of the sounding factor is partly by construction. The optimization objective in Eq. (6) already contains min_k CE(D_I^k, D_A^k), and k* is then defined as the argmin of the same quantity. This guarantees that the selected factor is aligned in descriptor space, so the matching step alone does not provide independent evidence that descriptor alignment identifies the true sound source. The external mask-IoU evaluations support the full pipeline, but the interpretability claim that k* 'corresponds to the dominant semantic component shared between the audio and image modalities' needs additional validation. Please report, for example, how often k* agrees with a ground-truth source label, or compare the final segmentation when k* is chosen versus a random factor or the second-best factor.","section":"Section 4.1 / Eq. (6) / Interpretation paragraph"}],"minor_comments":[{"comment":"Since UI is defined as an HW x K matrix, the spatial activation of factor k is the k-th column, not the k-th row; the sentence 'the kth row of UI represents the segmentation of the kth factor' should be corrected, as should the similar sentence after Eq. (6).","section":"Section 3.2"},{"comment":"References [43] and [44] have the same authors and title; one appears to be a duplicate, and the entry used in Table 1 should be aligned with the intended method.","section":"References"},{"comment":"The sentence 'only outperformed by MarginNCE in terms of F-score' is imprecise: in Table 1 MarginNCE also reports a higher mask-IoU on S4 than TACO without the segmenter (33.27 vs. 29.68).","section":"Section 4.1"},{"comment":"The F-score metric used in Tables 1-3 is not defined; please state how it is computed.","section":"Section 4 (Datasets and Metrics)"},{"comment":"The explanation for setting beta_temp = 0 for the multi-source task is confusing: AVS-MS3 is a multi-source video benchmark, so it is not clear why temporal consistency is disabled precisely for the multi-source setting; please clarify.","section":"Section 4 (Implementation Details)"},{"comment":"Line 14 calls s(V_It, X_I), but the pseudo-code input list defines only V_I and X_I; align the notation for the multi-frame case.","section":"Algorithm 1 (Appendix C)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable if the implementation discrepancy is resolved. I recommend requesting the actual code or a corrected algorithmic description, because the current text does not allow a reader to reproduce the exact objective that produced the reported numbers. The paper fits the journal's scope in audio-visual multimodal learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: TACO is a genuinely interesting training-free approach to sound-prompted segmentation, and it shows real skill in assembling frozen CLIP/CLAP features, a semantic word-bank penalty, and FC-CLIP prompting. The reported gains—64.04 mask-IoU on S4, 43.15 on MS3, 51.57 on ADE SP—are substantial, and the ablations are honest and reasonably thorough. The idea of using soft co-NMF with text-anchor descriptors to bridge unaligned CLIP and CLAP spaces is new, and the qualitative results show the decomposition does pick out meaningful audio-visual correspondences.\n\nBut I have two concerns, one serious, one moderate.\n\nThe serious one: the supplied Algorithm 1 does not enforce the non-negativity and [0,1] constraints that the paper's Eq. (6) and Section 3.2 claim. The main text says all matrices are initialized from Gaussians and updated directly with gradient descent, with no sigmoid, no ReLU, no projection. So the 'semantically constrained co-NMF' the paper describes is not what the pseudo-code implements. If the code runs as written, the factors and activations can be negative, which means the Hadamard products and cosine-similarity descriptors can be signed, and the claim that U^{k*}_I is a soft segmentation mask in [0,1] is false. That doesn't automatically kill the empirical results—unconstrained factorization might still work—but it makes the interpretability story and the mechanism description unreliable until code is released or the optimization is stated accurately. This needs to be fixed in revision.\n\nThe moderate concern is the metric. Appendix F explains that they report mask-IoU while several prior works may have reported mean-IoU, and they acknowledge this makes some comparisons uncertain. Yet the paper still claims 'state-of-the-art' based on those numbers. That's a defensible position if the numbers are truly comparable, but it needs a clearer statement of which prior numbers are actually mask-IoU. The word-bank sensitivity in Appendix B (a 4x reduction drops S4 by ~8.5 points) also shows the method leans heavily on the anchor set, which is worth stating more prominently.\n\nAlso worth noting: the sounding factor is selected by minimizing the same cross-entropy penalty used in training, so the matching step is partly by construction. That's not a fatal flaw—it's part of the method—but it does mean the 'unsupervised discovery' framing is a bit softer than it appears.\n\nOverall, this is a solid empirical paper with a real contribution and a correctable description gap. I'd send it to peer review, and I'd ask the authors to release code and clarify the actual optimization. The readers' conditional verdict is about right.","headline":"Strong training-free audio-visual segmentation results, but the algorithm description contradicts the claimed constraints and metric comparability needs tightening.","tokens_in":16720,"tokens_out":2783,"would_cite":true,"duration_ms":25092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sound-prompted segmentation can be done without any training, by co-factorizing frozen audio and image features with semantic word anchors, and that this training-free approach surpasses existing unsupervised…","keywords":["sound-prompted segmentation","audio-visual segmentation","non-negative matrix factorization","training-free","zero-shot","CLIP","CLAP","open-vocabulary segmentation"],"falsifier":"Use a test set of audio-image pairs in which the sounding object is deliberately excluded from the word bank (e.g., a rare instrument not among AudioSet tags). If TACO still localizes the object correctly, the semantic anchors are not doing the claimed work; if segmentation collapses, the word bank coverage is load-bearing. A second check: replace the word embeddings with random vectors of the same dimension; if performance stays high, the semantic alignment is unnecessary, and if it drops sharply, the anchors are genuinely aligning the two spaces.","tokens_in":15549,"feed_emoji":"🔊","tokens_out":5214,"duration_ms":40974,"temperature":0.7,"pith_summary":"The paper asks whether sound-prompted segmentation — locating the image regions that produce a heard sound — can be solved without any training, using only frozen pre-trained audio and image encoders. It answers yes: by co-factorizing the audio and image feature matrices with non-negative matrix factorization and steering the factorization with a small bank of semantic word anchors, the method surfaces a shared \"sounding\" factor, which is then fed to an open-vocabulary segmenter. Across AVSBench, ADE Sound Prompted, and AVSS, the resulting system, TACO, reports the best numbers among training-free and unsupervised approaches, including a mask-IoU of 64.04 on AVSBench S4. The practical point is that large frozen models already encode the cross-modal correspondence needed for localization, so task-specific fine-tuning is unnecessary.","feed_headline":"Training-free method tops unsupervised sound segmentation","feed_subtitle":"Frozen CLIP and CLAP features, aligned by semantic word anchors, localize the sound source and prompt FC-CLIP for precise masks.","key_machinery":"The load-bearing mechanism is Semantically constrained soft co-NMF (Sem co-NMF), a training-free factorization that jointly decomposes frozen audio and image token features while enforcing semantic agreement. Each modality is factorized as $X \\approx U V$ with non-negative factors and sigmoid-bounded activations; the semantic bridge is a word bank whose text embeddings are passed through the CLIP and CLAP text encoders to create paired anchors $(b_I^j, b_A^j)$. For each factor $k$, a soft-masked pooled component $C^k = \\mathrm{avg}(X \\odot U^k)$ is compared with these anchors to form a semantic descriptor $D^k$, and the objective penalizes the cross-entropy between the closest audio and image descriptor pair. The factor $k^*$ minimizing that penalty is the sounding concept; its activations give the localization and its factor vector prompts FC-CLIP for refined masks.","core_discovery":"The central claim is that frozen CLIP and CLAP features contain enough aligned information for audio-visual source localization without any parameter updates. The paper's Sem co-NMF decomposes the audio token matrix $X_A$ and the image token matrix $X_I$ into non-negative factor and activation matrices, with activations in $[0,1]$ via a sigmoid reparameterization. To bridge the fact that CLIP and CLAP live in different embedding spaces, the method projects each factor's soft-masked component onto a shared semantic space by computing cosine similarities against word embeddings from a word bank, and then penalizes the cross-entropy between the closest audio and image descriptors. The index $k^*$ with minimal cross-entropy identifies the sounding factor; its image activation row $U^{k^*}_I$ is a coarse segmentation, and the corresponding image factor $V^{k^*}_I$, which lives in CLIP space, is used to prompt the FC-CLIP open-vocabulary segmenter. When these components are combined, the paper reports state-of-the-art unsupervised performance on S4, MS3, ADE Sound Prompted, and the semantic variants.","pith_inferences":["A direct experimental prediction: if the semantic anchors are replaced by random vectors that are not tied to words, performance should collapse to near-chance; demonstrating that would confirm the anchors (not just the factorization) are carrying the alignment.","The word bank is effectively a hyperparameter; using dataset-specific class names improves results, which implies that a fixed general word bank may limit generalization to very rare or unseen sound concepts, making adaptive anchor selection a natural testable extension.","The cross-entropy penalty between semantic descriptors assumes the closest audio-image factor pair is the correct one; on clips with multiple simultaneous sounds this could be ambiguous, so an explicit multi-factor matching or a top-k penalty could improve MS3-style tasks.","The framework should transfer to other paired modalities (e.g., text-image or video-text) as long as a shared word-bank space exists, opening a route to training-free object localization from natural language descriptions."],"forward_implications":["If frozen CLIP and CLAP features are sufficient, then sound-prompted segmentation no longer requires task-specific training data or fine-tuning, so the same models can be repurposed for new scenes and audio types at inference time.","The interpretable factors let a user read what concept was matched (e.g., \"baby babbling\"), and the same decomposition can be reused for other downstream tasks such as semantic segmentation.","The method's robustness on multi-source audio, where several objects sound together, suggests a single sounding factor can encode multiple co-active sources, potentially simplifying multi-source localization.","Because the decomposition is done at inference, any improvement in the frozen backbones (better CLIP or CLAP) immediately improves segmentation without re-training the framework."],"supporting_citations":[{"why":"Supplies the CLIP image and text encoders whose frozen features and text embeddings are used as the visual backbone and image-side semantic anchors.","marker":"[35]"},{"why":"Supplies the CLAP audio and text encoders used to extract audio tokens and audio-side semantic anchors.","marker":"[10]"},{"why":"Provides the classic non-negative matrix factorization formulation that Sem co-NMF builds on.","marker":"[22]"},{"why":"Introduces soft co-NMF with separate activation matrices and a penalty term, which the paper revisits as Sem co-NMF.","marker":"[37]"},{"why":"FC-CLIP is the frozen open-vocabulary segmenter prompted with the sounding factor to produce the final precise masks.","marker":"[50]"},{"why":"Supplies the 527 AudioSet tags used as the general word bank for non-semantic sound-prompted segmentation.","marker":"[12]"},{"why":"Defines the AVSBench S4 and MS3 datasets with binary masks, the primary benchmark for the reported segmentation scores.","marker":"[53]"},{"why":"Introduces the ADE Sound Prompted dataset and the DenseAV baseline, providing a key comparison target and the harder zero-shot segmentation benchmark.","marker":"[18]"}],"fun_headline_variants":["Training-free NMF co-factorization tops unsupervised sound segmentation","Frozen CLIP and CLAP features achieve SOTA unsupervised sound segmentation","No fine-tuning: TACO co-factorizes audio and visual features for segmentation","Semantic anchors align audio-visual co-factors for training-free segmentation","Sound-prompted segmentation without training via semantic co-factorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that the same word, embedded by CLIP's text encoder and by CLAP's text encoder, lands in comparable semantic neighborhoods, so that cosine similarities to a shared word bank describe audio and visual factors on the same scale; if the two spaces drift apart or the sounding concept is absent from the word bank, the cross-entropy penalty cannot identify the correct factor.","fun_headline_variants_meta":{"raw":{"variants":["Training-free NMF co-factorization tops unsupervised sound segmentation","Frozen CLIP and CLAP features achieve SOTA unsupervised sound segmentation","No fine-tuning: TACO co-factorizes audio and visual features for segmentation","Semantic anchors align audio-visual co-factors for training-free segmentation","Sound-prompted segmentation without training via semantic co-factorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001305,"raw_usage":{"total_tokens":5316,"prompt_tokens":931,"completion_tokens":4385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":4292}},"tokens_in":547,"tokens_out":4385,"duration_ms":28543,"temperature":1.0,"reasoning_tokens":4292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:49.892490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a test set of audio-image pairs in which the sounding object is deliberately excluded from the word bank (e.g., a rare instrument not among AudioSet tags). If TACO still localizes the object correctly, the semantic anchors are not doing the claimed work; if segmentation collapses, the word bank coverage is load-bearing. A second check: replace the word embeddings with random vectors of the same dimension; if performance stays high, the semantic alignment is unnecessary, and if it drops sharply, the anchors are genuinely aligning the two spaces.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP image and text encoders whose frozen features and text embeddings are used as the visual backbone and image-side semantic anchors."},{"cited_title":"Clap learning audio concepts from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLAP audio and text encoders used to extract audio tokens and audio-side semantic anchors."},{"cited_title":"Algorithms for non-negative matrix factorization","cited_arxiv_id":null,"evidence_quote":"Provides the classic non-negative matrix factorization formulation that Sem co-NMF builds on."},{"cited_title":"Soft nonnegative matrix co-factorization","cited_arxiv_id":null,"evidence_quote":"Introduces soft co-NMF with separate activation matrices and a penalty term, which the paper revisits as Sem co-NMF."},{"cited_title":"Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip","cited_arxiv_id":null,"evidence_quote":"FC-CLIP is the frozen open-vocabulary segmenter prompted with the sounding factor to produce the final precise masks."},{"cited_title":"Audio set: An ontology and human-labeled dataset for audio events","cited_arxiv_id":null,"evidence_quote":"Supplies the 527 AudioSet tags used as the general word bank for non-semantic sound-prompted segmentation."},{"cited_title":"Audio–visual segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the AVSBench S4 and MS3 datasets with binary masks, the primary benchmark for the reported segmentation scores."},{"cited_title":"chirp\" from the","cited_arxiv_id":null,"evidence_quote":"Introduces the ADE Sound Prompted dataset and the DenseAV baseline, providing a key comparison target and the harder zero-shot segmentation benchmark."}],"review_version":1}