{"id":"c7e52f4e-cef2-42a8-938d-11b7b48b7484","arxiv_id":"2507.16736","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DFR integrates visual, textual, and audio guidance in a SAM-based framework and reports mIoU gains over state-of-the-art few-shot segmentation methods on PASCAL-5i and AVS-V3.","lead":"The paper introduces DFR, a framework that adds audio and text cues to few-shot image segmentation by combining a vision model, text descriptions, and sound signals. The authors report consistent gains over prior methods on two benchmarks, including a roughly 7 point mIoU improvement in one-shot PASCAL-5i.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic audio on PASCAL-5i is generated from the same category name as text, so the 1.7 mIoU audio gain may reflect added parameters rather than independent cues; AVS-V3 lacks audio ablation and uses non-SAM baselines.","rationale":"I read the paper's central claim as: audio provides complementary semantic information in few-shot segmentation, and DFR's fusion architecture exploits it. The PASCAL-5i experiments are the primary quantitative support for the tri-modal claim, but there the audio is generated from the category name T. This makes the audio branch a text-derived signal, so the 1.7-point gain over Visual+Text is the only direct evidence for audio's independent contribution. That gap is small, unreplicated, and confounded by added parameters. The AVS-V3 experiments use real audio but do not ablate audio, so they do not isolate the audio contribution; they also compare against methods that do not use SAM/LanguageBind, which may explain part of the gain. This is not an accusation of dishonesty—the framework may still work—but the current evidence is insufficient to support the 'audio as an additive cue' claim. A parameter-matched control and error bars would settle it. I agree with the reader's weakest assumption and recommend keeping the conditional verdict; the concern is addressable and does not by itself warrant rejection.","tokens_in":9675,"tokens_out":4213,"duration_ms":44181,"concrete_test":"Run DFR on PASCAL-5i with the audio branch replaced by a text-derived feature with no acoustic content—e.g., feed the text embedding of T through a randomly initialized MLP with the same output dimension as the audio encoder, keeping the total parameter count and training recipe identical. If the 1-shot mIoU remains at roughly 76.7 (within 0.5 of the full model), the audio modality is not contributing independent semantic information. Additionally, run both Visual+Text and Full models over 5 random seeds and report mean±std; if the 1.7-point gap falls within one standard deviation, the audio claim is not statistically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that audio is an additive cue for few-shot segmentation rests primarily on the PASCAL-5i experiments, where audio is synthesized as A = AudioLDM(T) (Sec III-B1, Sec IV-A). Since AudioLDM is a text-to-audio model, the audio embedding f_a is a (stochastic) function of the same category name T that feeds the text branch; it cannot provide information independent of the text modality. The only direct evidence for audio's contribution is the 1.7 mIoU gap between the full model and Visual+Text in Table III (76.7 vs 75.0). No error bars or significance tests are reported, so this gap could stem from the added audio encoder, the extra contrastive anchor, or modality-dropout regularization rather than from acoustic semantics. The AVS-V3 real-audio results do not resolve this: no ablation is reported there to show how much of the gain over GAVS comes from audio as opposed to text or from the SAM/LanguageBind backbone, and the Table II baselines are not SAM-based. The paper therefore has not demonstrated that audio carries information beyond the text description.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DFR, a multi-modal few-shot segmentation framework that combines visual support images, textual category descriptions, and audio signals. The method has three stages: a multi-modal decomposition stage that extracts SAM-based visual proposals, LLM-expanded textual descriptors, and AudioLDM-synthesized audio embeddings; a contrastive fusion stage using InfoNCE to align anchors, positives, and negatives across modalities; and a dual-path reconstruction stage that feeds semantic tokens and geometric location priors into a SAM decoder. Experiments are reported on PASCAL-5i with synthetic audio and on AVS-V3 with real audio, claiming state-of-the-art results in both settings.","tokens_in":9933,"tokens_out":3613,"duration_ms":39649,"significance":"If the empirical claims are reproducible, DFR would be a useful contribution: it extends few-shot segmentation from visual-only and visual-textual paradigms to tri-modal guidance, and the decompose-fuse-reconstruct design is clearly specified with ablation tables that are directionally consistent. The use of a real audio-visual dataset as a second evaluation setting is a strength, and the paper is transparent about the synthetic-audio pipeline. However, the central claim that audio is an additive cue is not yet supported: the synthetic audio is generated from the same category name used by the text branch, no statistical significance or variance information is reported, no code is released, and the real-audio experiments lack a modality ablation and use baselines with different backbones. These issues are load-bearing for the headline claim rather than cosmetic.","major_comments":[{"comment":"The synthetic audio input is defined as A = AudioLDM(T), so the audio embedding f_a is a transform of the same category name T that already produces the text branch. Consequently, the 1.7 mIoU gap between the full model and Visual+Text in Table III does not establish that audio provides information beyond the text description; the gain could come from the added audio encoder, the extra InfoNCE anchor, or the modality-dropout regularization. The authors should demonstrate that synthetic audio carries independent semantics, for example by comparing against a text-irrelevant audio signal, by matching parameter counts between the compared configurations, or by adding a real-oracle audio ablation on the same benchmark.","section":"Sec. III-B1 / Sec. IV-A / Table III"},{"comment":"All reported results are single-run numbers without standard deviations or significance tests. The 1.7 mIoU audio gain in Table III and the 7.3-point headline gain over Matcher in Table I could fall within run-to-run variance, especially under a 10-epoch training schedule with random SAM proposal sampling and stochastic AudioLDM generation. The authors should report mean and standard deviation over at least three seeds for the main comparisons and ablations, and, where feasible, a paired significance test.","section":"Tables I-III"},{"comment":"The AVS-V3 comparison does not isolate the contribution of audio or of text from the backbone change: the baselines (AVSBench, AVSegFormer, GAVS) do not use the SAM/LanguageBind backbone of DFR, and no AVS-V3 modality ablation is reported. The claim that real audio is an additive cue therefore rests on the synthetic PASCAL-5i experiments, whose audio branch is text-derived. An AVS-V3 ablation with Visual, Visual+Text, Visual+Audio, and full tri-modal configurations, using the same backbone throughout, is needed to support the generalization claim.","section":"Sec. IV-C / Table II"}],"minor_comments":[{"comment":"There is a typo, 'wher' should be 'where', and the paper should state the overlap threshold value for τ in the text rather than only in the equation.","section":"Eq. (1)"},{"comment":"The symbol f_a is used both for the anchor feature in the InfoNCE expression and for the audio embedding in the fusion module; this double use is confusing and should be disambiguated.","section":"Eq. (3) / Sec. III-B2"},{"comment":"The similarity thresholds δ_t and δ_a are introduced but their values are never reported, even though τ in Eq. (1) and λ in Eq. (7) are specified; please provide the hyperparameter values used.","section":"Eq. (5)"},{"comment":"The sentence 'δ is a similarity threshold' is repeated twice in two consecutive lines; the duplication should be removed.","section":"After Eq. (5)"},{"comment":"The paper does not state whether code or trained models will be released, which would materially help reproducibility given that the central comparisons depend on a specific SAM/LanguageBind/AudioLDM pipeline.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the venue and I see no citation or novelty-disclosure red flags. The main risk is evidentiary: the synthetic-audio setup makes the central 'audio is an additive cue' claim underdetermined, and the AVS-V3 experiments do not currently fill that gap. The requested controls and ablations are feasible within the manuscript's scope, so major revision rather than rejection is the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is audio as a third modality in few-shot segmentation, and that alone earns it a read. The framework is coherent and the results are consistent, but the paper does not yet prove that the audio cue is information-bearing beyond text in its main experiment.\n\nWhat it does well: this is the first FSS work I know to bring audio in alongside visual and textual support, and that formulation is a real step forward. The architecture is clearly specified at a high level — SAM proposals, LLM text expansion, contrastive fusion, dual-path reconstruction into the SAM decoder — and the ablations are directionally sensible: visual > text > audio, full > any pair, semantic and geometric paths both contribute. The AVS-V3 results with real audio give the tri-modal idea some independent support, which is more than many multi-modal papers offer.\n\nSoft spots, in order of importance. First, on PASCAL-5i the audio is synthesized as A = AudioLDM(T), so the audio embedding is a transform of the same category name that already feeds the text branch. The 1.7 mIoU gap between the full model and Visual+Text (76.7 vs 75.0) could just be extra parameters, a second contrastive anchor, or modality dropout. No error bars or significance tests appear anywhere in the paper, so we cannot tell. Second, the AVS-V3 table is reassuring but does not isolate audio: there is no audio ablation on real data, and the baselines there are not SAM-based, so part of the gain may come from the SAM/LanguageBind backbone rather than audio. Third, and minor by comparison, the Refiner is described in a single sentence and the hyperparameters (thresholds, temperature, dropout rate) are not reported.\n\nThe stress-test concern is fair. I do not think the paper has demonstrated that audio carries information beyond text in the synthetic setting. That said, this is not a load-bearing flaw that invalidates the framework. The real-audio results and the consistent ablations make a plausible case that audio helps, and the issues are addressable with variance estimates, an AVS-V3 audio ablation, and code or at least multiple seeds.\n\nWho should read this: anyone working on few-shot segmentation, audio-visual understanding, or multi-modal prompt engineering. It deserves a serious referee. I would accept it conditionally after seeing error bars, the missing AVS-V3 ablation, and better specification of the Refiner and thresholds. I would also bring it to a reading group to discuss what counts as evidence of modality independence.","headline":"First tri-modal (visual-text-audio) few-shot segmentation paper; the idea is genuinely new, results are consistent, but the synthetic audio branch is text-derived, so the audio-specific gain is not yet proven.","tokens_in":10488,"tokens_out":1973,"would_cite":false,"duration_ms":20368,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a decompose-fuse-reconstruct framework that adds audio to visual and textual guidance gives the best reported few-shot segmentation results on both synthetic-audio PASCAL-5i and real-audio AVS-V3.","keywords":["few-shot segmentation","multi-modal fusion","audio-visual segmentation","Segment Anything Model","contrastive learning","text-guided segmentation","PASCAL-5i","AVS-V3"],"falsifier":"Run the same framework on PASCAL-5i with the audio embedding replaced by a fixed random vector shared by all categories, keeping the parameter count fixed: if 1-shot mIoU stays near the reported 75.4 instead of falling toward the 75.0 of Visual+Text, the audio branch is not contributing category-specific signal.","tokens_in":9464,"feed_emoji":"🔊","tokens_out":10204,"duration_ms":94086,"temperature":0.7,"pith_summary":"Few-shot segmentation asks a model to label object categories from one or a handful of labeled examples, and most systems rely on support images or on image plus a text category name. This paper claims that adding audio as a third guidance modality, through a three-stage decompose-fuse-reconstruct pipeline built on the Segment Anything Model (SAM), gives better masks than visual-only or visual-textual systems. On PASCAL-5i, using sounds synthesized from the category name by the text-to-audio model AudioLDM, the framework reports 75.4% and 76.2% mean intersection-over-union (mIoU) in 1-shot and 5-shot settings; on the real audio-visual benchmark AVS-V3 it reports 59.5% and 66.2% mIoU in 0-shot and 1-shot settings. If the claim holds, audio is a usable extra cue for few-shot segmentation, and the alignment machinery could carry over to other modalities.","feed_headline":"Audio, text and vision lift few-shot segmentation by 7.3 points","feed_subtitle":"The DFR framework fuses vision, text, and audio prompts before SAM's decoder, with gains also on real audio-visual data.","key_machinery":"The load-bearing mechanism is the tri-modal decomposition that turns each modality into foreground and background features. SAM splits the support image into positive and negative region proposals by overlap with the support mask; a large language model expands the category name into descriptive attributes and background-context embeddings; and AudioLDM provides an audio embedding. An InfoNCE contrastive loss pulls anchor features (learnable foreground token, support prototype, category name, audio) toward positives (visual positive proposals, text description) and pushes them away from negatives (background token, negative proposals, co-occurring classes). The dual-path reconstruction then feeds a fused semantic token and a geometric embedding, computed from visual, textual, and audio similarity maps over query proposals, into SAM's decoder, followed by a refiner.","core_discovery":"The paper's central claim is that tri-modal guidance, in which visual, textual, and audio cues are first decomposed, then aligned by contrastive learning, and then reconstructed into SAM decoder prompts, outperforms single- and dual-modal guidance. Its reported numbers are 75.4% mIoU (1-shot) and 76.2% (5-shot) on PASCAL-5i, ahead of the visual-only SAM baselines Matcher, VRP-SAM, and FCP by 7.3, 3.5, and 2.2 points respectively in 1-shot, and 59.5% (0-shot) and 66.2% (1-shot) on AVS-V3, ahead of GAVS by 4.8 and 3.3 points. The authors interpret the consistent gains as evidence that audio carries complementary temporal-dynamic semantics rather than duplicating the text description. The discovery is method-level: a specific decomposition-fusion-reconstruction arrangement converts multi-modal prompts into measurable segmentation gains in both synthetic and real audio settings.","pith_inferences":["Beyond the paper: on PASCAL-5i the audio signal is generated as $A = \\mathrm{AudioLDM}(T)$ from the same category name that produces the text branch, so the 1.7 mIoU gap between the full model and Visual+Text may partly reflect extra parameters or regularization rather than new perceptual information; the real-audio results do not settle this.","Beyond the paper: a controlled comparison that matches parameter counts, or that replaces the audio branch with category-independent noise, would test whether the synthetic-audio contribution is genuinely categorical; the paper's ablations do not include such a control.","Beyond the paper: the decompose-fuse-reconstruct recipe suggests testable extensions to other non-visual guidance such as depth or haptic cues, and to tasks like few-shot detection where proposal generation already plays an analogous role."],"forward_implications":["Audio should be treated as a first-class guidance modality in few-shot segmentation, since the reported gains appear both with synthesized sounds and with real recordings.","SAM-based segmenters can accept multi-modal prompts without retraining the frozen image encoder; DFR only changes what is fed to the decoder and adds a contrastive alignment module.","The same fusion recipe transfers from a synthetic setting, where audio is generated from category names, to a real audio-visual dataset, suggesting the alignment mechanism rather than the audio source is what carries the gain.","Explicit negative cues, such as co-occurring classes and negative proposals, can be injected through contrastive learning to suppress background confusion in low-shot settings."],"supporting_citations":[{"why":"SAM supplies the frozen image encoder and promptable decoder that the whole framework is built around.","marker":"[13]"},{"why":"LanguageBind produces the shared embedding space that maps both text descriptions and audio onto the visual features.","marker":"[14]"},{"why":"AudioLDM synthesizes the audio signals used on PASCAL-5i from the category name.","marker":"[19]"},{"why":"Matcher is the strongest visual-only SAM-based baseline that DFR reports beating by 7.3 mIoU in the 1-shot setting.","marker":"[17]"},{"why":"VRP-SAM is the visual-reference-prompt baseline that DFR compares against.","marker":"[16]"},{"why":"FCP's foreground-covering prototype generation is the nearest baseline result DFR improves on.","marker":"[18]"},{"why":"GAVS defines the AVS-V3 zero- and few-shot benchmark and is the baseline DFR reports surpassing by 4.8 and 3.3 mIoU.","marker":"[27]"},{"why":"PASCAL-5i provides the folds, classes, and evaluation protocol for the synthetic-audio experiments.","marker":"[20]"},{"why":"DeepSeek-v3 generates the descriptive and background-context text used in the textual decomposition.","marker":"[26]"}],"fun_headline_variants":["Fusing audio, text, and vision boosts few-shot segmentation","Tri-modal guidance lifts few-shot segmentation by 7.3 points","DFR: Decompose-fuse-reconstruct for multi-modal few-shot segmentation","Audio, text, and vision fusion sharpens segmentation results","Multi-modal prompts enhance few-shot segmentation in real settings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"On PASCAL-5i the audio is synthesized from the category name itself, so the claim that audio adds information beyond the text assumes the generated sound carries discriminative content that the text embedding does not already provide.","fun_headline_variants_meta":{"raw":{"variants":["Fusing audio, text, and vision boosts few-shot segmentation","Tri-modal guidance lifts few-shot segmentation by 7.3 points","DFR: Decompose-fuse-reconstruct for multi-modal few-shot segmentation","Audio, text, and vision fusion sharpens segmentation results","Multi-modal prompts enhance few-shot segmentation in real settings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1386,"prompt_tokens":994,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":610,"tokens_out":392,"duration_ms":4687,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:03:40.354281+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same framework on PASCAL-5i with the audio embedding replaced by a fixed random vector shared by all categories, keeping the parameter count fixed: if 1-shot mIoU stays near the reported 75.4 instead of falling toward the 75.0 of Visual+Text, the audio branch is not contributing category-specific signal.","supporting_citations":[{"cited_title":"Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,","cited_arxiv_id":null,"evidence_quote":"LanguageBind produces the shared embedding space that maps both text descriptions and audio onto the visual features."},{"cited_title":"AudioLDM: Text-to-audio generation with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"AudioLDM synthesizes the audio signals used on PASCAL-5i from the category name."},{"cited_title":"Matcher: Segment anything with one shot using all-purpose feature matching,","cited_arxiv_id":null,"evidence_quote":"Matcher is the strongest visual-only SAM-based baseline that DFR reports beating by 7.3 mIoU in the 1-shot setting."},{"cited_title":"Vrp-sam: Sam with visual reference prompt,","cited_arxiv_id":null,"evidence_quote":"VRP-SAM is the visual-reference-prompt baseline that DFR compares against."},{"cited_title":"Prompting segmentation with sound is generalizable audio-visual source localizer,","cited_arxiv_id":null,"evidence_quote":"GAVS defines the AVS-V3 zero- and few-shot benchmark and is the baseline DFR reports surpassing by 4.8 and 3.3 mIoU."},{"cited_title":"One-shot learning for semantic segmentation,","cited_arxiv_id":null,"evidence_quote":"PASCAL-5i provides the folds, classes, and evaluation protocol for the synthetic-audio experiments."}],"review_version":1}