{"id":"21a59a12-1f38-4513-a757-2555dbdafe6c","arxiv_id":"2501.00645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SoundBrush trains a diffusion image editor to use audio features as editing instructions, letting sound insert objects or change scenery while preserving the original image structure.","lead":"SoundBrush changes photos based on the sound you give it, such as adding a train or making a scene snowier when it hears the matching audio. It is a diffusion-based image editor that translates audio into text-like tokens, and the same approach extends to editing 3D scenes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'sound as a brush' claim is not quantitatively established: all evaluations are consistent with a category-label shortcut, and no test shows that different audio from the same category produces different edits.","rationale":"Agreement with the reader is partial. The reader correctly identifies dataset fidelity as a weakness, but the more fundamental issue is that even if the generated pairs are clean, the supervision signal is category-level: the audio and the target image are linked through the category label, not through the specific acoustic content. The model is trained to map audio to text tokens, and the text-to-image prior already provides category-level edits. The evaluation metrics (AVS, TVS) are satisfied by any method that produces an image containing the category's object; they do not require the edit to depend on the particular audio clip. The only claimed use of audio-specific information is intensity/volume, which is shown qualitatively. A within-category audio variability test would directly measure whether the model's output changes when the audio changes while the category is held fixed. If it does not, the central claim that sound is a brush—with information text cannot provide—is not supported, and the method's contribution is largely a text-conditioned editor with an audio category recognizer. This is a load-bearing concern because it targets the paper's central claim and its stated motivation, and it can be settled by a concrete, relatively inexpensive experiment. The verdict should remain conditional, but with the explicit condition that the authors provide such evidence.","tokens_in":15248,"tokens_out":8068,"duration_ms":80629,"concrete_test":"Run a within-category audio discrimination test. Fix 10 source images and 5 categories (e.g., dog barking, rain, thunder, fireworks, footsteps). For each source image and category, sample 8 audio clips from the VGGSound test split within that category, with measurably different acoustics (e.g., volume, tempo, timbre), and generate edits with SoundBrush using identical seeds. Measure the mean pairwise LPIPS distance among the 8 edited images. As a control, generate 8 edits from the same single audio clip with 8 different random seeds and measure the mean pairwise LPIPS. If the within-category audio variance is not significantly larger than the seed variance (e.g., paired t-test, p<0.05), the model is ignoring audio-specific information and relying on the category label.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that audio alone is a sufficient conditioning signal for content-preserving visual editing, and that sound carries information text lacks (e.g., intensity). However, the entire training and evaluation pipeline is organized around VGGSound category labels. The fully synthetic subset generates target prompts from the sound category keyword (e.g., 'thunder'); the real subset pairs audio with an image whose category label is known. The mapping network is trained to map CLAP audio features to text tokens, and the diffusion prior is InstructPix2Pix, which already knows how to edit from text. It is therefore entirely possible that the model learns the shortcut: map any audio clip to a text token encoding its category name, then rely on the text-conditional prior. This would produce high AVS/TVS scores (because the audio and edited image share the category) and high IIS (preservation), exactly as reported. The only evidence against this shortcut is the qualitative volume experiment (Figs. 4, S3), which is not statistically tested and could be cherry-picked. No metric in the paper requires that two different audio clips from the same category (e.g., two different 'dog barking' sounds) yield different edits. Thus the distinctive part of the contribution—using audio as a brush rather than a category label—is unverified. If the shortcut is real, the method reduces to a text-prompt editor whose prompt is the category name, and the claimed advantage over text-based editing and the stated motivation collapse.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SoundBrush proposes a sound-conditioned image editing model that maps CLAP audio features through a learned mapping network into CLIP text-token space, feeding them into an InstructPix2Pix latent diffusion model jointly fine-tuned with LoRA. Training data are produced by an automated pipeline: a fully synthetic subset generated from GPT-4 prompt pairs with Prompt-to-Prompt, and a real subset built from VGGSound by removing the sound source with sound source localization and LaMa inpainting. The paper reports quantitative comparisons (AVS, IIS, TVS, FID), a user study, ablations, and a 3D extension via InstructNeRF2NeRF, concluding that audio alone can insert sounding objects and manipulate scene appearance while preserving original content.","tokens_in":15538,"tokens_out":3629,"duration_ms":39645,"significance":"Sound-guided editing is a timely and underexplored problem, and the paper's dataset-construction pipeline and token-level audio-to-text mapping are practical contributions. If the central claim is correct, the method would go beyond style-only sound editors and demonstrate a useful form of cross-modal control. The paper also includes a user study and ablations that broadly support the qualitative behavior. However, the evidence is currently not strong enough to establish the distinctive claim that the method uses sound as a brush rather than as a category label: no experiment varies audio within a category, and the headline metrics overlap with the training objective and data filtering criteria.","major_comments":[{"comment":"The central claim that \"sound is used as a brush\" is not distinguished from a category-label shortcut. All evaluation and training material is organized around VGGSound categories: the synthetic target prompts are generated from sound category keywords, the real subset is labeled with the same categories, and the AVS/TVS metrics reward category-level agreement between audio and edited image. No experiment tests whether two different audio clips from the same category (e.g., two different dog barking sounds) produce different edits. The volume experiments in Fig. 4 and Fig. S3 are the only evidence against the shortcut, but they are qualitative and not statistically tested. I request a controlled same-category experiment: multiple audio clips per category, with a quantitative measure of edit diversity (e.g., pairwise LPIPS/CLIP distance between edited images, or a human study where raters judge whether the edited images plausibly match each specific audio clip). Without such a test, the method may reduce to a text-prompt editor whose prompt is the category name.","section":"Experiments"},{"comment":"The headline AVS metric is partly circular. The fully synthetic subset is filtered by requiring ImageBind audio-visual similarity between the target image and the sound, and the real subset is filtered using the same ImageBind similarity to discard inpainted images that still contain the sound source; evaluation then measures AVS in ImageBind space between the input audio and the edited image. Because the training target images were selected to score highly on this exact metric, high AVS values are partially by construction. Please report AVS on evaluation data that was not filtered with ImageBind, or use a different held-out metric (e.g., human ratings of audio-image correspondence). In addition, the InfoNCE loss in Eq. (2) explicitly pulls the audio-token-derived CLIP features toward the ground-truth image's CLIP visual features, while IIS measures CLIP similarity between edited and ground-truth images; the paper should explain why IIS is not redundantly inflated by this training objective.","section":"Dataset construction"},{"comment":"The 3D editing claim is supported only by qualitative examples (Fig. 6 and Fig. S5). Unlike the 2D experiments, no quantitative metric or user evaluation is provided for 3D consistency or content preservation. Because the paper's abstract and introduction explicitly claim that the framework extends to 3D scene editing, this is a load-bearing part of the contribution. Please add at least a quantitative multi-view consistency measure (e.g., per-view IIS/AVS averages, or a user study on 3D samples), or clearly reframe the 3D results as a preliminary demonstration.","section":"3D experiments"},{"comment":"The real-data subset depends on the correctness of sound source localization and LaMa inpainting, and the paper itself acknowledges that human-related sound edits show low quality due to dataset bias (Discussion, Fig. 7). This limitation is not quantified. I ask for a per-category analysis of training-data quality (e.g., fraction of pairs passing each filtering threshold, and failure rates of SSL/inpainting per category) and a discussion of how these failures affect the final editing behavior. This is important because the supervised training signal is only as good as the automatically constructed before/after pairs.","section":"Discussion"}],"minor_comments":[{"comment":"There is a typo \"soley\" in the paragraph before Eq. (2); also Eq. (3) writes the L1 regularization term as \"λℓ1 |V A|1\", which should be parenthesized for clarity.","section":"Method"},{"comment":"The text says \"further increasing to ten (C) begins to degrade performance,\" but in Table 1 the ten-token configuration is labeled (D), while (C) is the five-token configuration with LNCE. Please correct the label references.","section":"Experiments"},{"comment":"The user study reports means without error bars or statistical significance testing. Given the strong claim that SoundBrush \"significantly outperforms\" other methods in Q3, please provide confidence intervals and a paired significance test.","section":"Experiments"},{"comment":"The filtering description in the Appendix states that pairs are discarded if \"the feature similarity between the audio and the inpainted image exceeds that between the audio and the before-inpainted image\"; this wording is ambiguous because the inpainted image is the before-editing image in the real subset. Please clarify which two images are being compared.","section":"Appendix"},{"comment":"The caption of Fig. 5 contains a typo, \"comaprison\"; also, the table in Fig. 5(a) is not formatted as a proper LaTeX table in the text, which makes the numeric comparison harder to read.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core idea is attractive, but the load-bearing claim of audio-specific, category-independent editing is not yet established by the experiments. The same-category test and the de-circularization of the AVS metric are essential before publication. I do not see evidence of deliberate overclaiming; the limitations discussion is candid. If the authors can supply the requested experiments, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SoundBrush is a real step forward for sound-guided editing, but the headline claim that sound acts as a brush rather than a label is not yet backed by the evidence. The model does what it says: it inserts sounding objects and adjusts scenes while preserving structure, and the 3D extension works as a pipeline swap. The dataset construction is substantial — 83k pairs combining synthetic prompt pairs and real inpainted pairs — and the ablations are coherent. The user study is small but supports the main capability.\n\nThe soft spots are real. The quantitative metrics overlap with the training objectives: the InfoNCE loss pushes audio-token features toward CLIP image features of the target, and the AVS metric uses ImageBind similarity between audio and edited image. That is not fatal, but it inflates the numbers. More importantly, the stress-test concern lands: no experiment shows that two different audio clips from the same category produce different edits. The volume experiments in Fig. 4 and S3 are suggestive and align with the authors' claim, but they are not statistically tested and could be cherry-picked. The category-label shortcut would explain the reported AVS/TVS scores. The paper acknowledges some limitations (object placement, human-related sounds) but does not address this shortcut directly.\n\nThe core construction — mapping CLAP features into CLIP text tokens with LoRA on InstructPix2Pix — is not radically new; AudioToken and GlueGen do similar audio-to-text-token mapping, and InstructAny2Pix is the editing baseline. The new parts are the dataset, the object-insertion behavior, and the 3D extension. The paper is honest about its dataset's biases. No code or data is released, which is a problem for verification.\n\nFor a serious referee: yes. The paper deserves review. The method is plausible, the empirical claim is testable, and the dataset is a contribution. A referee should push for (1) a same-category, different-audio test, (2) release of code or data, and (3) metrics that do not overlap with training objectives.","headline":"SoundBrush is a credible sound-guided editor with a substantial new dataset, but the 'sound as brush' claim is undercut by evaluation metrics that overlap with training and no test of same-category audio variation.","tokens_in":16091,"tokens_out":2661,"would_cite":true,"duration_ms":23354,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SoundBrush claims that audio alone can serve as a control signal for editing existing visual scenes, both 2D images and 3D scenes, without a text prompt.","keywords":["sound-guided image editing","latent diffusion model","audio-to-token mapping","multimodal scene editing","3D scene editing","audio-visual dataset","object insertion","audio-visual alignment"],"falsifier":"Take a large set of real images paired with sounds whose visible sources are human-annotated, run SoundBrush to insert the sounding object, then measure the spatial overlap between the inserted object and the annotated source location; if the model places the object at the true location no more often than chance, the claim that it accurately inserts sounding objects while preserving content is refuted.","tokens_in":15039,"feed_emoji":"🔊","tokens_out":4836,"duration_ms":41447,"temperature":0.7,"pith_summary":"SoundBrush claims that audio alone can serve as a control signal for editing existing visual scenes, both 2D images and 3D scenes, without a text prompt. The paper builds a large paired dataset of audio and before/after image pairs using off-the-shelf tools, then trains a mapping network that turns sound features into tokens in the textual embedding space of a latent diffusion model. If this works, sound becomes a brush for adding or changing visible objects and scenery while leaving the original image structure intact, something previous sound-guided methods did not achieve. The paper also reports that the model captures sound intensity and that it extends to 3D editing through a NeRF-based pipeline.","feed_headline":"Audio alone now edits images and 3D scenes","feed_subtitle":"SoundBrush turns a sound clip into scene edits, preserving the original image and structure.","key_machinery":"The central object is the audio-token mapping network, a Transformer that takes CLAP audio features and a set of learnable query tokens and outputs five continuous tokens in the textual vocabulary of the CLIP text encoder. These tokens are passed through the text encoder to form the conditioning input for InstructPix2Pix's diffusion denoiser. The work it does is to turn an audio waveform into a text-aligned conditioning vector that the diffusion model already knows how to use, with the InfoNCE loss aligning the token-derived features to the target image's CLIP features and an ℓ1 penalty keeping the tokens well-distributed.","core_discovery":"The central claim is that by learning to translate audio features into the text-token space of a latent diffusion model, a network can edit images so that the visual outcome matches the sound: inserting the sounding object (e.g., a dog or train) or altering the whole scene (e.g., rain) while preserving the original content and structure. The authors construct a dataset of 83,614 audio-paired editing samples, half synthetic and half real, and train a mapping network with a denoising objective plus an InfoNCE loss that aligns the audio-derived tokens with the visual features of the ground-truth edited image. They also adapt the diffusion model with low-rank adaptation to strengthen object insertion. The result is a model that, given an image and an audio clip, produces an edited image without any text instruction.","pith_inferences":["If the mapping network transfers across domains, the same audio-to-token module could be attached to other conditional diffusion models, such as video or multi-view generators, without retraining the whole backbone.","The volume-sensitivity finding suggests that audio-conditioned editors could be used to control not only what appears but how strongly it appears, offering a dial-like interface for scene attributes.","A stiffer test of the core claim would be to measure object insertion location against known sound source positions; the failure cases shown (train in the sky) suggest spatial grounding is the main unresolved weakness.","One could test whether the model generalizes to sounds outside the training categories; if the token space remains semantic, it should edit with unseen sounds, but if it overfits to categories, it will fail."],"forward_implications":["Editing with sound becomes a supervised, training-based task rather than a generation-only one, so any pre-trained text-edit diffusion model can be re-targeted to audio.","Users can edit images by example sounds, including intensity changes that text prompts cannot capture, such as light versus heavy rain.","The same audio-conditioned editor can be plugged into NeRF-based pipelines to edit 3D scenes in a sound-consistent way.","The dataset construction pipeline provides a way to create audio-visual editing supervision without manual annotation."],"supporting_citations":[{"why":"Supplies the latent diffusion backbone that SoundBrush extends with audio tokens.","marker":"(Rombach et al. 2022)"},{"why":"InstructPix2Pix, the text-edit model whose conditioning and training objective are adapted; load-bearing as the base architecture.","marker":"(Brooks, Holynski, and Efros 2023)"},{"why":"Prompt-to-Prompt is used to generate the fully synthetic before/after image pairs from paired prompts.","marker":"(Hertz et al. 2023)"},{"why":"Sound source localization is used to mask the sounding object in real images for the real-data subset.","marker":"(Park, Senocak, and Chung 2024)"},{"why":"LaMa inpainting removes the masked sounding object to create the before-edit image.","marker":"(Suvorov et al. 2022)"},{"why":"CLAP audio encoder produces the audio features input to the mapping network.","marker":"(Wu et al. 2023)"},{"why":"ImageBind is used to filter the dataset by audio-visual similarity and to compute the A VS evaluation metric.","marker":"(Girdhar et al. 2023)"},{"why":"InstructNeRF2NeRF is the pipeline used for 3D editing, with SoundBrush replacing InstructPix2Pix.","marker":"(Haque et al. 2023)"}],"fun_headline_variants":["Sound becomes a brush for editing images","Hear a sound, edit a scene with SoundBrush","Audio-only editing: SoundBrush matches sound to scenes","Turn audio into visual edits: SoundBrush","SoundBrush: sound-guided image and 3D scene editing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatically generated before/after image pairs are assumed to be faithful ground truth for what the audio implies; if the synthetic prompt pairs or real-data inpainting introduce spurious changes, the model learns the wrong audio-to-edit mapping.","fun_headline_variants_meta":{"raw":{"variants":["Sound becomes a brush for editing images","Hear a sound, edit a scene with SoundBrush","Audio-only editing: SoundBrush matches sound to scenes","Turn audio into visual edits: SoundBrush","SoundBrush: sound-guided image and 3D scene editing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":2113,"prompt_tokens":856,"completion_tokens":1257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1197}},"tokens_in":472,"tokens_out":1257,"duration_ms":9613,"temperature":1.0,"reasoning_tokens":1197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:45:38.567061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a large set of real images paired with sounds whose visible sources are human-annotated, run SoundBrush to insert the sounding object, then measure the spatial overlap between the inserted object and the annotated source location; if the model places the object at the true location no more often than chance, the claim that it accurately inserts sounding objects while preserving content is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion backbone that SoundBrush extends with audio tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InstructPix2Pix, the text-edit model whose conditioning and training objective are adapted; load-bearing as the base architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prompt-to-Prompt is used to generate the fully synthetic before/after image pairs from paired prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sound source localization is used to mask the sounding object in real images for the real-data subset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LaMa inpainting removes the masked sounding object to create the before-edit image."},{"cited_title":"V.; Joulin, A.; and Misra, I","cited_arxiv_id":null,"evidence_quote":"ImageBind is used to filter the dataset by audio-visual similarity and to compute the A VS evaluation metric."},{"cited_title":"A.; Holynski, A.; and Kanazawa, A","cited_arxiv_id":null,"evidence_quote":"InstructNeRF2NeRF is the pipeline used for 3D editing, with SoundBrush replacing InstructPix2Pix."}],"review_version":1}