{"id":"b4521000-b3f5-4d93-aa7b-d16cb3438ba1","arxiv_id":"2505.12154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A transformer model learns to transform poorly mixed audio into well-balanced audio guided by video content, trained on a new pseudo-data set derived from movies.","lead":"This paper introduces a new task: automatically rebalancing or 'highlighting' audio in a video by using the video's visual content as a guide. The authors train a transformer-based model on movie clips, where the original movie mix serves as the target, and show it outperforms baselines on a synthetic 'poorly mixed' benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative gains are measured against synthetic augmentation targets that an audio-only model can invert; video guidance adds little in the paper's own no-context ablation, so the real-world visual highlighting claim is not yet established.","rationale":"The reader's weakest assumption is that the synthetic poor-mixing process represents real-world poorly mixed audio. My concern is closely related but more specific: even on the synthetic benchmark, the evaluation is confounded because the target is the pre-augmentation movie audio, which can in principle be recovered by inverting the known stem gains without any visual reasoning. The paper's own no-context ablation shows that most of the improvement is attainable without video or text context, and the remaining gap is small and unquantified. Section 15's stated limitations and the failure case in Fig. 7 reinforce that the synthetic pipeline does not cover the dominant-source imbalance that occurs in real recordings. This does not invalidate the paper as a synthetic remixing benchmark, but it does mean the central claim of 'watching movies' -- i.e., visual semantics driving the highlighting -- is not yet demonstrated by the quantitative results. The existing CONDITIONAL verdict is appropriate; the paper should be asked to release code/data, report uncertainty, and provide a real-world evaluation that isolates the contribution of visual context. My concrete test directly isolates that contribution in the setting the paper itself proposes, and would settle whether the visual guidance claim survives contact with real footage.","tokens_in":16860,"tokens_out":5683,"duration_ms":68945,"concrete_test":"Run a paired, forced-choice listening test on at least 30 unedited real web or egocentric videos (the 'Real Web Video Refinement' setting from the supplementary material), comparing video-conditioned VisAH against its own no-context variant from Table 2, with participants asked which output better matches the visually salient content. If video-conditioned output is not chosen significantly above chance (e.g., 95% binomial CI excluding 50%), the central visually-guided highlighting claim lacks support. Complement this with bootstrap confidence intervals over clips for the Table 2 metrics to check whether the reported context-vs-no-context gaps are statistically distinguishable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim rests on a test set generated by the same three-step pipeline used for training (Sec. 4): the poorly mixed input a is produced from the ground-truth s by applying known {+12,+9,+6,-6,-9,-12} dB gains to the speech, music, and sound-effect stems, and all five metrics in Table 1 measure distance to that same s. The intended target is therefore recoverable from a by inverting the augmentation with a source separator, without needing visual semantics. Table 2 supports this concern: the no-context VisAH already achieves MAG 10.35 vs 10.08 for temporal-text guidance and KLD 11.95 vs 11.01, a modest gap with no error bars or significance test. The real-world claims rest on qualitative demos and the MovieGen example; the paper's own Section 8 (Fig. 7) shows a waterfall-dominated real clip that VisAH fails to rebalance, and Section 15 concedes the separation/adjustment/remixing strategy as a limitation. Thus the paper demonstrates an effective audio-restoration system for its synthetic augmentation, but not that VisAH learns visually grounded highlighting that transfers to non-synthetic poor mixes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, visually-guided acoustic highlighting (L2H), in which poorly mixed audio is transformed into a well-balanced mix using the accompanying video as guidance. The authors propose VisAH, a transformer-based dual U-Net architecture that takes both spectrogram and waveform inputs, encodes contextual information (video frames or text captions) with transformer encoders, and uses a transformer decoder with cross-attention to output highlighted audio. A new dataset, the Muddy Mix Dataset, is built from the Condensed Movie Dataset by separating movie audio into speech, music, and sound effects, applying random loudness gains/suppressions, and remixing. The method is evaluated on a held-out test set against several baselines using five metrics (MAG, ENV, KLD, delta-IB, Wasserstein distance), with ablations on context type, number of transformer encoder layers, and dataset difficulty, plus a small subjective ranking test. The central claim is that VisAH consistently outperforms baselines in both quantitative and subjective evaluation.","tokens_in":17143,"tokens_out":3911,"duration_ms":42583,"significance":"If the central claim holds, the paper proposes a genuinely new audio-visual task and a practical data-driven approach to audio rebalancing that extends prior music-only remixing work to general movie and video content. The Muddy Mix dataset and the separation-adjustment-remix pseudo-data pipeline are useful contributions, and the paper's systematic ablations (context types, encoder depth, difficulty levels) are a strength. The authors also provide a project page with qualitative examples and a downstream application to video-to-audio generation refinement. However, the evidence that the model actually learns visually guided highlighting is currently not as strong as the abstract claims: the quantitative evaluation is performed on a synthetic test set generated by the same pipeline used for training, and the no-context ablation is close to the full model. The small subjective test (9 participants, 10 videos) further limits the strength of the human-evaluation claim.","major_comments":[{"comment":"The main quantitative evaluation is performed on a test set constructed by the same three-step pipeline (separation, adjustment, remixing) that generates the training data, with the original movie audio as ground truth and known per-stem loudness gains. Because the degradation is an invertible, stem-level gain change and the source separator is the same one used in training-data creation, an audio-only model can learn to approximately undo the augmentation without any visual understanding. Table 2 indeed shows that the no-context VisAH model achieves MAG 10.35 vs. 10.08 for temporal-text guidance and KLD 11.95 vs. 11.01, a modest gap with no error bars. The paper therefore demonstrates effective audio restoration on a synthetic augmentation, but not yet that VisAH learns visually guided highlighting that transfers to non-synthetic poor mixes. I request either an evaluation on real-world poorly mixed audio with a reference or human study, or a test design in which the target highlighting is not recoverable from the input audio alone.","section":"Section 4 and Table 1"},{"comment":"All quantitative results are reported as single point estimates without error bars, confidence intervals, or significance tests. This is load-bearing because several key comparisons have small margins: in Table 2, temporal text guidance improves over no-context by only 0.27 MAG, 0.94 KLD, and 0.19 delta-IB, and in Table 3, vision context with 6 layers is worse than with 3 layers. Without variance estimates, the claims that contextual guidance matters and that VisAH significantly outperforms baselines are not statistically supported. Please report results over multiple training seeds or bootstrap confidence intervals, and use a paired significance test for the subjective ranking data.","section":"Tables 1-4 and Section 5.3"},{"comment":"The real-world applicability claims rest on qualitative demonstrations. Section 8 shows a real waterfall-dominated clip where VisAH fails to rebalance speech against the waterfall, and Section 15 explicitly lists the separation-adjustment-remix strategy as a limitation. The application to web videos and to MovieGen-generated audio is only shown qualitatively on the project page. Given that the quantitative benchmark is synthetic, the claim in the abstract and Section 6 that the method \"outperforms several baselines in both quantitative and subjective evaluation\" and generalizes to real content needs either a real-world benchmark with quantitative or human evaluation, or a clearly scoped statement that real-world transfer is a preliminary qualitative finding.","section":"Section 8, Section 15, and project page"}],"minor_comments":[{"comment":"There are several typos, including \"mutimodal\" in the contributions list and \"generes\" in Section 4; please proofread the manuscript.","section":"Throughout"},{"comment":"The MR-STFT loss is used in the main text but defined only in the supplementary material; please define it briefly in the main text or move the definition there.","section":"Section 3.2.3, Eq. (5)"},{"comment":"The text cites the \"Cinematic Sound Demixing Challenge\" but reference [8] is titled \"The Sound Demixing Challenge 2023 - Music Demixing Track\"; please verify the citation or add the correct reference for the cinematic demixing challenge.","section":"Section 4, reference [8]"},{"comment":"The statement that metrics show continuous improvement as difficulty decreases is not supported by Table 4: Low-P has worse MAG (9.55) than Moderate-P (8.73) and worse W-dis (0.80) than Moderate-P (0.65), so the trend is not monotonic across all metrics.","section":"Table 4 and Section 5.3.2"},{"comment":"The breathing example described as a failure actually demonstrates behavior aligned with the visual context (close-up of a breathing face) while diverging from the movie ground truth; the caption should clarify that this is a divergence from the particular movie mix, not a failure to follow visual guidance.","section":"Section 8, Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The core concern is evidentiary, not about the validity of the architecture: the method may well work, but the current evaluation does not isolate visually guided highlighting from audio-only restoration, and the real-world transfer claims are qualitative. If the authors add a real-world evaluation with human or objective references, report variance estimates, and temper the generalization claims, I would support acceptance. The paper is within the journal's scope as an audio-visual learning contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this paper introduces a genuinely new task, visually-guided acoustic highlighting, and a dataset, Muddy Mix, built from movie clips via a pseudo-data pipeline that separates, adjusts, and remixes speech, music, and effects. That part is solid and useful. The VisAH model, a HybridDemucs-style dual U-Net with a transformer latent module, is a reasonable assembly of known parts. On the synthetic test set it beats the baselines by a wide margin on all five metrics.\n\nThe soft spot, and it is a real one: the quantitative evidence does not actually show that visual guidance drives the improvement. The input \"poor mix\" is generated from the ground truth by applying known gain offsets to separated stems, so a model that learns to restore loudness from audio alone can get most of the way back. The paper's own Table 2 shows the no-context baseline reaches MAG 10.35 vs 10.08 with temporal text guidance and KLD 11.95 vs 11.01, a small gap, with no error bars or significance tests. The subjective test is tiny (9 participants, 10 videos) and the real-world claims rest on a few qualitative demos, including a failure case in Fig. 7. The authors are transparent about limitations in Section 15, which helps their credibility, but it does not close the gap.\n\nWhat the paper does well: it names a task that is worth naming, builds a dataset that could anchor future work, and reports ablations that let a reader see where the gains come from. The architecture details are mostly clear, and inference is fast enough for practical use.\n\nWho this is for: people working on audio-visual learning, automatic audio post-production, or video editing tools. They will find the task definition and dataset valuable even if the visual-conditioning story needs much stronger evidence. I would accept this for peer review because the task and resource are serious, but I would push the authors to (a) report error bars, (b) test on real poorly-mixed footage with a larger subjective study, and (c) show a baseline that explicitly inverts the synthetic augmentation, to isolate what video actually contributes.\n\nRecommendation: engage with it, but read Table 2 carefully before citing it as evidence for visually-guided highlighting.","headline":"New task and dataset are real contributions; the visual-guidance claim is thinner than the paper suggests, because the synthetic test can be largely passed by audio-only restoration.","tokens_in":17642,"tokens_out":1884,"would_cite":true,"duration_ms":18909,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Movies teach an AI to rebalance audio using the video as a guide.","keywords":["visually-guided acoustic highlighting","audio remixing","audio-visual learning","movie supervision","pseudo-data generation","transformer","source separation","video-to-audio refinement"],"falsifier":"Take a collection of genuinely raw recordings known to be poorly mixed (amateur vlogs, phone footage with wind or crowd noise), have audio engineers produce a reference rebalanced mix for each, and compare VisAH's output against both the input and an audio-only remixer on waveform distance and audio-visual alignment. If VisAH fails to beat the input or matches the audio-only baseline on these real recordings, the central claim that video guidance learned from movies transfers to real-world footage is refuted.","tokens_in":16700,"feed_emoji":"🎬","tokens_out":7653,"duration_ms":75485,"temperature":0.7,"pith_summary":"The paper introduces visually-guided acoustic highlighting: given a video whose audio track is unbalanced (dialogue drowned by waves, music swelling over speech), the goal is to automatically rebalance the audio so that what the viewer sees emphasized is also what is heard clearly. The central claim is that this can be learned by watching movies, because professionally mixed film audio already encodes the intended highlighting, and the authors make this learning signal usable by deliberately ruining good mixes. They separate movie audio into speech, music, and sound-effect stems, apply random boosts and cuts of 6–12 dB, and remix to create 'poorly mixed' inputs paired with the original as ground truth. They then train a transformer-based model, VisAH, which uses the video (or its text captions) as context to undo the damage, and report that it outperforms audio-only remixing baselines on waveform, semantic, and time-alignment metrics and in a human ranking test.","feed_headline":"AI learns audio remixing by watching movies","feed_subtitle":"A transformer model, trained on deliberately ruined film sound, rebalances speech, music, and effects from the video's cues.","key_machinery":"The load-bearing mechanism is the separate–adjust–remix pseudo-data generation process. A cinematic source-separation model splits each movie clip's ground-truth audio into speech, music, and sound-effects stems (plus a residual so the stems sum exactly to the original); the loudest stem is suppressed and the others boosted by randomly chosen amounts among $\\{-12,-9,-6\\}$ dB and $\\{+6,+9,+12\\}$ dB, and the stems are added back to form the poorly mixed input. The model that must undo this is VisAH: a dual U-Net whose magnitude-spectrogram and waveform encoders produce a shared latent representation, a latent highlighting transformer whose decoder cross-attends over temporal context (per-frame vision features or embedded frame captions, each passed through its own transformer encoder), and a zero-initialized $1\\times 1$ convolution so the network's output starts as no change and is added as a residual offset to the latent audio features. Training uses a multiscale STFT $\\ell^1$ loss between predicted and ground-truth audio.","core_discovery":"On its own terms, the paper establishes a new audio-visual task and a working solution: learn a mapping $(a, v) \\mapsto s$ from poorly mixed audio and its accompanying video to a highlighted audio signal that preserves the content of $a$ while rebalancing speech, music, and sound effects in line with what the video emphasizes. The enabling discovery is that movie audio, fetched from a large condensed-movie corpus, provides free ground-truth highlighting, and that a synthetic pipeline—imperfect separation into three stems, loudness adjustment with amounts from $\\{ \\pm 6, \\pm 9, \\pm 12 \\}$ dB, then linear remixing—produces input/output pairs that let a model learn the transformation. The paper's VisAH model combines a dual U-Net audio backbone (spectrogram and waveform branches) with a latent transformer that attends over temporal context, either per-frame vision features or text-caption embeddings, and predicts an offset added back to the original features so content is preserved. Evaluations show consistent improvement over three baselines across five metrics, with temporal context mattering more than single-frame context and text captions giving the best guidance; a small human study ranks VisAH first or second in 77% of cases, and in about a third of cases above the original movie audio.","pith_inferences":["The movie-trained model likely encodes Hollywood-style mixing aesthetics; presenting it with video from different cultures or genres, or with shuffled video–audio pairs, would reveal how much of the highlighting is driven by genuine visual semantics rather than a learned loudness prior.","Because the guidance channel is flexible, the same architecture could be steered by an explicit text prompt (such as 'focus on the music' or 'make the crowd recede'), turning highlighting into a controllable semantic editing tool that goes beyond video-only conditioning.","The separate–adjust–remix pipeline could be applied recursively or with multiple separators, as the paper itself notes as future work, letting the approach scale to any professionally mixed audio-visual content without new manual annotation."],"forward_implications":["If the central claim holds, any amateur video with lopsided audio can be automatically rebalanced by reading the video's visual emphasis, restoring dialogue or ambient sound without manual stem editing.","The same model can serve as a post-processing step for video-to-audio generation systems, measurably improving the audio-visual alignment of generated soundtracks.","Both video frames and text captions carry usable highlighting guidance, and temporal context across frames or captions matters more than a single frame, so future systems should condition on full scenes rather than stills.","The pseudo-data generation recipe (imperfect separation, loudness adjustment, linear remixing) yields a large training set of 15,078 movie clips, and the paper's difficulty analysis suggests performance scales with how far the input is from the target mix."],"supporting_citations":[{"why":"Supplies the condensed-movie corpus from which all training, validation, and test clips are drawn.","marker":"[2]"},{"why":"Supplies the dual spectrogram-plus-waveform U-Net design that the audio backbone extends.","marker":"[7]"},{"why":"Defines the cinematic sound demixing challenge setup, including three-stem categories and the loudness statistics used in the empirical baseline.","marker":"[8]"},{"why":"Separates movie audio into speech, music, and sound-effects stems, the first step of the pseudo-data pipeline.","marker":"[65]"},{"why":"Supplies per-frame video features via a contrastively trained vision-language model that the temporal transformer encodes as visual context.","marker":"[48]"},{"why":"Generates per-frame captions from video frames, providing the text-guidance modality.","marker":"[6]"},{"why":"Embeds the generated captions into token sequences for the text transformer encoder.","marker":"[49]"},{"why":"Provides the end-to-end neural remixing baseline that VisAH is compared against.","marker":"[69]"},{"why":"Provides the text-guided sound-scene editing baseline, given captions as guidance.","marker":"[26]"},{"why":"Supplies the multiscale STFT loss that trains the model.","marker":"[68]"}],"fun_headline_variants":["Movies teach AI to fix muddy sound mixes","Visual cues guide audio highlighting in films","Transformer rebalances film sound using visual guidance","Movie-watching AI learns to rebalance audio","Movies provide free labels for audio highlighting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that the synthetic ruin-and-restore procedure (separate, adjust, remix) produces inputs that represent how real videos are actually poorly mixed, so that a model trained on those pairs transfers to genuine recordings; the paper validates real-world behavior only qualitatively.","fun_headline_variants_meta":{"raw":{"variants":["Movies teach AI to fix muddy sound mixes","Visual cues guide audio highlighting in films","Transformer rebalances film sound using visual guidance","Movie-watching AI learns to rebalance audio","Movies provide free labels for audio highlighting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000944,"raw_usage":{"total_tokens":4074,"prompt_tokens":1026,"completion_tokens":3048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2983}},"tokens_in":642,"tokens_out":3048,"duration_ms":18941,"temperature":1.0,"reasoning_tokens":2983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:39:48.323858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a collection of genuinely raw recordings known to be poorly mixed (amateur vlogs, phone footage with wind or crowd noise), have audio engineers produce a reference rebalanced mix for each, and compare VisAH's output against both the input and an audio-only remixer on waveform distance and audio-visual alignment. If VisAH fails to beat the input or matches the audio-only baseline on these real recordings, the central claim that video guidance learned from movies transfers to real-world footage is refuted.","supporting_citations":[{"cited_title":"Condensed movies: Story based retrieval with con- textual embeddings, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the condensed-movie corpus from which all training, validation, and test clips are drawn."},{"cited_title":"Hybrid spectrogram and waveform source separation","cited_arxiv_id":null,"evidence_quote":"Supplies the dual spectrogram-plus-waveform U-Net design that the audio backbone extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the cinematic sound demixing challenge setup, including three-stem categories and the loudness statistics used in the empirical baseline."},{"cited_title":"Re- mastering divide and remaster: A cinematic audio source separation dataset with multilingual support","cited_arxiv_id":null,"evidence_quote":"Separates movie audio into speech, music, and sound-effects stems, the first step of the pseudo-data pipeline."},{"cited_title":"Don’t separate, learn to remix: End-to-end neural remixing with joint optimization","cited_arxiv_id":null,"evidence_quote":"Provides the end-to-end neural remixing baseline that VisAH is compared against."},{"cited_title":"Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discrimi- nators","cited_arxiv_id":null,"evidence_quote":"Supplies the multiscale STFT loss that trains the model."}],"review_version":1}