{"id":"c25d2cd7-8bdb-4ffc-8043-e2442758f109","arxiv_id":"2412.10768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A text-and-video conditioned flow transformer that generates onscreen plus offscreen audio, evaluated on a new curated benchmark and on VGGSound.","lead":"VinTAGe is a model that turns a silent video plus a text description into audio containing both onscreen and offscreen sounds, such as a dog barking while a vacuum hums. The paper also introduces a 636-pair benchmark for this 'holistic audio generation' task and reports improvements over text-only and video-only generators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'joint conditioning is necessary' claim rests on VinTAGe-Bench, a filtered synthetic two-source mix that may guarantee the result; Table 5's text-only accuracy is already higher, so the necessity claim is not yet established.","rationale":"This concern is load-bearing because the paper's headline conclusion about necessity is drawn directly from a benchmark the authors constructed and filtered. The benchmark does not represent the full difficulty of real holistic audio: it isolates one text-specified offscreen source per video, discards precisely the confusing mixtures that would stress-test a joint model, and makes the offscreen component inaccessible to the video by design. Consequently, the observed failure of text-only and video-only baselines is partly an artifact of benchmark construction rather than a discovered property of real audio-visual scenes. I still credit the paper's positive contributions: the joint architecture is plausible, the VGGSound results are competitive, and the controlled VinTAGe ablations in Table 5 do show that joint conditioning improves FID and alignment relative to either single modality within this benchmark. However, the concept-accuracy numbers in that same table complicate the necessity claim, since text-only achieves higher mean accuracy. The appropriate response is not rejection but a conditional verdict: the technical machinery may work, and the benchmark may be useful, but the central 'necessity' claim requires validation on a less curated, more realistic benchmark before it can be accepted as stated. The reader's weakest assumption points to the same benchmark-validity issue, so I agree with that assessment.","tokens_in":19114,"tokens_out":5042,"duration_ms":52752,"concrete_test":"Build an unfiltered evaluation set from the same 212 source videos but include all VGGSound offscreen candidates without removing similar-frequency or busy mixtures, or better, harvest real videos with naturally occurring offscreen ambience and annotate on/offscreen sources. Run the exact VinTAGe inference with (svis, stxt) = (2.5, 2.5) and compare against text-only (0.0, 5.0) and video-only (5.0, 0.0) on the same videos, reporting FID, alignment, and per-source accuracy. If the joint model's margin over text-only/video-only on this harder set is no larger than on VinTAGe-Bench, or if text-only accuracy remains higher, the 'necessity' claim is an artifact of the curated two-source mixing and filtering protocol.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim that joint text and visual interaction is necessary for holistic audio generation is inferred almost entirely from VinTAGe-Bench (Section 4 and Appendix C). That benchmark is built by taking VGGSound test videos, adding exactly one randomly chosen offscreen sound from a disjoint class, pressure-balancing the mix, and then discarding mixtures with similar frequencies or busy onscreen sounds. This procedure has two consequences. First, the offscreen component is by construction invisible in the video and present only in the text caption, so any model without text cannot produce it and any model without video cannot localize the onscreen source; the benchmark is thus designed so that single-modality models must fail on one of the two sub-tasks. Second, the filtering removes exactly the ambiguous cases (similar-frequency sources, busy onscreen audio) where a joint model's purported advantage would be most informative. The data therefore supports a weaker conclusion: a model that ingests both modalities can combine a visible onscreen source with a text-specified offscreen source on easy two-source mixes. It does not show that joint conditioning is necessary for holistic audio in general, especially when offscreen sound is ambient, multi-source, or partially predictable from visual context. Notably, the paper's own controlled comparison (Table 5) shows that text-only VinTAGe has higher mean concept accuracy (54.32 vs 50.66) and only modestly worse alignment, so the necessity claim depends on weighting FID/alignment over accuracy and on the chosen guidance scales, both of which were tuned on the same benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VinTAGe addresses holistic audio generation: given a silent video and a text prompt, it aims to synthesize both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with the text. The proposed model comprises a Visual-Text Encoder (CLIP visual features, optical-flow energy, frame indices, and FLAN-T5 text embeddings with gated cross-attention), a flow-matching SiT transformer with per-layer text/video cross-attention weights, and teacher-student alignment losses from pretrained text-only and video-only audio generators. The paper also introduces VinTAGe-Bench, a 636-pair benchmark built from 212 VGGSound test videos by mixing each original audio track with one randomly selected offscreen sound and filtering out mixtures that are hard to distinguish. Experiments compare against V2A, T2A, and joint text-video baselines on VinTAGe-Bench and on VGGSound, reporting state-of-the-art FAD/FID numbers and a small subjective study. The central claim is that joint text and visual interaction is necessary for holistic audio generation.","tokens_in":19418,"tokens_out":6217,"duration_ms":56116,"significance":"The paper is useful in several respects. It formulates a timely task, contributes a new benchmark (VinTAGe-Bench) that the community can reuse, and proposes a sensible architecture with teacher-guided modality balancing; the comparison set is broad, and the limitations section is candid about the difficulty of temporal alignment metrics for mixed audio. The paper also ships a demo and promises code and pretrained models. However, the central conceptual claim—that joint conditioning is necessary for holistic audio generation—is not supported by the evidence. The benchmark construction makes single-modality models fail by design on one of the two subtasks, and the paper's own controlled comparison (Table 5) shows text-only conditioning achieving higher concept accuracy than the joint model. The VGGSound state-of-the-art claim also needs qualification because VinTAGe receives text captions that the pure V2A baselines do not. The contribution would remain valuable if the claims were reframed around what is actually demonstrated: joint conditioning improves distribution-level quality and alignment on curated two-source mixtures and mitigates modality bias.","major_comments":[{"comment":"The claim that 'joint text and visual interaction is necessary for holistic audio generation' is not established by the experiments. VinTAGe-Bench is constructed so that the offscreen sound is invisible in the video and appears only in the text caption, while mixtures with similar frequencies or 'busy' onscreen sounds are removed (§4, Dataset paragraph). Single-modality models are therefore guaranteed to fail on one of the two subtasks by construction, and the ambiguous cases where joint disambiguation would be most informative are filtered out. More importantly, the paper's own controlled comparison in Table 5 shows that text-only conditioning (svis=0.0, stxt=5.0) achieves higher mean concept accuracy than the joint model (54.32 vs. 50.66), and the visual-only teacher is competitive in mean alignment. The data support a weaker conclusion: joint conditioning improves FID/FAD and mean alignment on this curated benchmark. The necessity claim should be removed or substantially qualified.","section":"Abstract, §4, Table 5"},{"comment":"The VGGSound state-of-the-art claim is not supported as stated. VinTAGe is conditioned on Auto-ACD text captions, which include the sound class label, while SpecVQGAN, Im2Wav, and Diff-Foley receive no text. Since VGGSound audio is largely determined by the class label, giving the model this text is an information advantage unrelated to joint video-text modeling. The authors update only the text+video baselines (FoleyCrafter and ReWaS) with Auto-ACD captions; the V2A baselines are not given an analogous captioning channel. The claim should be limited to joint text-video models with equalized inputs, or the comparison should include text-only and video-text variants of all baselines.","section":"§4, Table 2"},{"comment":"The subjective evaluation is too small and too poorly characterized to support the temporal-alignment and faithfulness claims. Ten listeners rated 35 samples on VinTAGe-Bench, with no confidence intervals, significance tests, or inter-rater agreement reported. The MOS gaps (for example, MOS-T 3.36 vs. 2.96 for FoleyCrafter) may be meaningful, but with this sample size and no statistics they are not load-bearing. This matters because the paper explicitly declines to use objective temporal metrics (Appendix F) and relies on subjective scores for the temporal-synchronization claim. The authors should report per-item variance, confidence intervals, and a statistical test, or weaken the corresponding conclusions.","section":"§4, Subjective Evaluation; Appendix D"},{"comment":"The classifier-free guidance scales svis and stxt are tuned empirically to 2.5 and used for all main results, but no validation protocol is described. Appendix B.3 sweeps one scale while fixing the other, yet the paper does not state whether these sweeps were performed on the test benchmark or a held-out set. With only 636 test pairs and no error bars, the reported improvements could reflect guidance-scale selection rather than the model's joint-conditioning design. The authors should describe the scale-selection procedure and, ideally, report results across a range of scales.","section":"§3.3, Appendix B.3"}],"minor_comments":[{"comment":"The sentence introducing Eq. (7) says 'stxt and svis represent the weights of visual and text guidance,' but stxt is the text weight and svis is the visual weight; the assignment is swapped.","section":"§3.3, Eq. (7)"},{"comment":"The caption states 'Top-2 results are highlighted,' but in the typeset table only one row appears highlighted; please check the formatting so the intended highlighting is visible.","section":"§4, Table 1"},{"comment":"The phrase 'for each video, we selected two off-screen categories and one no off-screen scenario from the offscreen categories' is confusing; clarify that the no-offscreen scenario is a caption-only condition without a mixed offscreen audio track.","section":"Appendix C"},{"comment":"The dimensions of cv and ct are not fully specified: ct has length L (token count), cv has length N (frame count), and the final MLP projection size is given only in Appendix A.1. Stating these dimensions in the main text would improve readability.","section":"§3.2, Gated Cross-Attention"},{"comment":"References [9] and [10] appear to be the same Lumina-T2X paper, and the author list in [10] contains an empty entry ('Chris Liu, ,'); these should be merged and corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the audio-generation community, and I would not reject it: the benchmark and the method are useful contributions. My main concern is the gap between the abstract's 'necessity' claim and the evidence, especially the paper's own Table 5, which shows text-only conditioning with higher concept accuracy. The authors should reframe the central claim, qualify the VGGSound comparison, and add statistical support for the subjective results. If these changes are made, the paper could become acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on conditional audio generation. The model is a solid integration of known pieces, and the benchmark is described transparently, but the headline claim that joint conditioning is necessary for holistic audio does not hold up against their own numbers.\n\nThe architecture is the strongest part. Gated cross-attention between CLIP video features and FLAN-T5 text, per-layer weights on modality cross-attention, and teacher alignment losses from uni-modal T2A/V2A models are a sensible way to attack modality bias. The ablations show each component moves the metrics, and the VGGSound results are competitive. The benchmark construction is also careful: filtering criteria, pressure-balanced mixing, and caption shuffling are all documented. That transparency is worth something.\n\nThe soft spot is the necessity claim. VinTAGe-Bench mixes one visible onscreen source with one randomly chosen offscreen source that is by construction invisible in the video and only present in the text. Any video-only model is structurally blocked from the offscreen sub-task; any text-only model is blocked from the onscreen sub-task. Filtering out similar-frequency and busy mixtures removes the cases where joint reasoning would actually matter. Their own Table 5 makes this worse: text-only VinTAGe gets higher mean concept accuracy (54.32 vs 50.66). The joint model wins on FID and alignment, but that is partly a function of the guidance scales they tuned on the same benchmark. So the data supports a weaker conclusion: a model with both modalities can combine a visible source and a text-specified offscreen source on easy two-source mixes. That is still useful, but it is not 'necessity' in general.\n\nThe evaluation has smaller issues. Ten listeners and 35 samples is a thin subjective study, and there are no error bars anywhere. The authors already flag in Appendix F that mean optical flow may not handle complex multi-source scenes and that no objective temporal alignment metric exists for mixed audio. So the main concerns are acknowledged in the paper, just not fully addressed.\n\nRecommendation: send it to a serious referee. The claims need to be toned down and the benchmark validated on unfiltered or external data, but the model is a real step for joint audio generation and the benchmark, if released, could be useful.","headline":"Useful joint video-text audio model and a transparent benchmark, but the 'necessity' claim is weaker than the paper's own Table 5 suggests.","tokens_in":19947,"tokens_out":2779,"would_cite":false,"duration_ms":26076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VinTAGe's claim: to hear a scene completely — on-screen foley and off-screen ambience — audio generation must be grounded in both the video and a text prompt.","keywords":["holistic audio generation","joint text-video conditioning","video-to-audio generation","text-to-audio generation","flow matching","modality bias","audio-visual synchronization","VinTAGe-Bench"],"falsifier":"Take uncurated real-world clips with natural multi-source soundscapes — several off-screen sounds at once, overlapping frequency bands, and multiple moving on-screen objects, the regime the paper's own limitations section flags — and compare VinTAGe against a strong text-to-audio model fed a detailed caption of the video. If the text-only model matches the joint model on semantic completeness and the temporal-synchronization advantage disappears on these clips, the claim that joint video-text conditioning is necessary for holistic audio would be falsified; running the same test on the benchmark's excluded hard cases (similar-frequency or busy mixtures) would show whether the filtering drove the result.","tokens_in":18897,"feed_emoji":"🔊","tokens_out":14659,"duration_ms":118480,"temperature":0.7,"pith_summary":"This paper addresses a post-production problem: a silent video usually needs both on-screen sounds (a dog barking in frame, timed to its movements) and off-screen sounds (a vacuum cleaner humming off camera), and existing tools only produce one of the two. Text-to-audio models generate the complete soundscape but cannot lock it to what is shown; video-to-audio models sync to the image but drop what is not visible. VinTAGe claims that conditioning on video and text together, with the two modalities kept in balance, lets a single model generate both layers in one audio track that is synchronized with the video and consistent with the caption. If the claim holds, a sound designer could go from silent footage plus a written description straight to a finished mix, and the new VinTAGe-Bench (636 video-text-audio pairs of mixed on-screen and off-screen sounds) makes the task measurable for the first time. The paper further reports the best results among the compared methods on the standard VGGSound video-to-audio benchmark.","feed_headline":"Full-scene audio from one silent video: foley and ambience at once","feed_subtitle":"A single flow-based model times on-screen sounds to the image while adding off-screen sounds named in the caption.","key_machinery":"The engine is a flow-matching transformer: following flow matching, the model learns a velocity field $v_\\theta(x_t, t, c_v, c_t)$ that linearly interpolates Gaussian noise into a compressed audio latent, and at inference an ODE solver integrates the field backwards. The backbone is SiT, the Scalable Interpolant Transformer, extended into a Joint VT-SiT whose blocks interleave self-attention with two cross-attention layers, one over text tokens and one over video tokens, fused by a learned per-layer weight $\\omega_l$. The conditioning signal comes from the Visual-Text Encoder, which cross-attends CLIP frame embeddings (concatenated with sinusoidal encodings of mean optical-flow energy and frame indices) to FLAN-T5 text embeddings through zero-initialized tanh gates, and pools both modalities into a global vector applied by adaptive layer normalization. The decisive training mechanism is teacher-student alignment: batches alternate joint, text-only, and video-only inputs, and on the single-modality entries the joint model must reproduce the velocity predicted by frozen uni-modal teachers through the losses $L_t$ and $L_v$, which stops either modality from dominating. At inference, two classifier-free guidance scales $s_{\\text{vis}}$ and $s_{\\text{txt}}$ let the operator fade the output between visual and textual priorities.","core_discovery":"The central claim is that joint text and visual interaction is necessary for holistic audio generation, and that a flow-based transformer can carry it out without drifting into one modality. The paper argues that each single-condition family is structurally incomplete: text-to-audio has no access to timing, and video-to-audio has no access to what lies off-screen, so neither can produce the full sound of a scene. Prior joint models assume the text and video describe the same content, which breeds a visual bias that suppresses off-screen sounds. VinTAGe counters this with three coordinated moves: a Visual-Text Encoder whose gated cross-attention lets video and text inform each other without forcing them to agree; a Joint VT-SiT transformer in which each block learns a weight $\\omega_l$ balancing the two modalities' cross-attention; and a teacher-student scheme in which pretrained text-only and video-only generators supply velocity targets for single-modality inputs, so the joint model learns to match both teachers. On VinTAGe-Bench the model posts the best generation quality (FAD 3.05, FID 16.43) and the best mean audio-text/audio-visual alignment (16.06), and it is the joint model whose on-screen and off-screen concept accuracies stay closest together (57.7% and 43.6%), where the strongest joint rival splits 64.9% against 21.7%. The same model posts the best FID (6.65) and MKL (4.12) on VGGSound among the compared methods.","pith_inferences":["The necessity claim is only as strong as the benchmark: VinTAGe-Bench mixes exactly one on-screen source with one off-screen source drawn from VGGSound and deletes hard cases (spectrally overlapping sounds, busy on-screen audio, poorly visible objects), so a real-world soundscape with several overlapping off-screen sources is an untested regime in which the conclusion could weaken.","The paper's own limitations section (Appendix F) concedes that mean optical-flow magnitude is a coarse temporal guide that may fail when several moving objects each carry their own sound; an object-wise temporal-energy variant is the natural next test of whether fine-grained synchronization survives in complex scenes.","The two guidance scales suggest an interactive application the authors do not build: a two-knob foley-versus-ambience fader that a sound editor could adjust per shot, exploiting the classifier-free guidance the model already learns.","Because on-screen temporal alignment is measured only by subjective listening (existing metrics assume single-source audio), scoring the model's own output with a neural audio separator would give an objective, scalable test of the synchronization claim."],"forward_implications":["Post-production sound for a silent clip reduces to a single pass: feed the video and a written description, and receive one mixed track containing both foley and off-screen ambience, with no manual mixing of separate stems.","Joint conditioning beats both uni-modal families: on VinTAGe-Bench, VinTAGe's FAD (3.05) and FID (16.43) are the best in the table, and its mean audio-text/audio-visual alignment (16.06) is the highest.","Modality bias is measurable and curable: joint baselines that assume aligned text and video (ReWaS, FoleyCrafter) generate on-screen sounds well but drop off-screen sounds, while teacher-guided training keeps VinTAGe's two accuracies close (57.7% versus 43.6%).","The two guidance scales make the text-video trade-off controllable at inference: raising $s_{\\text{vis}}$ shifts the output toward visual-only sounds, raising $s_{\\text{txt}}$ adds off-screen content, and balanced values produce both.","Because the same model is evaluated directly on VGGSound, the architecture doubles as a video-to-audio system, posting the best FID (6.65) and MKL (4.12) among the compared methods."],"supporting_citations":[{"why":"The flow-matching Scalable Interpolant Transformer backbone that the Joint VT-SiT model extends with dual cross-attention.","marker":"[31]"},{"why":"Supplies the velocity-field training objective and the reverse-time ODE integration used to sample audio latents.","marker":"[24]"},{"why":"Provides the pretrained audio autoencoder latent space and HiFi-GAN vocoder through which all audio is encoded and decoded.","marker":"[25]"},{"why":"Source of the instruction-tuned FLAN-T5 text encoder and of the pressure-level audio mixing used in training augmentation and in building the benchmark.","marker":"[11]"},{"why":"Visual encoder for frame embeddings and the shared space where audio-visual and audio-text alignment scores are computed.","marker":"[38]"},{"why":"Training corpus for the model and source of the videos and off-screen mixes that make up VinTAGe-Bench.","marker":"[4]"},{"why":"Main video-to-audio baseline; its need for large-scale pretraining for temporal alignment motivates the simpler optical-flow energy guide.","marker":"[30]"},{"why":"Strongest previous joint text+video baseline compared in tables and qualitative studies, showing the modality-bias gap VinTAGe closes.","marker":"[54]"},{"why":"Joint text+video baseline whose strong visual bias motivates the teacher-guided alignment losses.","marker":"[18]"},{"why":"Template for multi-condition classifier-free guidance with two separate guidance scales at inference.","marker":"[2]"}],"fun_headline_variants":["VinTAGe: text + video = full audio scene","Holistic audio: onscreen foley + offscreen ambience","Joint conditioning beats single-modality audio","Flow transformer fuses language and pixels for sound","New SOTA on VGGSound with joint audio gen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that VinTAGe-Bench's recipes — one on-screen sound mixed with one randomly chosen off-screen sound, with hard cases filtered out — stand in for real holistic audio, so the paper's conclusion that joint conditioning is necessary inherits everything from that curated two-source mix; if genuine off-screen soundscapes are messier than the benchmark, the necessity claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["VinTAGe: text + video = full audio scene","Holistic audio: onscreen foley + offscreen ambience","Joint conditioning beats single-modality audio","Flow transformer fuses language and pixels for sound","New SOTA on VGGSound with joint audio gen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2741,"prompt_tokens":1184,"completion_tokens":1557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":800,"completion_tokens_details":{"reasoning_tokens":1478}},"tokens_in":800,"tokens_out":1557,"duration_ms":10736,"temperature":1.0,"reasoning_tokens":1478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:14.338377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take uncurated real-world clips with natural multi-source soundscapes — several off-screen sounds at once, overlapping frequency bands, and multiple moving on-screen objects, the regime the paper's own limitations section flags — and compare VinTAGe against a strong text-to-audio model fed a detailed caption of the video. If the text-only model matches the joint model on semantic completeness and the temporal-synchronization advantage disappears on these clips, the claim that joint video-text conditioning is necessary for holistic audio would be falsified; running the same test on the benchmark's excluded hard cases (similar-frequency or busy mixtures) would show whether the filtering drove the result.","supporting_citations":[{"cited_title":"Albergo, Nicholas M","cited_arxiv_id":null,"evidence_quote":"The flow-matching Scalable Interpolant Transformer backbone that the Joint VT-SiT model extends with dual cross-attention."},{"cited_title":"Audi- oLDM: Text-to-audio generation with latent diffusion mod- els","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained audio autoencoder latent space and HiFi-GAN vocoder through which all audio is encoded and decoded."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Visual encoder for frame embeddings and the shared space where audio-visual and audio-text alignment scores are computed."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"Training corpus for the model and source of the videos and off-screen mixes that make up VinTAGe-Bench."},{"cited_title":"Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models, 2023","cited_arxiv_id":null,"evidence_quote":"Main video-to-audio baseline; its need for large-scale pretraining for temporal alignment motivates the simpler optical-flow energy guide."},{"cited_title":"Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds, 2024","cited_arxiv_id":null,"evidence_quote":"Strongest previous joint text+video baseline compared in tables and qualitative studies, showing the modality-bias gap VinTAGe closes."},{"cited_title":"Read, watch and scream! sound generation from text and video","cited_arxiv_id":null,"evidence_quote":"Joint text+video baseline whose strong visual bias motivates the teacher-guided alignment losses."},{"cited_title":"In- structpix2pix: Learning to follow image editing instructions","cited_arxiv_id":null,"evidence_quote":"Template for multi-condition classifier-free guidance with two separate guidance scales at inference."}],"review_version":1}