{"id":"bb484c66-e515-40bb-9c04-09d25f0a03ab","arxiv_id":"2412.15191","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AV-Link unifies video-to-audio and audio-to-video generation by aligning frozen diffusion-model activations with temporally matched rotary position embeddings in a shared Fusion Block.","lead":"AV-Link is a single framework that can both add sound to silent videos and generate video from audio, using the internal features of frozen text-to-audio and text-to-video diffusion models instead of separate pretrained encoders. The authors report large gains in audio-video synchronization, including a 76% relative improvement in onset accuracy over prior methods on VGGSounds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal alignment hinges on an unverified linear token mapping; the paper never states η_v/η_a or whether video tokens are indexed temporally or spatiotemporally, so the Onset ACC gain may not reflect a correct alignment mechanism.","rationale":"The paper's headline contribution is that frozen diffusion activations replace pretrained feature extractors while improving temporal alignment, with Onset ACC as the strongest evidence (0.531 vs 0.301). That metric is exactly what the temporal-alignment mechanism in Eq. 4 is designed to optimize; if the RoPE ratio is mis-specified, the entire advantage collapses. The reader's weakest assumption correctly identifies the fixed linear mapping. I agree this is central, but I would sharpen it into an implementation ambiguity: the paper does not specify the token rates η_v and η_a, and because the video model is a 2×2-patchified pixel DiT, the flattened token sequence contains spatial indices that would corrupt a naive 1D temporal RoPE. The §3.2 notation ('T_v indicates the total number of video pixels') and Appx. B do not resolve how video tokens are ordered in the Fusion Block. This is not merely a robustness limitation; it is an unverified detail at the core of the claim. The ablations show the RoPE-based Fusion Block outperforms alternatives, but they do not isolate whether the RoPE alignment is correct. Other concerns (conditioning timestep tuned on the evaluation benchmarks, single A2V baseline, no released code) also support CONDITIONAL, but they affect confidence in the numbers rather than the mechanism itself. I therefore recommend keeping the CONDITIONAL verdict while adding a specific verification test for the alignment ratio. If the test shows the mapping is correctly implemented and robust to small rate perturbations, the central claim is much stronger.","tokens_in":22832,"tokens_out":17376,"duration_ms":107968,"concrete_test":"Concrete check: (1) Inspect the released config/code to obtain η_v and η_a and confirm that Fusion-Block RoPE is applied over the temporal frame index rather than the flattened spatiotemporal token index; this settles whether Eq. 4 is implemented as claimed. (2) Resample a set of VGGSounds audio clips to a slightly different temporal token rate (e.g., ±10%) while preserving duration, generate A2V video from the resampled audio, and measure whether audio-video onset alignment degrades relative to the unmodified clips. If it degrades substantially, the fixed linear ratio is a genuine limitation of the claimed temporal alignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that diffusion activations provide temporally aligned conditioning depends entirely on Eq. 4's RoPE mapping τ(n)=n·η_v/η_a for audio tokens. Two load-bearing assumptions are unverified. First, the paper never gives numerical values for η_v and η_a, nor defines whether η_v counts temporal frames only or all spatiotemporal video tokens after 2×2 patchification. If video tokens are indexed in the flattened sequence as frame·S + spatial, then two video tokens at the same time but different spatial locations receive different rotations, and no single audio token can align to the correct video time; attention would be smeared by the spatial index. The §3.2 notation is ambiguous ('T_v indicates the total number of video pixels'), and Appx. B does not resolve the index ordering. Second, the fixed linear, zero-offset mapping is never stress-tested: all evaluations use synchronized clips of a fixed duration, so a constant offset or a variable token rate would break the claimed alignment. The Table 2 ablation shows this particular implementation beats tested alternatives on this benchmark, but it does not establish that the mapping is correctly implemented or that the alignment would survive perturbations to the assumed linear correspondence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes AV-Link, a unified framework for video-to-audio (V2A) and audio-to-video (A2V) generation built from two frozen flow-matching generators. The key components are Fusion Blocks that run self-attention over concatenated audio and video token sequences, a time-aligned 1D RoPE scheme intended to align tokens by seconds, and a symmetric feature reinjection mechanism that feeds cross-modal activations back into the conditioning backbone. The authors claim that diffusion activations contain rich temporal information and can replace specialized feature extractors, and report large gains in Onset ACC on VGGSounds (0.531 vs. 0.301 for the best baseline), competitive semantic metrics, and user-study preferences for temporal alignment over Movie Gen Audio. The paper also provides ablations of the conditioning timestep, fusion block placement, feature types, and feature injection methods.","tokens_in":23032,"tokens_out":6814,"duration_ms":55213,"significance":"If the central claim holds, AV-Link is a useful step toward unified audiovisual generation, since it avoids task-specific encoders (CLIP, CLAP, CAVP, BEATs) and lets two frozen unimodal generators exchange temporally aligned features. The paper's strengths include a broad ablation matrix, quantitative and user evaluations on two benchmarks, reproduction of several baselines from released code, and qualitative in-the-wild tests. The symmetric design and the idea of using diffusion activations as cross-modal conditioning are likely to be of interest to the community. However, the significance is contingent on resolving the test-set tuning of the conditioning timestep, on demonstrating that the temporal alignment mechanism is correctly specified and robust, and on strengthening the A2V comparison.","major_comments":[{"comment":"The temporal alignment mapping in Eq. (4) is underspecified. The paper never states the numerical values of η_a and η_v, and the definition of T_v in §3.2 as 'the total number of video pixels' is ambiguous about whether the video token sequence is ordered temporally or spatiotemporally after the 2×2 patchification described in Appx. B. If tokens are ordered as frame·S + spatial, then tokens from the same video time receive different rotations, so no single audio token can align to a given video time, and the self-attention in the Fusion Block would be smeared over spatial indices. If tokens are ordered purely temporally, the meaning of η_v is unclear because the model uses 3D-RoPE. The authors should specify the exact token layout, give concrete η_a and η_v for the 5.16s/6fps/36×64 setting, and report a diagnostic (e.g., attention localization or a synthetic offset test) showing that the RoPE-based correspondence is what drives the Onset ACC improvement.","section":"Sec. 3.3, Eq. (4); Sec. 3.2; Appx. B"},{"comment":"The conditioning flow timestep hyperparameters (0.96 for V2A, 0.8 for A2V) are selected by scanning on the VGGSounds test set and then used to train the final models whose results are reported on the same test set. Section 4.4 states that after identifying the optimal timestep, 'we train subsequent models using these fixed timesteps,' and the evaluation protocol in §4.2 uses 2048 videos from the VGGSounds test set. This is a form of test-set tuning that can inflate the reported gains in Tab. 2 and Tab. 1. Please select timesteps on a held-out validation split, or provide evidence (e.g., via the Movie Gen benchmark or a different dataset) that the optimal timesteps transfer across distributions.","section":"Sec. 4.4, Fig. 3; Sec. 4.2; Tab. 2"},{"comment":"Automatic metrics are reported without error bars or multiple seeds, and Section 4.4 explicitly states that 'we perform evaluation using a single fixed seed for all test videos.' Several ablation differences in Tab. 2 are small (e.g., Onset ACC 0.415 vs. 0.410 for symmetric cross attention vs. fusion blocks; IB-AV 0.155 vs. 0.150), so with one seed these may not be significant. The central claim that the Fusion Block design improves temporal alignment would be substantially strengthened by reporting means and standard deviations over at least three seeds, and by statistical comparisons for the key ablations.","section":"Sec. 4.4; Tabs. 1–3"},{"comment":"The A2V evaluation compares only against TempoTokens, and the comparison crops AV-Link's 5.16s outputs to 2s to match TempoTokens. A single baseline, especially one that generates only 2s clips, is a weak basis for the claim that AV-Link surpasses prior work in A2V; other A2V methods (AADiff, Seeing-and-Hearing A2V, AVSyncD) are discussed but not included in quantitative comparisons. The cropping protocol may also affect FVD and temporal metrics differently for the two methods. Please add at least one additional strong A2V baseline using published numbers where code is unavailable, and analyze the sensitivity of the metrics to the cropping protocol.","section":"Sec. 4.3, Tab. 3; Appx. D.2"}],"minor_comments":[{"comment":"The abstract writes 'Video-to-Audio (A2V) and Audio-to-Video (A2V)'; the first expansion should be V2A. The running header and Figure 1 caption also contain 'A V-Link' with an extra space.","section":"Abstract"},{"comment":"The phrase 'We show for the first time' is stronger than necessary and may invite unnecessary priority disputes; consider softening it, especially given prior work on diffusion hyperfeatures for spatial alignment [56].","section":"Sec. 1"},{"comment":"The sentence 'Given an RGB video V1, we flatten it to shape R^{T_v×3}' is confusing because T_v is later called 'the total number of video pixels'; please separate the sequence length (number of tokens) from the spatial dimensions.","section":"Sec. 3.2"},{"comment":"FID is an image metric; specify how it is applied to video (e.g., per-frame) and whether the same frames are used for all methods. In addition, the ✓ marks in Tab. 1 should be defined in the caption to clarify which results are text-conditioned.","section":"Sec. 4.2; Tab. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper reports strong results, but the test-set tuning of the conditioning timestep is a significant evaluation concern that should be resolved before publication. The A2V comparison is also thin. If the authors can address these without changing the architecture, the paper could be suitable for acceptance. There is no evidence of author misconduct, but the 'for the first time' phrasing should be toned down."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is one of the stronger cross-modal generation papers I've seen recently. The idea is genuinely new—instead of CLIP/CLAP/CAVP features, it uses activations from frozen flow-matching generators of the other modality and feeds them through trainable Fusion Blocks with temporally-aligned RoPE. It works: Onset ACC 0.531 vs 0.301 for the best baseline, and user studies prefer it over Movie Gen Audio for temporal alignment 63.6% of the time. The ablations are broad: feature extractor type, fusion block placement, injection method, timestep choice, shared vs separate parameters. That's real work, and the paper earns its central claim that diffusion activations can replace specialized feature extractors.\n\nSoft spots, in order of importance. First, the temporal alignment mechanism is underspecified. Eq. 4 defines tau(n)=n·eta_v/eta_a for audio tokens, but the paper never gives eta_v or eta_a, and the video model is pixel-based with 2x2 patchification, so \"T_v indicates the total number of video pixels\" is ambiguous about whether tokens are indexed temporally only or spatiotemporally. If video tokens carry spatial indices in the flattened sequence, a single audio token cannot align to the correct time—attention would be smeared by spatial position. The paper also assumes a fixed linear, zero-offset mapping and never stress-tests it against constant offsets or variable token rates. This may not be fatal—the empirical gains are consistent—but it needs to be spelled out before the mechanism is fully credible. Second, the headline numbers come from a single seed, with no error bars, and the conditioning timestep (0.96 V2A, 0.8 A2V) was tuned on the same benchmarks used for the main results. That's mild, but it means the margin over baselines is likely optimistic. Third, the A2V comparison has exactly one baseline (TempoTokens), and no code or checkpoints are released. Minor: the abstract mislabels V2A as A2V. On the credit side, the authors list limitations honestly (low-res video backbone, extra inference compute from reinjection).\n\nWho this is for: anyone working on audio-video generation, conditional diffusion, or temporal alignment. It deserves a full peer review. The referee should focus on the RoPE token-indexing question and ask the authors to state eta_v/eta_a, define the token ordering, and run a robustness check with offset perturbations. If they can answer that, this is a solid accept-level paper.\n\nSend it to review.","headline":"Genuinely new and empirically strong unified V2A/A2V framework using frozen diffusion activations; the reported gains are impressive, but the RoPE alignment mechanism is underspecified and the evaluation has some mild selection issues.","tokens_in":23674,"tokens_out":3204,"would_cite":true,"duration_ms":27691,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion activations are temporally precise enough to replace dedicated feature extractors like CLIP, CLAP, and ImageBind for cross-modal generation, and a symmetric framework built on that idea handles both video-to-audio and…","keywords":["video-to-audio generation","audio-to-video generation","diffusion activations","temporal alignment","flow matching","fusion blocks","frozen generators","rotary position embedding"],"falsifier":"Construct a test set in which the same visual event is paired with audio whose onset leads or lags the visual by controlled offsets (for example 50 to 300 milliseconds, varying per clip) and measure whether AV-Link's generated audio tracks the true onsets or locks onto the linear token-ratio grid: the claimed mechanism predicts Onset ACC should collapse as the offset leaves the fixed τ(n) = n·η_v/η_a mapping. A second check: encode the same video content at different frame rates or token densities and see whether alignment degrades, which would show how load-bearing the fixed ratio is.","tokens_in":22562,"feed_emoji":"🎬","tokens_out":9198,"duration_ms":73316,"temperature":0.7,"pith_summary":"AV-Link claims that the activations of frozen audio and video diffusion generators carry precise temporal information, enough to replace specialized pretrained feature extractors such as CLIP, CLAP, ImageBind, BEATs, and CAVP in cross-modal generation. It delivers this as a single symmetric framework that handles both video-to-audio and audio-to-video: trainable Fusion Blocks inserted between two frozen flow-matching generators let each modality condition the other through temporally-aligned self-attention. On the VGGSounds benchmark the method raises Onset ACC to 0.531 from the previous best 0.301, and in user studies its temporal alignment is preferred over Movie Gen Audio, a 13B-parameter model, 63.6% of the time. The paper's bet is that diffusion activations, because they are trained to produce temporally-structured output, are better raw material for audio-video synchronization than embeddings learned from contrastive tasks.","feed_headline":"Diffusion activations beat clip features at syncing sound and video","feed_subtitle":"One framework handles video-to-audio and audio-to-video; onset accuracy 0.531 vs 0.301 for the best prior baseline.","key_machinery":"The load-bearing object is the Fusion Block: a multi-head self-attention layer applied to the concatenation of projected audio and video activations, followed by an MLP, whose outputs are reinjected into subsequent DiT blocks of both frozen backbones (symmetric feature reinjection). Temporal correspondence inside the block is set by a time-aligned 1D Rotary Position Embedding, which rotates a token at temporal index n by an angle proportional to n·θ_base for video tokens and n·(η_v/η_a)·θ_base for audio tokens, so that tokens representing the same moment in time share the same rotation. A second design choice matters nearly as much: the conditioning modality is fed to the fusion blocks at a fixed flow timestep near full denoising (0.96 for video-to-audio, 0.8 for audio-to-video), where the high-frequency details that determine sound onsets survive.","core_discovery":"The paper's central claim, stated in the introduction, is that video and audio diffusion activations contain rich temporal information and can for the first time replace specialized pretrained feature extractors while improving temporal alignment. The evidence is an architecture that keeps two pretrained generators frozen and adds only 186M trainable parameters in eight Fusion Blocks, yet surpasses dedicated V2A and A2V systems on quality and alignment metrics while being preferred by human raters on temporal alignment. The conceptual discovery is that the conditioning signal does not have to be a static embedding from a task-specific encoder; it can be the evolving internal state of a generative model of the other modality, refined throughout sampling.","pith_inferences":["The same frozen-generator fusion recipe suggests itself for other timing-critical modality pairs, such as music-to-dance or lip-synchronized speech, wherever each side already has a generator whose activations encode temporal structure; the paper does not test these cases.","A stress test the paper does not run: offset sound events from their visuals by varying lead or lag times, or re-encode clips at different frame rates; the fixed token-ratio mapping τ(n) = n·η_v/η_a predicts alignment should break as offsets grow.","The finding that near-clean conditioning activations work best is a transferable design principle for conditional generation: for onset-sensitive outputs, destroying high-frequency detail in the conditioning stream is the wrong default, even though uniform noisy-condition sampling is a common training choice."],"forward_implications":["Video-to-audio and audio-to-video collapse into one self-contained system: 8 Fusion Blocks (186M parameters) on top of frozen 576M-parameter generators replace the separate feature extractors and task-specific models prior work required.","Temporal alignment improves even against far larger systems: human raters prefer AV-Link's timing over Movie Gen Audio (13B parameters) 63.6% of the time, and Onset ACC rises to 0.531 from the 0.301 best prior baseline.","A single shared set of Fusion Block parameters can serve both directions with only a marginal drop, so one checkpoint can cover text-to-audio, text-to-video, V2A, and A2V generation.","Because both backbones stay frozen, future improvements to single-modality generators should transfer directly to cross-modal quality without retraining the fusion machinery.","The conditioning stream should be injected at a high, fixed flow timestep rather than sampled uniformly, since high-frequency visual details drive onset-sensitive sounds."],"supporting_citations":[{"why":"Supplies the 13B-parameter video-to-audio generator and its benchmark that AV-Link is measured against in user studies.","marker":"[68]"},{"why":"Diff-Foley is the CAVP-conditioned V2A baseline the paper contrasts with its activation-based conditioning.","marker":"[57]"},{"why":"Frieren is the strongest prior V2A baseline (Onset ACC 0.301) against which AV-Link's 0.531 is compared.","marker":"[89]"},{"why":"FoleyCrafter is the CLIP-based V2A baseline with a temporal onset controller that AV-Link outperforms.","marker":"[104]"},{"why":"TempoTokens is the A2V baseline with public code, using BEATs features, that AV-Link surpasses on FVD and user preference.","marker":"[100]"},{"why":"Seeing and Hearing is the ImageBind-based aligner baseline for both V2A and A2V tasks.","marker":"[92]"},{"why":"Diffusion Hyperfeatures supplies the prior evidence that diffusion activations encode semantic and spatial structure, motivating their use as conditioning features.","marker":"[56]"},{"why":"RoFormer provides the Rotary Position Embedding mechanism that AV-Link's time-aligned RoPE is built on.","marker":"[77]"},{"why":"GenAU supplies the audio generation backbone, the 1D-VAE audio tokenizer, and the AutoCap captioning pipeline used by the audio generator.","marker":"[27]"},{"why":"Defines the Onset ACC metric used to measure temporal alignment on the VGGSounds benchmark.","marker":"[20]"}],"fun_headline_variants":["Diffusion activations, not clip features, sync AV generation","Frozen diffusion models exchange features for cross-modal sync","One framework for V2A and A2V with better sync than MovieGen","Audio-video sync improved by fusing diffusion activations","Bidirectional diffusion fusion beats dedicated V2A and A2V models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Alignment depends on a single fixed linear mapping between audio time and video time set by the ratio of tokens per second, so any sound that leads or lags its visual event, or any clip sampled at a different rate, falls outside what the mechanism can represent.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion activations, not clip features, sync AV generation","Frozen diffusion models exchange features for cross-modal sync","One framework for V2A and A2V with better sync than MovieGen","Audio-video sync improved by fusing diffusion activations","Bidirectional diffusion fusion beats dedicated V2A and A2V models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000927,"raw_usage":{"total_tokens":3920,"prompt_tokens":841,"completion_tokens":3079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2990}},"tokens_in":457,"tokens_out":3079,"duration_ms":15778,"temperature":1.0,"reasoning_tokens":2990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:32:58.850020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set in which the same visual event is paired with audio whose onset leads or lags the visual by controlled offsets (for example 50 to 300 milliseconds, varying per clip) and measure whether AV-Link's generated audio tracks the true onsets or locks onto the linear token-ratio grid: the claimed mechanism predicts Onset ACC should collapse as the offset leaves the fixed τ(n) = n·η_v/η_a mapping. A second check: encode the same video content at different frame rates or token densities and see whether alignment degrades, which would show how load-bearing the fixed ratio is.","supporting_citations":[{"cited_title":"Sampson, Shikai Li, Si- mone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petro- 11 vic, and Yuming Du","cited_arxiv_id":null,"evidence_quote":"Supplies the 13B-parameter video-to-audio generator and its benchmark that AV-Link is measured against in user studies."},{"cited_title":"Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models","cited_arxiv_id":null,"evidence_quote":"Diff-Foley is the CAVP-conditioned V2A baseline the paper contrasts with its activation-based conditioning."},{"cited_title":"Frieren: Efficient video-to-audio generation with rectified flow matching","cited_arxiv_id":null,"evidence_quote":"Frieren is the strongest prior V2A baseline (Onset ACC 0.301) against which AV-Link's 0.531 is compared."},{"cited_title":"Diverse and aligned audio-to- video generation via text-to-video model adaptation","cited_arxiv_id":null,"evidence_quote":"TempoTokens is the A2V baseline with public code, using BEATs features, that AV-Link surpasses on FVD and user preference."},{"cited_title":"Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners","cited_arxiv_id":null,"evidence_quote":"Seeing and Hearing is the ImageBind-based aligner baseline for both V2A and A2V tasks."},{"cited_title":"Diffusion hyperfeatures: Searching through time and space for semantic correspon- dence","cited_arxiv_id":null,"evidence_quote":"Diffusion Hyperfeatures supplies the prior evidence that diffusion activations encode semantic and spatial structure, motivating their use as conditioning features."},{"cited_title":"Roformer: Enhanced transformer with rotary position embedding","cited_arxiv_id":null,"evidence_quote":"RoFormer provides the Rotary Position Embedding mechanism that AV-Link's time-aligned RoPE is built on."}],"review_version":1}