{"id":"8f81a9a2-90c3-4d5c-b078-323011897feb","arxiv_id":"2508.12918","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"FoleySpace generates binaural audio from silent video by estimating a 3D sound-source trajectory from object detection and depth and conditioning a diffusion model on that trajectory plus monaural audio.","lead":"This paper presents FoleySpace, a system that turns silent video into stereo binaural audio whose perceived direction follows the visible sound source. It matters because most video-to-audio models output mono sound, while directional, immersive audio is needed for extended reality, gaming, and media production.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation of the trajectory condition: the central claim that visual trajectories drive spatial audio is unsupported.","rationale":"The reader's weakest assumption identifies exactly the missing ablation of the trajectory condition, and I agree that this is the most load-bearing gap. The paper's claimed contribution is that the estimated 3D trajectory, derived from YOLO-World and DepthMaster and mapped by Eqs. (3)-(5), guides the diffusion model to produce binaural audio whose perceived source position matches the video. However, the only evidence is a full-system user study and a few waveform examples; no experiment removes, perturbs, or swaps the trajectory condition while holding monaural audio fixed. This matters especially because training uses random ground-truth trajectories while inference uses estimated trajectories with a reported 39.62 degree azimuth MAE, so a system that simply ignores the trajectory and produces generic binaural audio could plausibly reproduce the observed results, since the baselines also lack dynamic visual alignment. The proposed concrete test would settle whether the trajectory condition has causal influence, and would directly support or undermine the central claim. I do not see an internal inconsistency in the method itself; the concern is that the central mechanism is unverified, which warrants the same conditional verdict the reader gave. Hence I recommend no change to the verdict, while emphasizing that the missing ablation should be a condition for full acceptance.","tokens_in":13737,"tokens_out":3629,"duration_ms":42040,"concrete_test":"Run inference with the fine-grained FoleySpace model on a fixed set of VGGSound-Solo clips (or the 32 user-study videos) with identical monaural inputs but with \\tilde{T} replaced by (i) a constant center trajectory and (ii) a trajectory taken from a different video. Have the same 24 raters score Spatial Alignment and Perceived Spatiality, and also compute the time-varying left/right channel energy difference. If SA/PSS for the manipulated conditions are statistically indistinguishable from the full model and the energy traces do not track the intended trajectory, the trajectory condition is not causally used; if SA/PSS degrade significantly, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the trajectory condition \\tilde{T} in Section III-D causally shaping the diffusion output. No experiment in Section V isolates this. The user study compares full pipelines, Table III reports MAE_alpha=39.62 degrees for the estimator, and Section IV-B trains on perfect randomly generated ground-truth trajectories, so the model has never been shown to respond correctly to the estimated trajectories used at inference. If the binaural output is unchanged when \\tilde{T} is replaced by a constant or by a trajectory from a different video, then the visual trajectory is not causally responsible for the reported spatial consistency, and the claim of 'spatially consistent stereo sound guided by visual information' fails. This is a missing causal test, not just a missing metric; it is the load-bearing link between the proposed visual mapping and the perceived spatial alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FoleySpace, a framework for generating binaural spatial audio from silent 2D video. It first estimates the sound source's 2D position via YOLO-World object detection and depth via DepthMaster monocular depth estimation. A mapping mechanism (Eqs. (3)-(5)) converts these per-frame pixel/depth coordinates into a 3D trajectory T in a listener-centered sound field, with hand-set parameters S_y=1.47m and gamma=W/2, followed by outlier-based smoothing. A diffusion model (DiffWave backbone) is conditioned jointly on monaural audio generated by a pre-trained MMAudio model and on the trajectory T~; the model is trained on binaural audio synthesized by convolving VGGSound mono clips with HUTUBS HRIRs, with distance-varying resampling and moving-source simulation via segmentation and smoothing. Two variants are implemented: fine-grained (per-frame) and coarse-grained (5x3 grid, 1 Hz). Experiments include a 24-participant user study across 32 videos (PSS, SA, TA, SC, AQ), objective metrics (FD, KL, IS, IB-score, DeSync) on a newly constructed VGGSound-Solo evaluation set, and source-estimation MAE. The authors report better spatial perception consistency than See2Sound, AudioX, and ThinkSound, with preserved audio quality.","tokens_in":13912,"tokens_out":4118,"duration_ms":42299,"significance":"If the trajectory condition is truly causally effective, the framework provides a practical modular solution: off-the-shelf detection and depth models plus a synthetic HRIR-based training set, and a publicly released evaluation dataset. The user study is a genuine external validation, and the comparison to monaural MMAudio provides a useful sanity check. However, the central mechanism is not isolated, and the quantitative source-localization metric is partially self-referential. The work is therefore valuable but requires stronger causal evidence before the core claim, that visual trajectories drive spatial output, can be accepted.","major_comments":[{"comment":"No experiment in Section V isolates the trajectory condition T~ (Section III-D): the diffusion model is always trained and evaluated with the full condition (monaural audio plus estimated trajectory), so the paper does not demonstrate that the trajectory, rather than the monaural audio or the conditioning architecture, is causally responsible for the reported spatial improvements in Table I and Table II. A direct test would replace T~ with a constant or zero trajectory, or with a trajectory from a different video, and measure PSS/SA or a spatial objective; without such a test, the claim that the visual trajectory guides spatial consistency is unsupported.","section":"§III-D, §V"},{"comment":"The diffusion model is trained on simulated perfect trajectories (Section IV-B) but at inference uses estimated trajectories whose azimuth MAE is 39.62 degrees (Table III); no experiment shows how the model's spatial output degrades with estimation error, and the reported end-to-end results do not reveal whether the system is robust to this error or relies on the monaural audio alone. The authors should either train or fine-tune with noisy/estimated trajectories, or evaluate the generated audio's spatial accuracy as a function of trajectory estimation error.","section":"§IV-B, §V-D"},{"comment":"The MAE metric in Section V-C2 is defined by mapping both estimated and manually annotated pixel positions through the same hand-set linear camera model (Eqs. (3)-(5)) with S_y=1.47m and gamma=W/2; thus it measures consistency with the authors' own mapping rather than physical source direction, and the paper does not establish that 39.62 degrees of azimuth error is perceptually negligible. Because the evaluation set VGGSound-Solo is constructed and annotated by the authors, an independent or at least a more detailed validation of the mapping and annotation procedure (e.g., inter-annotator agreement, comparison against a geometric calibration) is needed to support the quantitative claims.","section":"§V-C2, Eqs. (3)-(5)"}],"minor_comments":[{"comment":"The summation in Eq. (8) uses index i in the loop but the formula uses subscript n; the notation is inconsistent and should be unified.","section":"§IV-B, Eq. (8)"},{"comment":"The word 'construted' appears in the sentence beginning 'The training dataset for the diffusion models construted in Section IV'; it should be 'constructed'.","section":"§V-A"},{"comment":"The word 'segement' appears twice near Eq. (8); it should be 'segment'.","section":"§IV-B"},{"comment":"The caption begins 'Ilustration of binaural audio waveforms'; 'Ilustration' should be 'Illustration'.","section":"Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a modular pipeline and a synthetic training set; the user study is a positive. The missing trajectory-condition ablation is the key blocker; the MAE circularity and the training/inference mismatch are fixable with additional experiments. I would also encourage the authors to report statistical significance in the user study and to clarify the annotation protocol for VGGSound-Solo."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the bottom line: this is a genuine system contribution to video-to-binaural audio, but the paper's central causal claim isn't yet supported because the trajectory condition is never ablated. The novelty is real: it's the first (to my knowledge) work that estimates a 3D sound-source trajectory from a silent 2D video using open-vocabulary detection and monocular depth, maps it to a 3D sound field with a simple camera model, and feeds it as a condition to a diffusion model that already takes monaural audio from a pretrained V2A model. They also put together a large training set by convolving 10k VGGSound clips with HUTUBS HRIRs under randomly generated moving-source trajectories, and a new evaluation set, VGGSound-Solo, with manual location annotations.\n\nWhat's good: the user study is well-designed (24 participants, 32 videos, 5 dimensions) and FoleySpace beats all four baselines on perceived spatiality and spatial alignment. The HRIR-vs-RIR comparison in Table IV is a nice controlled experiment. The writing is clear and the system is easy to understand.\n\nThe soft spot is the one the stress-test note identifies: there is no ablation that removes or perturbs the trajectory condition. If you replace the estimated trajectory with a constant or with one from a different video and the output doesn't change, then the spatial consistency doesn't come from the visual trajectory. That's a missing causal test, and it's the pillar under the abstract's claim that the audio is 'guided by visual information.' In the same vein, the source estimator's 39.6-degree azimuth MAE is high, and the model is trained on perfect random trajectories but run on noisy estimated ones at inference; the mismatch is never discussed. The user study also lacks significance testing, and the benchmark is self-made, so some caution is warranted.\n\nNone of these are fatal. The baseline comparisons are in the paper's favor, and the design is sound enough that the missing experiments are probably the reason it's not a stronger claim, not evidence of a broken method.\n\nI'd send this to peer review. A good referee will ask for a trajectory ablation, significance testing, and ideally a release of the dataset. If those come back positive, this is a solid addition to the V2A and spatial-audio literature.","headline":"FoleySpace is a genuinely novel video-to-binaural system, but the missing trajectory ablation leaves its central spatial-guidance claim undemonstrated.","tokens_in":14433,"tokens_out":3229,"would_cite":false,"duration_ms":31874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A silent 2D video can drive binaural audio whose perceived source position moves with the visible object, via a trajectory-conditioned diffusion model.","keywords":["video-to-audio generation","binaural audio","spatial audio","diffusion model","sound source localization","head-related impulse response","audio-visual spatial consistency","open-vocabulary object detection"],"falsifier":"Remove the trajectory condition and keep only the monaural audio, then repeat the user study on the same videos; if spatial perception and spatial alignment scores do not drop, the visual trajectory is not carrying the claimed spatial consistency.","tokens_in":13546,"feed_emoji":"🎧","tokens_out":8288,"duration_ms":75507,"temperature":0.7,"pith_summary":"FoleySpace is a framework for turning silent, ordinary 2D video into binaural stereo audio in which the perceived direction of the sound follows the visible source. The paper argues that current video-to-audio methods mostly produce monaural audio, and that stereo methods either ignore visual spatial alignment or only handle static sources. To close that gap, FoleySpace estimates the sound source's position in each frame, converts those positions into a 3D trajectory, and feeds that trajectory together with monaural audio into a diffusion model that renders left- and right-ear signals. The paper reports that the generated audio carries interaural level and time cues matching the on-screen source, and that user studies rate its spatial perception and spatial alignment above existing spatial-audio baselines. If correct, the framework would let creators obtain spatially consistent sound for video without capture hardware or per-video engineering.","feed_headline":"Silent video becomes binaural audio that follows the on-screen source","feed_subtitle":"A video-derived 3D trajectory conditions a diffusion model so the sound lands where the eye is looking.","key_machinery":"The load-bearing mechanism is the mapping from the 2D visual plane to a listener-centered 3D sound field. With $\\delta = 2 S_y / W$, the source's 3D coordinate at frame $k$ is $x_k = \\delta \\tilde d_k$, $y_k = \\delta (w_k - W/2)$, $z_k = -\\delta (h_k - H/2)$, where $\\tilde d_k$ is the min-max normalized depth scaled by $\\gamma = W/2$ and $S_y = 1.47$ m is the hand-set maximum lateral distance. This mapping turns a bounding-box center $(w_k, h_k)$ and a depth value $d_k$ into a trajectory $T = \\{(x_k,y_k,z_k)\\}_{k=1}^K$ that aligns the image center with the listener and simulates a reasonable listening distance. The trajectory is smoothed by removing frames whose motion exceeds the 95% threshold and linearly interpolating the gaps, then it is replicated along time, concatenated with the monaural audio, and passed through a condition block into the diffusion backbone. This mechanism is what is supposed to make the left- and right-ear cues match the visual position at each moment.","core_discovery":"The paper's central claim is that binaural spatial audio for dynamic scenes can be synthesized from a silent video by conditioning a waveform diffusion model on a trajectory derived from the video itself. The trajectory is obtained by locating the sound source with an open-vocabulary object detector, reading its depth from a monocular depth estimator, and mapping the normalized pixel coordinates and depth into a listener-centered 3D sound field through a linear camera-like transformation. That 3D trajectory, together with monaural audio from a pretrained video-to-audio model, is the conditioning signal for the diffusion model, which outputs left- and right-ear signals whose interaural level and time differences encode direction. Training data are synthesized by convolving monaural audio clips with recorded head-related impulse responses along randomly generated moving-source trajectories, including distance-varying time delays. Experiments compare FoleySpace with monaural, stereo-VAE, and RIR-based baselines and report higher user-study scores for spatial perception and spatial alignment while keeping semantic, temporal, and audio-quality metrics close to monaural generation.","pith_inferences":["Editorial inference: the reported experiments do not include an ablation that removes the trajectory condition while keeping the monaural audio, so the causal role of the trajectory in producing spatial consistency is not directly demonstrated.","Editorial inference: with the estimator's azimuth error near 40 degrees on the evaluation set, the rendered interaural cues can point well away from the true source even when the model is faithful to its input, so applications requiring accurate localization would inherit this error.","Editorial inference: the geometric-center assumption treats the source as a compact point inside the detected box; large, occluded, or off-screen sound sources would produce a trajectory that misrepresents the audible source, so the method is best suited to single, visible, compact sources.","Editorial inference: the same trajectory-conditioned diffusion recipe could naturally extend to multiple simultaneous sources by conditioning on several trajectories, but the single-source dataset and 5×3 grid evaluation do not yet test that scenario."],"forward_implications":["If the claim holds, a video of an object moving across the screen yields binaural audio whose dominant ear channel shifts accordingly, because the trajectory condition enters the diffusion model at every frame.","The framework removes the need for external spatial metadata: any ordinary 2D video can be spatialized, provided the source can be detected and depth estimated.","Because training uses simulated moving sources with head-related impulse responses that include distance-induced time delays, the model can render dynamic sound fields rather than only fixed positions.","The coarse-grained variant, which estimates the source once per second on a 15-cell grid, scores nearly as well in user ratings, suggesting that low-rate trajectory cues are enough for perceived spatial consistency.","Objective audio-quality metrics place FoleySpace between monaural generation and the stereo-VAE baselines, so spatialization is achieved without a large sacrifice in audio fidelity."],"supporting_citations":[{"why":"supplies the monaural video-to-audio generator used as the semantic condition, and its objective metrics are adopted for evaluation","marker":"[11]"},{"why":"is the RIR-based spatial audio baseline that FoleySpace must beat and that motivates the HRIR-based data construction","marker":"[16]"},{"why":"provides the open-vocabulary object detector used to locate the sound source in each frame","marker":"[25]"},{"why":"provides the monocular depth estimator whose per-frame depth feeds the coordinate mapping","marker":"[26]"},{"why":"is the waveform diffusion backbone that is adapted to output binaural audio","marker":"[27]"},{"why":"supplies the monaural clips used to synthesize the binaural training dataset and the evaluation source","marker":"[28]"},{"why":"supplies the recorded head-related impulse responses from which the binaural training examples are rendered","marker":"[29]"},{"why":"is the annotated single-source evaluation dataset used for the objective and user-study comparisons","marker":"[32]"}],"fun_headline_variants":["Binaural audio tracks video's visible sound source","Video trajectory steers binaural audio in space","From silent video to spatially aligned binaural sound","Diffusion model converts video into moving binaural audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire spatial alignment depends on the estimated 3D trajectory—the bounding-box center plus min-max normalized depth through hand-set constants—being the true sound-source position, and on the diffusion model actually using that trajectory rather than ignoring it.","fun_headline_variants_meta":{"raw":{"variants":["Binaural audio tracks video's visible sound source","Video trajectory steers binaural audio in space","From silent video to spatially aligned binaural sound","Diffusion model converts video into moving binaural audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1895,"prompt_tokens":976,"completion_tokens":919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":855}},"tokens_in":592,"tokens_out":919,"duration_ms":9292,"temperature":1.0,"reasoning_tokens":855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:17:37.781571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove the trajectory condition and keep only the monaural audio, then repeat the user study on the same videos; if spatial perception and spatial alignment scores do not drop, the visual trajectory is not carrying the claimed spatial consistency.","supporting_citations":[{"cited_title":"Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis,","cited_arxiv_id":null,"evidence_quote":"supplies the monaural video-to-audio generator used as the semantic condition, and its objective metrics are adopted for evaluation"},{"cited_title":"Yolo-world: Real-time open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"provides the open-vocabulary object detector used to locate the sound source in each frame"},{"cited_title":"Vggsound: A large-scale audio-visual dataset,","cited_arxiv_id":null,"evidence_quote":"supplies the monaural clips used to synthesize the binaural training dataset and the evaluation source"},{"cited_title":"The hutubs hrtf database,","cited_arxiv_id":null,"evidence_quote":"supplies the recorded head-related impulse responses from which the binaural training examples are rendered"},{"cited_title":"VGGSound-Solo,","cited_arxiv_id":null,"evidence_quote":"is the annotated single-source evaluation dataset used for the objective and user-study comparisons"}],"review_version":2}