{"id":"47476fc4-5260-4d29-8e05-6a0a4bfaafc0","arxiv_id":"2506.05414","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.","lead":"The paper introduces SAVVY-Bench, a benchmark for 3D spatial reasoning in dynamic scenes with spatial audio, and SAVVY, a training-free pipeline that improves audio-visual LLMs on this benchmark. The work targets a capability humans use constantly, locating and tracking things you hear, that current AI models largely lack.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark answer-key validity is the load-bearing assumption: sound-source 3D positions and event times are derived from EFM3D plus human annotation with no reported inter-annotator agreement or independent validation, so the +7.1% could partly reflect label noise.","rationale":"The paper does something real: it constructs a benchmark with synchronized spatial audio and reports a modular, training-free pipeline with consistent gains across tasks, and the ablations show that each component contributes. The reader's CONDITIONAL verdict is appropriate. My stress-test converges on the same weakest link: the answer key. The benchmark's quantitative claims rest entirely on the accuracy of object centers, sound-source positions, and event intervals. The construction pipeline (Figure 5, Section C.2) is plausible but does not report inter-annotator agreement, correction statistics, or independent sensor-based validation of speaker positions. EFM3D is a foundation-model benchmark rather than a certified measurer, and manual calibration in a point-cloud viewer is not quantified. A systematic error of roughly 1 m in source position would directly change distance accuracies at the 0.1-1.0 m thresholds and could flip direction labels near boundaries. Since the benchmark is not released, the community cannot yet probe this. I do not think this warrants rejection: the concern is about missing evidence, not demonstrated fraud or an internal contradiction. The one concrete check that would settle it is a blinded re-annotation study on a sample. I also note a smaller internal issue in the CDR distance formula (Section 4.2), where the printed objective sum over D_t^2 * (CDR_t - K)^2 does not correspond to the stated calibration K approximately equal to D_t^2 * CDR_t; this is likely a typo but should be corrected in revision. The benchmark ground-truth validity concern is primary.","tokens_in":28808,"tokens_out":6960,"duration_ms":69580,"concrete_test":"Release a random subset of about 100 QA pairs with the raw AEA sensor streams; have two independent annotators re-label the sound-source 3D position at the event time and the event interval using an independent tool without seeing existing labels; compute per-question agreement and re-score Gemini-2.5-pro and SAVVY on the subset where both annotators agree. If label disagreement exceeds about 5% or the SAVVY gain disappears on the agreed subset, the benchmark answer key is not stable enough to support the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SAVVY-Bench's ground truth is assembled from EFM3D object detections manually corrected in a point-cloud viewer, speech topics generated by Gemini from transcripts, and human-labeled sound events (Section 3.2, Section C.2). The paper reports no inter-annotator agreement, no correction statistics, and no independent validation of the 3D positions used as answers. For direction questions, a label flips if the source azimuth crosses a quadrant or 120-degree boundary; for distance questions, the metric rewards estimates within 1 m, so a systematic source-position error of about 1 m materially changes scores. Since every model in Table 2, including SAVVY, is scored against this key, a systematic bias in event time, sound-source identity, or object center would change all comparisons and could absorb or inflate the reported +7.1% gain. The benchmark artifacts are not released, so third parties cannot currently check this risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAVVY-Bench, a new benchmark for 3D spatial question answering in dynamic, audio-visual scenes, built from the Aria Everyday Activities dataset and comprising egocentric and allocentric direction and distance questions. It also proposes SAVVY, a training-free pipeline that combines an AV-LLM 'Snapshot Descriptor', CLIPSeg/SAM2 segmentation with metric depth, SRP-PHAT direction-of-arrival estimation, CDR-based distance estimation, and SLAM-derived global mapping. The main empirical claim is that SAVVY raises overall QA accuracy from 50.9% (Gemini-2.5-pro) to 58.0% on this benchmark, with ablations attributing gains to the three egocentric track components.","tokens_in":29039,"tokens_out":7263,"duration_ms":65588,"significance":"The benchmark is the first of its kind for dynamic 3D audio-visual spatial reasoning and could become a useful community resource if the ground-truth pipeline is validated and the data are released. The paper is strong in its detailed ablations, its use of real egocentric data with 7-channel spatial audio, the inclusion of a human baseline, and the qualitative error analysis. The central claims, however, rest on two points that need work: the CDR distance calibration is partly circular, and the benchmark answer-key validation is not reported. Both issues are fixable within the scope of a revision, but they affect the credibility of the headline +7.1% improvement and the claimed complementary role of spatial audio for distance estimation.","major_comments":[{"comment":"The CDR-based distance estimate is calibrated using the same visual distance estimates D_t that the pipeline is later said to complement. The text defines d_hat_t = sqrt(K/CDR_t) with K = argmin_K sum_t (D_t^2 · [CDR_t - K])^2. As written the optimization is dimensionally inconsistent: the product D_t^2·CDR_t does not appear in the objective. If the intended expression is sum_t (D_t^2·CDR_t - K)^2, then K is fitted to the visual D_t on the same video, and the audio distance estimate reduces to a rescaled copy of the visual estimate. Consequently, the 'Audio' track in Table 6 is not an audio-only distance estimator, and the claim that spatial audio improves distance accuracy is overstated. Please fix the equation, calibrate K on held-out scenes or with a fixed per-environment constant that does not use the test video's visual D_t, and re-evaluate the audio-only ablation.","section":"Section 4.2"},{"comment":"The benchmark answer key is the load-bearing element for every comparison in Table 2, but its reliability is not quantified. Object positions are EFM3D detections manually corrected in a point-cloud viewer, speech topics are generated by prompting Gemini on word-level transcripts, and sound-event times and source identities are labeled by human annotators; no inter-annotator agreement, correction statistics, or independent 3D validation is reported. Because direction labels flip on quadrant boundaries and distance accuracy is judged at 1 m tolerance, systematic errors in source positions or event times would shift all model scores and could absorb or inflate part of the reported +7.1% gain. Please report agreement statistics and validate a sample of 3D positions against an independent reconstruction or multiple annotators, and release the benchmark artifacts so that the community can check this risk.","section":"Section 3.2 / C.2"},{"comment":"The claimed 'Audio-only' performance is not achieved with audio alone for distance. Since K is fitted from the SD/Seg visual distance estimates on the same video, the audio distance component inherits the visual estimates, so the 48.1% egocentric distance entry in Table 6 does not measure audio-only distance estimation. An analysis that separates the SRP-PHAT direction (which is vision-free) from the CDR distance (which currently is not) is needed to support the paper's conclusion that 'spatial audio cues complement visual input for robust tracking'.","section":"Section 5.3 / Table 6"},{"comment":"The paper states that SAVVY-Bench comprises 'thousands of QA pairs' but does not report the total number of QA pairs, videos, or sound events, and the project website does not currently provide the benchmark artifacts or code. For a benchmark paper, this is a load-bearing reproducibility issue: without a dataset release or an explicit release plan, the community cannot verify the ground-truth quality or use the benchmark. Please include a data card with exact counts, a release link, and a commitment to make the evaluation pipeline available.","section":"Section 3.1 / release"}],"minor_comments":[{"comment":"The figure shows d_t = K * CDR, which conflicts with the text's d_hat_t = sqrt(K/CDR_t); please make the notation consistent.","section":"Figure 3(c)"},{"comment":"In the K optimization expression, the placement of brackets makes the intended objective ambiguous; write the objective explicitly as sum_t (D_t^2 * CDR_t - K)^2 or state the intended expression in words.","section":"Section 4.2"},{"comment":"Please specify how the 'overall' accuracy is computed, in particular whether it is a weighted average over the four task types or an unweighted average.","section":"Table 2"},{"comment":"Please add the exact counts or percentages on the pie slices and label the axes of the density plots; the current figure does not convey the number of questions behind each distribution.","section":"Figure 2"},{"comment":"The distance metric called 'average relative accuracy' is defined as the average across absolute error thresholds; this name is easy to confuse with relative error, so please clarify that it is a thresholded absolute-error accuracy.","section":"Section 5.1"},{"comment":"The main text says a proprietary AV-LLM is used to extract word-level transcriptions, while Section C.2 says the AEA raw annotations already contain word-level transcriptions and the annotators cluster them; please clarify which source is used to avoid an apparent inconsistency.","section":"Section 3.2 vs C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be a useful contribution if the calibration and validation issues are fixed. I would encourage the editor to make data and code release a condition of acceptance, since the benchmark's value depends on access. The CDR circularity is a correctness issue that the authors can address with a held-out calibration protocol; the ground-truth validation can be addressed with agreement statistics and an independent 3D check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that SAVVY-Bench is the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio, and the SAVVY pipeline is a sensible, training-free way to improve AV-LLM spatial QA. The main weak spot is that the audio distance estimate is calibrated against the visual distance estimates, so the claim of an independent hearing contribution is overstated. And the benchmark's answer key—the 3D object centers, event times, and sound-source identities—rests on human-corrected EFM3D detections and Gemini-generated speech topics, with no inter-annotator agreement or release of the benchmark artifacts. That makes the +7.1% gain harder to verify.\n\nWhat is genuinely new: the task design, covering egocentric and allocentric direction and distance over dynamic audio-visual scenes. The pipeline components are established, but the integration into a global map via SLAM is sensible, and the ablations (Tables 5, 6, 9) are informative. The human evaluation is a good sanity check. The paper is also honest in its limitations section, noting the reliance on a strong backbone and the difficulty of distance estimation.\n\nThe circular calibration (Section 4.2) is real but not fatal—audio still contributes egocentric direction via SRP-PHAT and helps when the target is out of view. But the 'Audio' track's distance numbers should be read with caution. The bigger issue is benchmark validity: no inter-annotator agreement, no correction statistics, no independent validation of the 3D positions. Since every model is scored against this key, a systematic ~1m bias in source positions or a boundary flip in quadrants would change all comparisons. The artifacts are not released, so no one can check. Family bias from using Gemini in both benchmark construction and as the SAVVY backbone is a minor concern; the QA templates and human correction limit it, but it should be disclosed more explicitly.\n\nWho should read this: anyone working on embodied QA, AV-LLMs, or spatial reasoning benchmarks. It deserves a serious referee; with release of the benchmark and an independent validation of the answer key, or at least inter-annotator statistics, the concerns would be largely resolved. I'd recommend sending it to peer review with a request for those additions.","headline":"A genuinely new dynamic audio-visual 3D spatial reasoning benchmark with a clever training-free pipeline, but the audio distance contribution is partly circular and the benchmark's answer key needs independent validation before the gains can be fully trusted.","tokens_in":29587,"tokens_out":3588,"would_cite":true,"duration_ms":40506,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces SAVVY-Bench and shows that a training-free pipeline lifts dynamic audio-visual 3D QA from 50.9% to 58.0%.","keywords":["audio-visual LLMs","3D spatial reasoning","egocentric reference frame","allocentric reference frame","spatial audio","dynamic scenes","benchmark","global map"],"falsifier":"Re-annotate a random subset of SAVVY-Bench with independent ground truth, for example lidar-measured object centers and manually time-stamped sound events, and re-run the model comparisons; if the 58.0% versus 50.9% gap narrows or flips, the original answer key, not spatial reasoning, was driving the results.","tokens_in":28641,"feed_emoji":"🧭","tokens_out":11735,"duration_ms":95387,"temperature":0.7,"pith_summary":"SAVVY-Bench is a new benchmark that asks audio-visual language models to answer 3D spatial questions about dynamic scenes with synchronized spatial audio, covering egocentric and allocentric direction and distance across thousands of QA pairs. The paper's main claim is that existing AV-LLMs, even the strongest proprietary ones, perform poorly on these questions because they ignore spatial audio and cannot track a sounding object through time in a consistent world frame. To fix this, the authors propose SAVVY, a training-free pipeline that estimates per-frame egocentric tracks for the objects in a question, merges them into a global map via SLAM camera trajectories, and then reads off the answer from that map. On the benchmark, SAVVY raises overall QA accuracy from 50.9% (Gemini-2.5-pro) to 58.0%, a +7.1% gain, with especially large gains on allocentric direction questions. If the results hold, the practical point is that strong spatial reasoning can be bolted onto existing AV-LLMs without retraining, and that spatial audio plus explicit geometry is the missing ingredient.","feed_headline":"SAVVY lifts audio-visual 3D QA from 50.9% to 58.0%","feed_subtitle":"A training-free pipeline maps sound sources into a global frame on a new dynamic-scene benchmark.","key_machinery":"The load-bearing object is the egocentric spatial track, a per-frame tuple $(t, \\theta, r)$ giving the timestamp, azimuth relative to the camera ($0^\\circ$ forward, negative left, positive right), and distance in meters of an object. The second stage folds these tracks into a dynamic global map $M_q = \\{p_{\\mathrm{sound}}(t) \\mid t \\in T_q\\} \\cup \\{p_{\\mathrm{ref}}, p_{\\mathrm{face}}\\}$ by adding the SLAM camera trajectory $L(t)$ to each track: $p(t) = L(t) + (r\\cos\\theta,\\, r\\sin\\theta)$. The map is what converts egocentric observations into allocentric answers: reference and facing objects become static landmarks defining a rotated coordinate frame, while the sounding object remains a time-varying trajectory. The identity doing the work is the coordinate transformation—once everything lives in one global frame, direction and distance questions reduce to reading off the map, which is also why missing tracks can be filled in by whichever modality (vision, audio, or language) observes the object at that moment.","core_discovery":"The central discovery is that dynamic 3D spatial reasoning does not require a new trained model: it can be decomposed into egocentric track estimation and global map construction. Given a question and a video with multi-channel audio, SAVVY first uses an audio-visual LLM to write a sparse snapshot description of the query-referenced event, identifying the target, reference, and facing objects with time spans and rough positions. It then fills in trajectory gaps with text-guided segmentation (CLIPSeg/SAM2 plus metric depth) and with spatial audio cues (SRP-PHAT for direction, coherent-to-diffuse ratio for distance). All per-frame estimates are lifted into a single global 2D map using the SLAM camera trajectory, static objects are clustered, and a Kalman filter smooths the dynamic target's path. The answer is obtained by a coordinate transformation aligned to the queried viewpoint, which is why the same pipeline handles both egocentric and allocentric questions. This construction yields the reported 58.0% overall accuracy, up from 50.9% for Gemini-2.5-pro, and 78.6% sounding-object localization accuracy when all three track types are combined.","pith_inferences":["If the +7.1 point gain is driven by the global-map stage rather than any single sensor module, then the same two-stage decomposition could be compressed into a training objective for AV-LLMs; the paper names this as future work, but the ablation tables make it a testable prediction.","The paper's constant-$K$ approximation $D_t^2 \\cdot \\mathrm{CDR}_t \\approx K$ predicts that audio-based distance estimates degrade in rooms with changing reverberation; measuring CDR stability across the recorded environments would reveal whether audio distance or visual depth is the weaker link.","Because the speech topics in the benchmark are produced by a prompted language model from transcripts, part of the measured difficulty may be topic paraphrase rather than spatial reasoning; re-running the QA templates with independently human-generated speech topics would isolate that factor."],"forward_implications":["Augmenting an AV-LLM with a training-free spatial pipeline raises overall benchmark accuracy from 50.9% to 58.0%, with the largest relative gain on allocentric direction questions (+12.3%).","Spatial audio is the decisive cue when the sounding object leaves the camera view: audio-only tracking reaches egocentric direction accuracy comparable to Gemini-2.5-pro, and the full pipeline corrects errors that arise from extrapolating the last visible location.","Combining all three track types (snapshot descriptor, segmentation, audio) gives the best sounding-object localization (78.6% loc_acc) and the best egocentric QA, while the snapshot descriptor is what makes allocentric reference objects reliable.","Humans score 78.7% against the best model's 58.0%, and 7B-scale AV-LLMs are near chance on allocentric direction and below 5% temporal-mIoU, indicating that event-time grounding and object referral, not spatial math alone, are the current bottleneck."],"supporting_citations":[{"why":"Supplies the egocentric videos, 7-microphone spatial audio, and camera trajectories that SAVVY-Bench and SAVVY are built on.","marker":"[60]"},{"why":"The proprietary AV-LLM used as the Snapshot Descriptor's temporal grounding and object referral engine, as the strongest baseline, and to generate speech topics.","marker":"[41]"},{"why":"Provides the initial 3D object detections that are manually corrected into ground-truth object positions.","marker":"[61]"},{"why":"Closest prior static-scene spatial reasoning benchmark, used for task framing and the fuzzy-matching direction metric.","marker":"[3]"},{"why":"SRP-PHAT supplies the direction-of-arrival estimates used in the spatial audio cues.","marker":"[66]"},{"why":"The coherent-to-diffuse ratio method supplies the audio-based distance estimates.","marker":"[67]"},{"why":"CLIPSeg supplies text-guided segmentation masks for object trajectory estimation.","marker":"[63]"},{"why":"ZoeDepth supplies per-frame metric depth, converting segmentation masks into distance estimates.","marker":"[65]"},{"why":"The SLAM camera trajectory is the bridge that converts egocentric tracks into the global map.","marker":"[62]"}],"fun_headline_variants":["SAVVY: training-free 3D spatial reasoning for audio-visual LLMs","SAVVY lifts audio-visual 3D QA to 58% without retraining","Hear where it is: SAVVY adds 3D audio spatial maps","Audio-visual LLMs see sound in 3D with SAVVY"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground-truth positions, event times, and source labels must be accurate—object centers are automated detections that annotators corrected in a point-cloud viewer, speech topics come from a prompted model, and sound events are human-labeled—because if the answer key is systematically wrong, every model score, including the reported +7.1 gain, measures errors in the key rather than spatial reasoning.","fun_headline_variants_meta":{"raw":{"variants":["SAVVY: training-free 3D spatial reasoning for audio-visual LLMs","SAVVY lifts audio-visual 3D QA to 58% without retraining","Hear where it is: SAVVY adds 3D audio spatial maps","Audio-visual LLMs see sound in 3D with SAVVY"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3107,"prompt_tokens":1041,"completion_tokens":2066,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":1975}},"tokens_in":657,"tokens_out":2066,"duration_ms":13415,"temperature":1.0,"reasoning_tokens":1975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:43:24.135204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of SAVVY-Bench with independent ground truth, for example lidar-measured object centers and manually time-stamped sound events, and re-run the model comparisons; if the 58.0% versus 50.9% gap narrows or flips, the original answer key, not spatial reasoning, was driving the results.","supporting_citations":[{"cited_title":"Brown University, 2000","cited_arxiv_id":null,"evidence_quote":"SRP-PHAT supplies the direction-of-arrival estimates used in the spatial audio cues."},{"cited_title":"Coherent-to-diffuse power ratio estimation for dere- verberation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(6):1006– 1018, 2015","cited_arxiv_id":null,"evidence_quote":"The coherent-to-diffuse ratio method supplies the audio-based distance estimates."},{"cited_title":"Image segmentation using text and image prompts","cited_arxiv_id":null,"evidence_quote":"CLIPSeg supplies text-guided segmentation masks for object trajectory estimation."},{"cited_title":"Photo-slam: Real-time simul- taneous localization and photorealistic mapping for monocular stereo and rgb-d cameras","cited_arxiv_id":null,"evidence_quote":"The SLAM camera trajectory is the bridge that converts egocentric tracks into the global map."}],"review_version":1}