{"id":"7f40a4e9-71c3-42d6-aa59-ece920d20c48","arxiv_id":"2505.12693","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TACOcc reaches 28.4% mIoU on nuScenes 3D occupancy prediction by learning adaptive fusion neighborhoods and adding 3DGS-based photometric supervision.","lead":"This paper introduces TACOcc, a multi-modal fusion method for 3D semantic occupancy prediction that learns a per-query fusion neighborhood and adds a 3D Gaussian splatting renderer as extra training supervision. On the nuScenes validation set it reports 28.4% mean IoU, 1.8 points above the previously published Co-Occ result.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing fixed-k=4 control confounds the adaptive-fusion claim: the dynamic candidate set includes k=4, so the gain over fixed k=3 may reflect neighborhood size, not target-adaptive selection.","rationale":"The reader's conditional verdict is appropriate: the paper presents a coherent dual-module framework and internally consistent improvements, but the specific mechanism claimed in the abstract is not yet pinned down. My concern is closely related but sharper than the reader's. The reader focuses on the absence of per-object measurements for k versus target scale. I add that the comparison set in Table 4 omits fixed k=4, which is the natural control for the dynamic range 1-4. Without that control, the improvement attributed to adaptive selection could be an artifact of the candidate set simply containing a larger k. This matters because the paper's first contribution and the abstract's core statement are specifically about expanding neighborhoods for large targets and shrinking them for small ones. If the gain is only from allowing k=4, the contribution is not target-adaptive fusion but a larger fixed neighborhood. The tables also lack error bars and code, so the numerical claims are not independently reproducible, but I do not treat that as a separate defect; it amplifies the need for the missing control. My proposed check is therefore a single fixed-k=4 run under the identical protocol, which would settle whether the dynamic range's headline improvement is really due to adaptivity. This does not change the reader's CONDITIONAL verdict; it reinforces the conditions under which the paper should be accepted.","tokens_in":15381,"tokens_out":13556,"duration_ms":147928,"concrete_test":"Run the full TACOcc pipeline (fusion plus Lrgb and Lpc) with a fixed k=4 on the nuScenes validation split, using the same 15-epoch, 8xA6000 protocol as Table 4. If the resulting IoU and mIoU are close to or above 41.8 and 28.4, respectively, then the dynamic range's advantage over fixed k is not established, and the target-adaptive claim is unsupported. If fixed k=4 is clearly worse, the adaptive mechanism may still be beneficial, but a separate per-query analysis of k against ground-truth object size, controlling for point density and distance, would then be needed to validate scale-awareness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is that per-query k values adapt to target scale. The evidence for this is Table 4, which compares fixed k=1,2,3 with dynamic ranges 1-2, 1-3, and 1-4. Fixed k=4 is never evaluated. Because the dynamic model may select k=4 for many queries, the reported improvement of dynamic 1-4 over fixed k=3 (mIoU 28.4 vs 26.7, IoU 41.8 vs 38.2) could be due simply to allowing a larger maximum neighborhood, not to scale-adaptive selection. Table 3's fixed-k baseline is also not specified, so the dynamic-over-fixed gain of 1.6 mIoU in that ablation cannot be audited. Additionally, Tables 5-6 report k distributions aggregated over scenes grouped by dominant target size, which confounds target scale with scene density, object count, and average distance. The load-bearing issue is that the adaptive fusion contribution, and its claimed role in the SOTA result, is not isolated from the choice of maximum k.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TACOcc, a multi-modal 3D semantic occupancy prediction framework combining an adaptive cross-modal fusion module with a 3D Gaussian Splatting volume-rendering supervision module. The fusion module uses an MLP with Gumbel-Softmax and a straight-through estimator to predict a per-query neighborhood size k, then performs bidirectional symmetric retrieval between lidar and image features. The rendering module initializes Gaussians from fused features and supervises them with photometric consistency against the input images, plus a Gaussian parameter consistency loss. Experiments on nuScenes and SemanticKITTI report a 28.4% mIoU on nuScenes, surpassing Co-Occ by 1.8%, with ablations of the two modules and an efficiency analysis.","tokens_in":15559,"tokens_out":3926,"duration_ms":37236,"significance":"If the reported numbers are stable, the paper demonstrates a new state of the art for multi-modal 3D occupancy prediction and provides a plausible mechanism for scale-adaptive fusion. The strengths include the two-benchmark evaluation, the component ablations, and the explicit efficiency comparison. However, the central adaptive-fusion claim is currently under-supported: the missing fixed-k=4 control and the scene-level aggregation in Tables 4-6 leave open the possibility that the gains are due to a larger maximum neighborhood rather than target-adaptive behavior. The method description also contains notational inconsistencies that impede reproducibility.","major_comments":[{"comment":"The adaptive-fusion claim is not isolated from the maximum neighborhood size. Table 4 compares fixed k=1, 2, 3 with dynamic ranges 1-2, 1-3, and 1-4, but never evaluates fixed k=4. Since the dynamic ranges include k=4, the improvement of dynamic 1-4 over fixed k=3 (mIoU 28.4 vs 26.7, IoU 41.8 vs 38.2) could reflect simply a larger maximum neighborhood, not target-adaptive selection. Please add a fixed k=4 baseline and report per-query k statistics to separate the two effects.","section":"Sec. 4.4, Table 4"},{"comment":"The scale-perception mechanism is supported only by scene-level aggregate k distributions. Tables 5 and 6 report k frequencies across scenes labeled by dominant target size, which confounds target scale with scene density, object count, and average distance; no per-object or per-voxel correlation between k and object size is provided. The abstract-level claim that neighborhoods expand for large targets and shrink for small ones is therefore not established at the claimed granularity. Please provide per-instance statistics, such as average k as a function of object size or distance.","section":"Sec. 4.4, Tables 5-6"},{"comment":"Equations (1)-(4) are not stated rigorously enough to reproduce the fusion module. Equation (2) mixes a forward definition y_ste = e_k with a backward definition z_c = GumbelSoftmax(z) inside a single expression for ∂L/∂z, and the notation ∇_z L versus ∂L/∂z is inconsistent. Equation (3) defines ω_i, but Eq. (4) uses ω_I and ω_L without specifying how these weights are derived from the key matrix or how padding and the linear mapping are applied. Please rewrite the equations with consistent notation and explicit tensor dimensions.","section":"Sec. 3.1, Eqs. (1)-(4)"},{"comment":"The empirical claims, including the headline 28.4% mIoU and all ablation deltas, are reported as single runs without code or error bars. Given the small margins over Co-Occ (1.8% mIoU) and the absence of code, the reader cannot assess whether the observed gains are stable. Please report means and standard deviations over at least three seeds for the main results and the key ablations, and release code and models to support the SOTA claim.","section":"Sec. 4.3, Table 1 and Sec. 4.4"}],"minor_comments":[{"comment":"The dataset name is misspelled as 'SematicKITTI' in Table 2 and the accompanying text; it should be 'SemanticKITTI'.","section":"Throughout"},{"comment":"The 'Fix k' row in Table 3 does not specify which fixed k value is used; please state this explicitly so the comparison with the dynamic strategy can be audited.","section":"Sec. 4.4, Table 3"},{"comment":"The total loss in Eq. (8) switches between Lrgb and a weighted combination of Locc and Lpc, but the training schedule for this switch is not specified. Please clarify how many iterations are spent on rendering optimization before the occupancy and parameter-consistency losses become active.","section":"Sec. 3.3, Eq. (8)"},{"comment":"The citation 'BEVFormer Li et al.' lacks a year and venue; the reference entry is also incomplete. Please correct this and the typo 'V oxformer' in the reference list.","section":"Sec. 2 and References"},{"comment":"Figures 4 and 5 are not referenced or described in the text beyond their captions; please add a sentence explaining what they show and how they support the k-distribution analysis.","section":"Sec. 4.4, Figures 4-5"},{"comment":"The limitation that extreme-sized objects require adjusting the candidate k range is directly relevant to the adaptive-fusion claim; please discuss whether this indicates a sensitivity to the maximum k and how it interacts with the missing fixed-k=4 control.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent empirical systems paper with a plausible contribution, but the central adaptive-k claim needs a fixed-k=4 control and per-object evidence; without these, the SOTA claim is not fully substantiated. The revision must address the ablation gap and the equation clarity. I do not recommend rejection, but the load-bearing empirical evidence should be strengthened before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dynamic per-query k selection is the genuinely new piece here. Prior fusion work uses fixed neighborhoods or global channel attention; using Gumbel-Softmax with a STE to pick a neighborhood size per query is a real departure, and the 3DGS renderer with multi-modal initialization is a sensible upgrade over Co-Occ's NeRF-style regularizer. Credit where due: the ablations are internally consistent, each module adds mIoU, and the largest gains are on traffic cones and motorcycles, which fits the small-object story. The paper also reports SemanticKITTI results, range-based breakdowns, and latency, which is more than many occupancy papers bother with.\n\nNow the soft spots. The load-bearing issue, as the stress-test note says, is that Table 4 never evaluates fixed k=4. The dynamic candidate set includes k=4, and in large-target scenes the model selects k=4 nearly 40% of the time. So the dynamic 1-4 improvement over fixed k=3 could be nothing more than allowing a larger maximum neighborhood. Without a fixed k=4 control, or better, per-query k correlated with object size rather than scene-level aggregates, the 'scale-adaptive' claim is not established at the granularity the abstract states. This is fixable, but it is the core of the paper, so it matters. The STE equations (Eq. 1-2) are also sloppy; the forward/backward definitions bleed into each other. The photometric loss reuses the same input images that produced the image features, so calling it 2D-3D consistency is generous; it is self-supervision. And the evaluation is a single run on the validation set with no error bars and no code release.\n\nWho is this for? Anyone working on multi-modal occupancy or cross-modal fusion. It is a plausible SOTA result, not a discredited one. The central mechanism deserves scrutiny, not dismissal. I would send it to review, but request the missing fixed-k baseline, per-object analysis, and a cleaner presentation of the STE trick before it comes back.","headline":"The per-query k selection is a real novelty, but the missing fixed k=4 baseline blurs the central adaptive-fusion claim; still worth a serious referee's time.","tokens_in":16160,"tokens_out":2724,"would_cite":true,"duration_ms":27843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TACOcc claims that per-query adaptive fusion neighborhoods plus 3D Gaussian volume-rendering supervision raise 3D occupancy mIoU to 28.4%, 1.8% above the prior multi-modal state of the art.","keywords":["3D semantic occupancy prediction","multi-modal fusion","adaptive neighborhood selection","3D Gaussian Splatting","volume rendering supervision","lidar-camera fusion","autonomous driving","nuScenes"],"falsifier":"Compute, on the nuScenes validation set, the distribution of selected k for each semantic instance or object-size bin while controlling for point density, distance, and neighboring-object count; if small objects do not systematically receive small k and large objects large k under those controls, the target-scale adaptation claim fails. A simpler check is to shuffle the predicted k values among queries with matched density and distance and compare mIoU with the learned assignment: if a random or density-based assignment matches it, the learned mechanism is not doing the claimed work.","tokens_in":15135,"feed_emoji":"🚗","tokens_out":7968,"duration_ms":73363,"temperature":0.7,"pith_summary":"This paper is trying to show that two weaknesses in multi-modal 3D occupancy prediction, fixed cross-modal fusion neighborhoods and sparse supervision on object surfaces, can be fixed together. The proposed framework, TACOcc, lets each query voxel choose its own fusion neighborhood size: a learned module picks one of four candidate k values, so large objects aggregate more context and small objects avoid noise. It then feeds the fused features into a 3D Gaussian Splatting volume-rendering pipeline that renders multi-view images, applies photometric and Gaussian-parameter consistency losses, and pushes 2D detail back into the 3D features. On the nuScenes validation benchmark the authors report 28.4% mIoU, surpassing the previous best multi-modal method by 1.8%, with the largest per-class gains on small, sparse objects like traffic cones and motorcycles. If true, this would make adaptive neighborhood selection and rendering-based self-supervision a scalable recipe for occupancy perception in autonomous driving.","feed_headline":"Occupancy prediction reaches 28.4% mIoU with adaptive fusion","feed_subtitle":"Per-query neighborhood sizes and Gaussian volume rendering lift small-object accuracy 1.8% over prior multi-modal best.","key_machinery":"The load-bearing mechanism is a per-query dynamic neighborhood selector fused into a bidirectional symmetric retrieval unit. An MLP maps each non-zero query voxel's feature to logits over candidate k values; Gumbel-Softmax makes the discrete choice differentiable and a straight-through estimator keeps gradients flowing through the rounding or argmax, so the model can learn to widen neighborhoods for large objects and narrow them for small ones. The second mechanism is a multi-modal 3D Gaussian Splatting volume-rendering pipeline: fused features initialize Gaussian primitives, differentiable splatting renders six viewpoints, and photometric loss Lrgb plus Gaussian parameter consistency loss Lpc create a 2D-to-3D supervision path intended to recover surface detail lost to sparse annotations.","core_discovery":"TACOcc's central claim is that target-scale-adaptive bidirectional retrieval plus volume-rendering supervision is a better way to fuse lidar and image features for dense 3D semantic occupancy prediction. Instead of using a fixed number of neighboring voxels when matching geometric lidar features with semantic image features, the model learns a per-query k in {1,2,3,4} via a Gumbel-Softmax relaxation with a straight-through estimator, then performs symmetric K-nearest-neighbor retrieval in both directions: image queries retrieve lidar keys and lidar queries retrieve image keys. The volume-rendering module initializes 3D Gaussian primitives from the fused features, using both point-cloud coordinates and non-zero voxel centers, renders six camera views, and supervises with an L1 plus D-SSIM photometric loss and a Gaussian parameter consistency loss. With this dual module, TACOcc reports 41.8% IoU and 28.4% mIoU on nuScenes, 1.8 percentage points above Co-Occ, and improves traffic cone and motorcycle IoU by about 3 points; on SemanticKITTI it reports 24.7% mIoU, better than leading lidar-only methods that use extra segmentation supervision.","pith_inferences":["Editorial inference: the paper's k-distribution evidence is scene-level aggregates, so the claim that k tracks object scale would be strengthened by per-object measurements; if k instead tracks scene density or distance, a simpler density- or distance-based schedule might reproduce the gains.","Editorial inference: the 3D Gaussian Splatting volume-rendering module may be a plug-and-play supervision add-on for other occupancy predictors, since it only needs fused features and images; the paper does not test this separation.","Editorial inference: larger gains on small, sparse classes suggest rendering supervision could reduce dependence on dense 3D labels for rare classes, an annotation-cost saving the paper does not quantify.","Editorial inference: the same adaptive-neighborhood idea could transfer to other point-image fusion tasks such as 3D detection or BEV map segmentation, where fixed-radius neighbors cause the same mismatch."],"forward_implications":["If the central claim holds, dynamic k gives a better accuracy-latency trade-off than fixed neighborhoods: fixed k=3 reaches 26.7% mIoU at 0.65 s, while dynamic k in {1,2,3,4} reaches 28.4% mIoU at 0.64 s.","The reported k distributions imply the fusion module automatically specializes: in small-target-dominant scenes 60.1% of queries choose k=1 or 2, while in large-target-dominant scenes 70.3% choose k=3 or 4.","Volume-rendering supervision is not just cosmetic: adding Lrgb then Lpc to the adaptive fusion baseline raises mIoU from 26.9% to 27.1% and then to 28.4% in the ablation table.","If correct, TACOcc provides a template for combining adaptive cross-modal matching with rendering-based self-supervision on any occupancy backbone, not only lidar-camera inputs.","The range-dependent evaluation shows the method improves over Co-Occ at 25 m, 50 m, and 100 m, suggesting the gains are not confined to near-field objects."],"supporting_citations":[{"why":"Co-Occ, the prior multi-modal state of the art and the main baseline TACOcc must beat; its fixed-neighborhood fusion is the design the adaptive module replaces.","marker":"Pan et al. [2024a]"},{"why":"OpenOccupancy supplies the voxelized occupancy benchmark and the cascaded occupancy head/decoder used for final predictions.","marker":"Wang et al. [2023]"},{"why":"3D Gaussian Splatting provides the differentiable splatting renderer that the volume-rendering module builds on.","marker":"Kerbl et al. [2023]"},{"why":"Gumbel-Softmax reparameterization makes discrete k selection differentiable for the adaptive fusion module.","marker":"Jang et al. [2016]"},{"why":"Straight-Through Estimator supplies the surrogate gradient that lets gradients flow through the rounding and argmax in k selection.","marker":"Yin et al. [2019]"},{"why":"SurroundOcc defines the nuScenes-based occupancy labels used as ground truth and provides a camera-only baseline.","marker":"Wei et al. [2023]"},{"why":"nuScenes dataset provides the surround-view lidar and camera inputs and the validation split for the main experiments.","marker":"Caesar et al. [2020]"},{"why":"SemanticKITTI test set is the second benchmark used to demonstrate cross-dataset generalization.","marker":"Behley et al. [2019]"},{"why":"Supplies the photometric rendering loss formulation (L1 plus D-SSIM) and rendering-supervision strategy reused in Lrgb.","marker":"Fu et al. [2024]"},{"why":"2D-to-3D view transformer used to project image features into the same 3D space as lidar voxels.","marker":"Philion and Fidler [2020]"}],"fun_headline_variants":["Adaptive fusion plus Gaussian rendering lifts occupancy to 28.4% mIoU","Target-adaptive fusion and volume rendering boost small-object IoU by 3","Per-query neighbor sizes adapt fusion for 3D occupancy","Gaussian volume rendering sharpens 3D occupancy with adaptive fusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the network's learned per-query neighborhood size responds to the scale of the object being fused, not to scene density, distance, occlusion, or class frequency, because the paper's evidence for target-scale adaptation is scene-level k statistics rather than per-object measurements.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive fusion plus Gaussian rendering lifts occupancy to 28.4% mIoU","Target-adaptive fusion and volume rendering boost small-object IoU by 3","Per-query neighbor sizes adapt fusion for 3D occupancy","Gaussian volume rendering sharpens 3D occupancy with adaptive fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3122,"prompt_tokens":1030,"completion_tokens":2092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":646,"tokens_out":2092,"duration_ms":15574,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:28:26.900510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on the nuScenes validation set, the distribution of selected k for each semantic instance or object-size bin while controlling for point density, distance, and neighboring-object count; if small objects do not systematically receive small k and large objects large k under those controls, the target-scale adaptation claim fails. A simpler check is to shuffle the predicted k values among queries with matched density and distance and compare mIoU with the learned assignment: if a random or density-based assignment matches it, the learned mechanism is not doing the claimed work.","supporting_citations":[{"cited_title":"Ssc-rs: Elevate lidar semantic scene completion with representation separation and bev fusion","cited_arxiv_id":null,"evidence_quote":"OpenOccupancy supplies the voxelized occupancy benchmark and the cascaded occupancy head/decoder used for final predictions."}],"review_version":1}