{"id":"68400b16-8f90-4029-9f2c-cde213ebb112","arxiv_id":"2412.20881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fusing depth features into Mask2Former with dynamic weighting, plus location-aware and time-aware queries, improves image and video panoptic segmentation on Cityscapes while avoiding video-specific losses.","lead":"Adding depth data from stereo or simulated LiDAR to a standard Mask2Former image segmentation model improves panoptic segmentation, and two small query changes allow video panoptic segmentation to approach video-supervised models without video-specific training. This matters for autonomous vehicles, which already carry cameras and LiDAR, because it could reduce the need for costly video annotation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'without video training' claim is contradicted by the §4.5 fine-tuning on Cityscapes-VPS; the paper never reports VPS results without that fine-tuning.","rationale":"The reader flagged simulated LiDAR as the weakest assumption, which is a legitimate external-validity concern: the LiDAR results are generated by downsampling stereo disparity, so real LiDAR noise and sparsity patterns may alter the gains. However, I consider the contradiction between the 'without video training' claim and the §4.5 fine-tuning protocol to be more load-bearing, because it directly undermines the paper's central VPS contribution and its title. Section 3.1.1 explicitly states that no video datasets are needed for training, yet §4.5 fine-tunes on Cityscapes-VPS, a video dataset. Without reporting results without that fine-tuning, the claimed 'video-free' performance is not established. This is a fixable issue: the authors could reword the claim to 'without temporal supervision' or provide the no-fine-tuning numbers. Because the reader's CONDITIONAL verdict already notes this contradiction in the rationale, my concern does not change the overall verdict; it reinforces it. I therefore recommend UNCHANGED. The simulated LiDAR issue remains secondary but worth addressing in revision.","tokens_in":8928,"tokens_out":8043,"duration_ms":76561,"concrete_test":"Evaluate the proposed final model (LiDAR, pretrained depth, LAQ+TAQ) on Cityscapes-VPS validation using only the Cityscapes-trained weights, with the same TAQ inference, and without the 5,000-iteration VPS fine-tuning. Report VPQ, VPQ_things, and VPQ_stuff, and compare with Table 2. As a control, also run the Mask2Former baseline without VPS fine-tuning. If the no-fine-tuning VPQ drops substantially (e.g., toward the 51.71 baseline), the 'without video training' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central VPS contribution is that location-aware and time-aware queries enable strong video panoptic segmentation 'without video training' (title, abstract, §3.3). Yet §4.5 explicitly states that 'all models in this section are first trained on the Cityscapes dataset and then fine-tuned on the Cityscapes-vps for an additional 5,000 iterations.' Cityscapes-VPS is a video dataset, so the reported 57.24 VPQ in Table 2 is obtained after training on the same video benchmark used for evaluation. This directly contradicts §3.1.1, which claims 'the network also does not require video datasets for training.' The paper never reports a comparison with models trained only on Cityscapes (no VPS fine-tuning), so the observed improvement from the query modifications could be confounded with the additional training on video frames. The 'without video training' claim is load-bearing for the novelty of the VPS contribution, and the experiments as described do not support it.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a LiDAR-camera fusion module for Mask2Former-based panoptic segmentation, using two separate ResNet50 backbones and a dynamic feature-weighting function, plus location-aware queries (LAQ) and time-aware queries (TAQ) for video panoptic segmentation. It claims these modifications improve image panoptic quality by up to 5 points and achieve a VPQ of 57.24 on Cityscapes-VPS without training on video data. Experiments compare the proposed model against a Mask2Former baseline and the video-supervised Video K-Net on Cityscapes and Cityscapes-VPS.","tokens_in":9110,"tokens_out":4470,"duration_ms":41098,"significance":"If the central claim held, the contribution would be valuable for autonomous driving, since video panoptic segmentation without video supervision would remove the need for expensive video annotations. The paper honestly reports that VPQ_things still lags behind Video K-Net (39.86 vs 45.0), and the architecture is simple and easy to reproduce. However, the claim is currently undermined by the experimental protocol: all VPS results in Table 2 include fine-tuning on the Cityscapes-VPS video dataset, and all LiDAR results use simulated rather than real LiDAR. These issues must be resolved before the significance of the contribution can be assessed.","major_comments":[{"comment":"The paper's central claim that the proposed model performs video panoptic segmentation 'without video training' is not supported by the experimental protocol. Section 4.5 states that 'all models in this section are first trained on the Cityscapes dataset and then fine-tuned on the Cityscapes-vps for an additional 5,000 iterations'; Cityscapes-VPS is a video dataset, so the VPQ values in Table 2 are obtained after training on the evaluation benchmark's video frames. The manuscript never reports results for models trained only on Cityscapes without this fine-tuning, so the observed VPQ gains from LAQ/TAQ could be confounded with the additional video training. The claims in the abstract, Section 3.1.1, and the conclusion must be revised or supported by experiments without VPS fine-tuning.","section":"§4.5, Table 2, Abstract, §3.1.1"},{"comment":"The claimed 'up to 5 points' improvement in image panoptic quality is based on an unfair baseline comparison. The first row of Table 1 reports the original Mask2Former with batch size 16 (PQ 61.10), while all subsequent rows use batch size 6, giving the base model PQ 57.18; the batch-size reduction alone accounts for a 3.92 PQ drop. The LiDAR fusion gain (62.12 vs 57.18) should be compared against a batch-size-matched baseline, or the paper must explicitly justify why the batch-6 baseline is the appropriate reference.","section":"Table 1"},{"comment":"All LiDAR results in the paper are obtained from simulated LiDAR, not real sensor data. Section 4.1 explains that LiDAR is mimicked by taking stereo disparity, applying angle-based downsampling to resemble a Velodyne-64, and randomly dropping 30% of rays; Section 4.4 then presents these as 'LiDAR' results. The conclusion that LiDAR-camera fusion improves panoptic segmentation may not transfer to real LiDAR sensors with different noise, sparsity, and systematic errors. The paper should either evaluate on a real LiDAR dataset (e.g., nuScenes or SemanticKITTI) or explicitly restrict the claims to simulated depth and LiDAR.","section":"§4.1, §4.4"},{"comment":"The time-aware query (TAQ) component is applied only during evaluation, as stated in Section 3.3.2 ('will only be applied during the evaluation phase while network training remains unchanged'). This means the 'video-free' model nevertheless uses temporal information from the previous frame at inference. The paper should clarify the distinction between 'no video training' and 'no video data at inference,' and should discuss the potential for error accumulation from reusing non-empty queries at test time.","section":"§3.3.2, Table 2"}],"minor_comments":[{"comment":"Section 4.5 refers to 'Table ??'; the table number must be filled in.","section":"§4.5"},{"comment":"In the Introduction, 'segmenting (e.g. sky, road, etc.)' should read 'segments' or 'stuff classes'.","section":"§1"},{"comment":"Reference [9] is cited as 'Gang et al.' but the first author is Keke Geng; the citation should be corrected.","section":"§2.3, References"},{"comment":"The description of pretraining the depth network is unclear: it says a depth image classification network was trained from scratch using estimated depth [1] from ImageNet, but reference [1] is a metric-depth estimator, not an ImageNet classification pretraining procedure; please specify the architecture and training data.","section":"§4.3"},{"comment":"The footnote in Section 1 states that 'there is no difference in our proposed model in terms of what the 3D modality is,' yet Section 4.4 treats stereo and LiDAR separately with different preprocessing; these statements should be reconciled.","section":"§1, §4.4"},{"comment":"The qualitative discussion mentions 'the second image' for the red-boxed mistakes, but the red boxes appear in the third image of the figure; verify the wording.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact workshop-style manuscript with a strong but currently overstated central claim. I believe the issues are fixable within revision: either add no-video-fine-tuning results or reframe the contribution as 'video panoptic segmentation with minimal video fine-tuning.' The simulated-LiDAR limitation should be prominently acknowledged. The novelty relative to MinVIS is incremental, but the fusion component is a useful addition to the literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nQuick take: this is a modest but useful empirical paper that trips over its own headline. The image-side fusion result is coherent and worth knowing; the video-side \"without video training\" claim does not survive Section 4.5, where every model is first fine-tuned on Cityscapes-VPS for 5,000 iterations. That is video training, and the paper never reports a no-fine-tune comparison. So treat Table 1 and Table 2 separately: Table 1 is a clean depth-fusion study, Table 2 is a warm-start-on-video study labeled as video-free.\n\nWhat is actually new: a simple gating fusion (Eq. 3-2), where a 1x1 conv on image features produces a per-location gate for depth features, applied to ResNet50 features from depth. It beats naive summation by about 1 PQ and more after pretraining the depth backbone on ImageNet depth. The location-aware query MLP and previous-frame query reuse are not wildly new in isolation, but their combination in this setup is new. The authors are also honest that their VPQ_things (39.86) still trails Video K-Net (45.0), which is more candor than many papers offer.\n\nSoft spots, in order. (1) The central VPS claim is contradicted by Section 4.5; the LAQ/TAQ gains could simply come from 5k extra iterations on the evaluation dataset's video frames. (2) The \"LiDAR\" is simulated from stereo disparity by angle-based downsampling and random ray-drop; acceptable as a proof-of-concept, but the title and abstract should say simulated LiDAR. (3) Results are single-run, no code is released, and the batch-size drop from 16 to 6 hurts the base model, so the numbers are suggestive rather than authoritative. (4) Minor: the text references \"Table ??\" in Section 4.5.\n\nThe math is simple and consistent, and there is no circularity: the learned gamma and the LAQ MLP are trained in the standard sense, not fitted to make the metric identity hold. Citations are fair; MinVIS and prior query-reuse work are credited. For a reader working on depth inputs for transformer segmenters, Table 1 is a useful data point. For the video-free VPS story, the paper as written should not be cited.\n\nRecommendation: I would not accept as-is. I would send it for major revision rather than desk reject, because the image-fusion half is solid and the video claim is fixable. The authors need to reword the claims, add a no-fine-tune baseline, release code, and report variance. With those fixes, it becomes a legitimate short-conference paper.","headline":"Modest depth-fusion results and a fixable but real overclaim: the video-free VPS story is contradicted by the paper's own fine-tuning on Cityscapes-VPS.","tokens_in":9688,"tokens_out":4011,"would_cite":false,"duration_ms":43006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing depth from LiDAR or stereo into Mask2Former with image-conditioned weighting improves PQ by up to five points and matches a video-supervised baseline in VPQ without video training.","keywords":["panoptic segmentation","LiDAR-camera fusion","video panoptic segmentation","dynamic feature weighting","location-aware queries","time-aware queries","Mask2Former","autonomous vehicles"],"falsifier":"Run the same Mask2Former training protocol on a dataset with real projected LiDAR and panoptic labels, such as nuScenes or SemanticKITTI, and compare PQ and VPQ against the paper's simulated-LiDAR results; the claim is falsified if the real-LiDAR gain over the camera-only baseline drops below the reported margin or the VPQ advantage disappears.","tokens_in":8711,"feed_emoji":"🚗","tokens_out":7596,"duration_ms":70700,"temperature":0.7,"pith_summary":"This paper tries to show that adding depth information, whether from stereo or LiDAR, to a camera-only panoptic segmentation network improves both image and video panoptic segmentation, and that the video version can rival video-supervised models without any video training. The authors build on Mask2Former and fuse depth features through a learned, image-conditioned weighting of depth features, which they report raises panoptic quality from 57.18 to 62.12 on Cityscapes. For video, they add two small query modifications—location-aware queries that predict each segment's position and time-aware queries that reuse previous-frame queries at inference—and report VPQ of 57.24, essentially matching the video-supervised Video K-Net's 57.08. The practical significance is that autonomous-vehicle perception could get better segmentation from cheap depth sensors and skip expensive video annotations.","feed_headline":"LiDAR-guided queries match video-trained panoptic segmentation","feed_subtitle":"Dynamic feature weighting plus location- and time-aware queries reach 57.24 VPQ with no video supervision.","key_machinery":"The image-side mechanism is the fusion function $\\phi(F_I, F_D) = F_I + \\sigma(\\mathrm{conv}_{1\\times1}(F_I)) \\cdot \\gamma F_D$, where $\\sigma$ is a sigmoid gate computed from image features, so the model decides per spatial location how much depth to trust. The video-side mechanisms are location-aware queries (an MLP trained with L1 loss to predict each thing segment's bounding-box center, making query matching position-sensitive) and time-aware queries (feeding only non-empty queries from the previous frame as the initial decoder queries for the current frame, applied only at evaluation). Segment association across frames is done by Hungarian matching of output queries, inherited from the video-free baseline.","core_discovery":"The central claim is that depth is a cheap, effective auxiliary modality for panoptic segmentation, and that the depth signal can be injected at the feature level without redesigning the segmentation head. Concretely, the paper reports that replacing simple feature summation with a learned gate, formed by a 1x1 convolution over image features that scales depth features before adding them, raises panoptic quality on Cityscapes from 57.18 to 62.12 when the depth branch is pretrained. For video, the same model with two query-level changes—location-aware queries that predict segment positions, and time-aware queries that carry non-empty queries from the previous frame into the current decoder at inference—reaches 57.24 VPQ on Cityscapes-VPS, effectively matching the video-supervised Video K-Net at 57.08. The authors interpret this as evidence that video-supervised training is not strictly necessary for video panoptic segmentation in autonomous driving settings.","pith_inferences":["Because the LiDAR experiments are simulations from stereo depth, the natural next test is whether the gains survive with real LiDAR noise and occlusions, where the fusion gate may need to be more conservative.","The fusion formula is architecture-agnostic, so the same dynamic weighting could be transplanted into other transformer-based or convolutional segmentation and detection heads with minimal change.","Since time-aware queries are used only at inference, the proposed video model adds no training-time cost, so the recipe is directly usable when video labels are scarce.","The stronger improvement on things than stuff suggests depth contributes mostly geometric boundary cues; one could probe this by ablating on datasets with many thin or partially occluded objects, where such cues should matter most."],"forward_implications":["Over the batch-size-6 camera-only baseline, stereo depth adds about 2.1 PQ points and simulated LiDAR depth with a pretrained depth backbone adds about 4.9 PQ points (57.18 to 62.12).","In video panoptic segmentation, depth fusion alone improves VPQ from 51.71 to 54.36, location-aware queries raise it to 55.41, and time-aware queries push it to 57.24, essentially tying the video-supervised Video K-Net's 57.08.","The gains are concentrated in thing classes such as vehicles and people, whose boundaries are geometry-defined, while stuff classes improve less.","A camera-plus-depth perception stack can approach video-supervised performance while skipping video annotation and video-specific losses.","The remaining gap to video-supervised methods is specifically in tracking thing segments (39.86 versus 45.0 VPQ for things), not in stuff segmentation."],"supporting_citations":[{"why":"Supplies the Mask2Former architecture and training recipe that all variants build on.","marker":"[5]"},{"why":"Provides the video-free query-matching baseline that the video extension follows.","marker":"[11]"},{"why":"Offers the video-supervised Video K-Net baseline whose VPQ the proposed model is compared against.","marker":"[17]"},{"why":"Provides the fast depth completion method used to densify projected LiDAR points into dense depth images.","marker":"[14]"},{"why":"Supplies the stereo depth estimation used to generate depth on Cityscapes, from which LiDAR is simulated.","marker":"[22]"},{"why":"Defines the Panoptic Quality metric used to evaluate image panoptic segmentation results.","marker":"[13]"},{"why":"Defines the Cityscapes-VPS dataset and the VPQ metric used for video panoptic segmentation evaluation.","marker":"[12]"},{"why":"Supplies the pretrained depth backbone weights that produce the best LiDAR fusion results.","marker":"[1]"}],"fun_headline_variants":["Depth fusion lifts panoptic quality 5 points, no video training","LiDAR features beat video supervision for panoptic segmentation","Video panoptic segmentation without video training, thanks to LiDAR","Learned depth gate boosts panoptic segmentation, matches video-trained","No video training? LiDAR fusion makes it possible for panoptic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole LiDAR-camera fusion claim rides on treating simulated LiDAR, created by angle-based downsampling of stereo depth, as an accurate stand-in for real LiDAR's noise, sparsity, and ray-drop; if that simulation does not transfer, the reported PQ and VPQ gains may not appear with a real sensor.","fun_headline_variants_meta":{"raw":{"variants":["Depth fusion lifts panoptic quality 5 points, no video training","LiDAR features beat video supervision for panoptic segmentation","Video panoptic segmentation without video training, thanks to LiDAR","Learned depth gate boosts panoptic segmentation, matches video-trained","No video training? LiDAR fusion makes it possible for panoptic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1080,"prompt_tokens":893,"completion_tokens":187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":101}},"tokens_in":509,"tokens_out":187,"duration_ms":2395,"temperature":1.0,"reasoning_tokens":101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:08:06.440653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Mask2Former training protocol on a dataset with real projected LiDAR and panoptic labels, such as nuScenes or SemanticKITTI, and compare PQ and VPQ against the paper's simulated-LiDAR results; the claim is falsified if the real-LiDAR gain over the camera-only baseline drops below the reported margin or the VPQ advantage disappears.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the video-free query-matching baseline that the video extension follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers the video-supervised Video K-Net baseline whose VPQ the proposed model is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the fast depth completion method used to densify projected LiDAR points into dense depth images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the stereo depth estimation used to generate depth on Cityscapes, from which LiDAR is simulated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Cityscapes-VPS dataset and the VPQ metric used for video panoptic segmentation evaluation."}],"review_version":1}