{"id":"4662ec82-e958-4b51-8311-e39319f9e7f0","arxiv_id":"2412.05548","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Replacing 3D object trackers with a 2D foundation model plus LiDAR and a motion-learning correction produces state-of-the-art street-scene reconstructions without ground-truth object poses.","lead":"This paper builds a 3D driving-scene reconstruction system that tracks moving vehicles with a 2D image tracker and LiDAR instead of expensive 3D object trackers or hand-labeled poses. It uses a learned motion model to fix tracking mistakes and produces sharper novel views on Waymo and KITTI than several prior systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central robustness claim — that HexPlane motion learning corrects 2D-tracker trajectory errors and recovers missed detections — is not directly tested; the motion-loss schedule in Eq.","rationale":"The reader's conditional verdict is well aligned with my reading. I focused on the single most load-bearing technical uncertainty: whether the motion-learning module actually delivers the robustness that distinguishes this method from pose-dependent baselines. The paper's own ablation F (GT pose, 28.98 PSNR vs 28.85 for the full method) is encouraging evidence that the estimated trajectories are close to GT on this benchmark, but it does not measure how the system behaves under the tracker failures the paper claims to handle. A system that works well because its 2D tracker happens to be accurate is different from a system that corrects tracking errors, and the paper's wording asserts the latter. The proposed corruption experiment would settle this directly: it isolates error correction from trajectory quality, uses GT poses from Waymo as the reference, and provides a quantitative threshold for the claim. I did not treat the KITTI PSNR overclaim (lower PSNR than Street Gaussians with GT pose) as load-bearing, because it is a presentation issue that can be fixed by qualifying 'outperforms' and does not affect the validity of the core method. The lack of error bars, code, and the human-motion limitation are real but secondary; they do not change the conditional verdict. My concern is therefore consistent with the reader's weakest_assumption, and I recommend no change to the CONDITIONAL verdict.","tokens_in":20352,"tokens_out":6194,"duration_ms":64342,"concrete_test":"On the Waymo-NOTR dynamic32 split, compute per-frame translation/rotation error between the estimated Tt (Eq. 4) and GT object poses for every object; report mean/median and the distribution of missed-frame gaps. Then rerun the full pipeline with corrupted trajectories: (a) add Gaussian noise to Tt at levels 0.1m/1°, 0.5m/5°, 1m/10°; (b) drop 10% and 30% of randomly chosen frames (treat as missed). Plot DPSNR/DSSIM vs corruption level, with the GT-pose ablation (Table 3F) as upper bound. If DPSNR at 0.5m/5° noise or 30% drop stays within ~0.5 dB of the clean trajectory result, the error-correction claim holds; if it falls by several dB, the claimed robustness is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that replacing 3D trackers with 2D tracker + LiDAR + ICP is robust because motion learning in HexPlane 'autonomously corrects trajectory errors and recovers missed detections.' This robustness is the load-bearing part of the contribution: if learned motion cannot absorb realistic tracking failures, the method reduces to a noisy-pose Street Gaussians and the generalization argument collapses. The paper provides only indirect evidence. Ablation B vs H shows removing Lmotion hurts DPSNR by 1.04 dB, but Lmotion also provides the initial motion prior; this does not isolate error correction. No comparison of the estimated trajectory Tt against GT poses is reported in the main text (the supplementary shows only clipped error distributions), and no experiment measures degradation under controlled trajectory noise or missed-detection gaps. Because Lmotion forces ΔXt ≈ TtX − X for the first 40% of iterations, any systematic error in Tt is baked into the HexPlane features before photometric loss can act; whether photometric refinement can then recover depends on error magnitude, and the paper gives no threshold. The claim 'recover missed detections' is especially unsupported: frames with <10% overlap are discarded, and if a vehicle is unobserved for many frames, the HexPlane must interpolate across a gap with no supervision, on a representation trained partly on wrong targets. Without quantifying typical tracker failure statistics on Waymo-NOTR/KITTI, the headline result could be an artifact of favorable tracking rather than of the proposed correction mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing 3D object trackers in dynamic street-scene Gaussian splatting with a pipeline built on a 2D foundation-model tracker. Vehicles are tracked in 2D with GLEE, lifted to 3D by projecting LiDAR points through segmentation masks, associated across cameras, and accumulated into a canonical point cloud via incremental ICP alignment, which also yields per-frame object poses. These poses are used to supervise per-point deformations predicted by a HexPlane network for the first 40% of training iterations, after which the motion loss is removed and photometric and geometric losses dominate. The method is evaluated on Waymo-NOTR and KITTI for scene reconstruction and novel view synthesis, with ablations targeting the tracking module, the motion loss, the isotropic marble representation, and the choice of 2D tracker.","tokens_in":20553,"tokens_out":5516,"duration_ms":51589,"significance":"The practical motivation is strong: avoiding 3D trackers and ground-truth object poses would broaden the applicability of dynamic street-scene reconstruction, and the use of 2D foundation models is a reasonable route toward better domain generalization. The paper also reports useful ablations, a comparison against rigid-transformation motion modeling, an analysis of tracking error distributions, and explicit failure cases in the supplementary material. If the central robustness claim were quantitatively established, the contribution would be significant. However, the headline claim of outperforming existing methods is not uniformly supported by the KITTI results, and the load-bearing claim that the HexPlane motion learning corrects trajectory errors and recovers missed detections is currently supported only indirectly.","major_comments":[{"comment":"The abstract and the contribution list state that the method \"outperforms existing approaches\" on Waymo-NOTR and KITTI, but Table 2 shows KITTI-75% PSNR 25.49 versus 25.79 for Street Gaussians (GT pose) and KITTI-50% PSNR 25.11 versus 25.52. The method wins on SSIM and LPIPS in both settings but loses on PSNR. Please qualify the claim, for example by saying \"competitive on KITTI with improved SSIM/LPIPS\" or by providing an aggregate or per-scene analysis that justifies a global superiority statement. As written, the empirical headline is overstated.","section":"Abstract; §4.2, Table 2"},{"comment":"The central claim that motion learning \"autonomously corrects trajectory errors and recovers missed detections\" is not directly tested. Ablation B versus H removes the motion loss entirely, causing a 1.04 dB drop in DPSNR; this demonstrates the value of the initial motion prior, but it does not isolate the error-correction effect. Because Lmotion is active for the first 40% of iterations, any systematic error in Tt is imprinted into the HexPlane features before photometric loss can act. Please add a controlled experiment in which the estimated trajectory Tt is corrupted by known translation/rotation noise, or in which synthetic missed-detection gaps of varying length are introduced, and report final DPSNR as a function of the noise level or gap length. Please also report the fraction of frames rejected by the 10% overlap rule and the typical missed-detection statistics on the test scenes. Without such evidence, the \"recovers missed detections\" statement remains unsupported.","section":"§3.4, Eq. (7); §4.3, Table 3"},{"comment":"The 30%/10% overlap thresholds and the 50-point cross-camera association rule are introduced without sensitivity analysis. A misclassified frame either corrupts the canonical model with a misaligned partial point cloud or discards a useful observation, and since the canonical model provides both the initial Gaussian positions and the trajectory Tt used by Eq. (7), these thresholds directly affect the downstream motion prior. Please report the distribution of τoverlap on the evaluated scenes and a sensitivity study over the threshold values, or otherwise justify these choices with data.","section":"§3.3, Eq. (4)"}],"minor_comments":[{"comment":"The notation is ambiguous: after defining Tt = ICP(Ot, O), the overlap is written as O ∩ Tt^{-1}Ot. If Tt aligns Ot to O, the overlap should be O ∩ Tt Ot unless Tt is defined with the opposite convention. Please clarify the transformation direction.","section":"§3.3, Eq. (4)"},{"comment":"The caption contains a typo: \"Street Guassians\" should be \"Street Gaussians\".","section":"Table 2"},{"comment":"No error bars or standard deviations are reported. Since the Waymo-NOTR dynamic32 subset has only 32 scenes, reporting per-scene variance or confidence intervals would strengthen the comparisons.","section":"Tables 1–3"},{"comment":"The depth loss in Eq. (8) is said to compare rendered depth with LiDAR-estimated depth, but the normalization distance d = 80 appears only in the supplementary Eq. (12). Please state this normalization in the main text for reproducibility.","section":"§3.5 and Supplementary §12"},{"comment":"The overview figure is dense and many labels are hard to read at normal print size. Please enlarge the pipeline labels or split the figure into two panels.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper does something concrete and useful: it removes the 3D-tracker dependency from Street Gaussians by using a 2D foundation model (GLEE) plus LiDAR projection and ICP to build per-object trajectories, and it supervises HexPlane motion with those trajectories only during the first 40% of training, then lets photometric loss refine. That's a legitimate extension of S3Gaussian/Street Gaussians, and the Waymo-NOTR numbers are consistently stronger than the baselines, including a fair comparison where Street Gaussians uses nuScenes-pretrained 3D tracker weights. The ablations are informative: removing the tracking module or the motion loss hurts, swapping the 2D tracker to UNINEXT keeps performance stable, and the GT-pose oracle (F) is only slightly above their system. The KITTI results are more nuanced—PSNR is 0.3 dB below Street Gaussians with GT poses—but they beat it on SSIM and LPIPS, and they say so in the text. The abstract's blanket 'outperforms existing approaches' overreaches; that should be fixed.\n\nThe soft spots are real but not fatal. No error bars or multiple seeds, so small metric gaps are unverified. More importantly, the central robustness claim—that HexPlane motion learning autonomously corrects trajectory errors and recovers missed detections—is not directly measured. Ablation B vs H shows the motion loss matters, but that loss also supplies the initial motion prior, so it doesn't isolate error correction. There's no controlled-noise experiment, no comparison of the estimated trajectory against GT poses in the main text, and no quantification of how much tracking error the system can absorb. Frames with less than 10% overlap are discarded, so recovery of genuinely missed detections has no direct evidence. The supplementary Table 7 (rigid transform vs HexPlane on the same 2D trajectories) is suggestive, but a referee should demand the direct test.\n\nThe paper also doesn't release code or configs, so the numbers can't be checked independently. Human motion modeling is acknowledged as weak in the limitations section, which matters, but the paper is honestly scoped to vehicles.\n\nNet: this is a solid empirical systems paper. It deserves a serious referee; the requests should be for the trajectory-noise experiment, error bars, and code. I'd bring it up at a reading group as a useful data point on how far 2D foundation models can go in 3D reconstruction.","headline":"A solid empirical extension of Street Gaussians/S3Gaussian that replaces 3D trackers with a 2D foundation model plus LiDAR/ICP and a scheduled motion loss, with strong Waymo-NOTR results but an under-tested claim of autonomous error correction.","tokens_in":21224,"tokens_out":3739,"would_cite":true,"duration_ms":32929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Street scenes reconstruct without 3D object trackers.","keywords":["street scene reconstruction","3D Gaussian Splatting","2D object tracking","LiDAR point cloud","HexPlane","motion learning","novel view synthesis","autonomous driving"],"falsifier":"Take a Waymo sequence with known vehicle poses, run the pipeline with the 2D tracker's masks, and compare the estimated trajectory $T_t$ against the ground-truth poses frame by frame. The central claim would be falsified if frames accepted by the 30%/10% overlap rule systematically carry translation errors above a few tens of centimeters, or if removing the motion loss after 40% of training fails to improve dynamic-region rendering on occluded frames compared with keeping trajectory supervision throughout.","tokens_in":20029,"feed_emoji":"🚗","tokens_out":10311,"duration_ms":85721,"temperature":0.7,"pith_summary":"The paper aims to show that dynamic street-scene reconstruction does not need 3D object trackers or ground-truth object poses. It builds vehicle trajectories by projecting 2D segmentation masks from a general-purpose 2D tracker onto LiDAR points and aligning the partial clouds frame-by-frame with ICP into a canonical model. Those trajectories supervise a HexPlane-based motion field only during early training; afterwards the field is free to correct pose errors and interpolate through missed detections. The authors report that this pipeline outperforms existing pose-based and self-supervised methods on Waymo-NOTR and KITTI without any tracking annotations. If correct, it removes the main annotation bottleneck for turning raw driving logs into editable, re-renderable 3D scenes.","feed_headline":"Street scenes reconstruct without 3D object trackers","feed_subtitle":"A 2D foundation-model tracker and LiDAR build the trajectories; a learned motion field then fixes its errors.","key_machinery":"The load-bearing machinery is a two-stage trajectory-to-motion pipeline. First, 2D segmentation masks $M$ from a general-purpose tracker are re-projected onto LiDAR to obtain per-view object clouds $O_t^{(j)}$, and ICP aligns each successive partial cloud to an incrementally built canonical model $O$; overlap thresholds decide whether a frame updates the model (above 30%), is recorded as pose only (10-30%), or is discarded as a failed detection (below 10%). Second, dynamic Gaussians are deformed through a HexPlane feature grid: six learnable planes ($xy$, $xz$, $yz$, $xt$, $yt$, $zt$) encode spatial-temporal features, and decoders predict per-point motion $\\Delta X_t$ and color change $\\Delta C_t$. The motion loss $\\mathcal{L}_{\\text{motion}} = \\mathrm{avg}_{X \\in O} |\\Delta X_t - (T_t X - X)|$ supervises the field with the tracker-derived trajectory for the first 40% of iterations, then is removed so the implicit field can smooth over and correct tracker errors. Isotropic Gaussian marbles keep dynamic points simple so all deformation is carried by motion and color, making the learned motion the main carrier of the argument.","core_discovery":"The central claim is that a 3D object tracker is unnecessary for high-quality dynamic street reconstruction. The paper's replacement is a stable object-tracking module that lifts 2D tracks to 3D, plus a motion-learning module that uses those tracks as soft supervision rather than rigid truth. The key move is to treat the trajectory-derived pose $T_t$ as a temporary teacher: the model learns per-point motion $\\Delta X_t$ in HexPlane feature space, is pushed toward the teacher for the first 40% of training, and is then free to deviate, which lets it correct pose errors and recover missed detections. The paper reports that on Waymo-NOTR this beats Street Gaussians (3D-tracker-based) and S3Gaussian (self-supervised), and on KITTI it matches pose-supervised Street Gaussians on PSNR while exceeding it on SSIM and LPIPS.","pith_inferences":["Editorial inference: the same 2D-mask-plus-ICP canonicalization could be applied to other dynamic classes such as pedestrians, where per-point deformation would handle articulated motion more naturally than rigid poses.","Editorial inference: the 40% supervision-removal schedule suggests a broader principle, that explicit pose priors work best as an initial curriculum for implicit motion fields, which could transfer to other dynamic reconstruction settings.","Editorial inference: because the method relies only on 2D tracking and LiDAR, it could in principle reconstruct scenes from any driving dataset with those sensors, even ones with no 3D annotations; the Waymo and KITTI experiments support but do not prove that."],"forward_implications":["New street scenes can be reconstructed from raw multi-view video plus LiDAR with no manual object poses and no 3D tracker, needing only a pretrained 2D tracker and sensor calibration.","Tracking errors no longer break rendering: frames discarded by the 10% overlap rule simply drop out of trajectory supervision, and the motion field can interpolate through them.","Because motion is per-Gaussian rather than a single rigid pose, within-object non-rigid changes such as open doors are representable.","Novel views can be rendered at arbitrary timestamps, including frames where the object was never successfully tracked, by querying HexPlane features at the desired time.","The method's performance is stable across two different 2D trackers (GLEE and UNINEXT), indicating the pipeline does not hinge on one specific tracker."],"supporting_citations":[{"why":"Supplies the 2D foundation-model tracker whose segmentation masks seed all 3D vehicle trajectories.","marker":"[84]"},{"why":"Segments scenes into static and dynamic parts so vehicles can be isolated before lifting to LiDAR.","marker":"[12]"},{"why":"Provides the HexPlane feature representation that stores and interpolates per-point motion in space and time.","marker":"[7]"},{"why":"The 4DGS deformation framework whose HexPlane decoders predict motion and color changes.","marker":"[81]"},{"why":"Gives isotropic Gaussian marbles, the dynamic primitives whose simplicity lets motion be carried purely by point displacement.","marker":"[74]"},{"why":"The 3D Gaussian Splatting base renderer and optimization scheme the whole pipeline builds on.","marker":"[37]"},{"why":"The 3D-tracker-based Street Gaussians baseline that the paper argues is limited by tracker generalization.","marker":"[91]"},{"why":"The self-supervised S3Gaussian baseline showing what happens when motion learning has no explicit trajectory guidance.","marker":"[32]"}],"fun_headline_variants":["2D tracks replace 3D object trackers in street scenes","Street Gaussians go tracker-free via 2D foundation models","Lifting 2D tracks to 3D for dynamic street reconstruction","Soft supervision from 2D trackers beats rigid 3D poses","No 3D tracker needed: motion learning fixes trajectory errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2D segmentation masks and the camera-LiDAR calibration are accurate enough that frame-by-frame ICP builds a clean canonical model of each vehicle; if masks are wrong or calibration is off, the trajectory loss teaches the motion field the wrong motion and the overlap thresholds cannot tell a real tracking failure from a bad alignment.","fun_headline_variants_meta":{"raw":{"variants":["2D tracks replace 3D object trackers in street scenes","Street Gaussians go tracker-free via 2D foundation models","Lifting 2D tracks to 3D for dynamic street reconstruction","Soft supervision from 2D trackers beats rigid 3D poses","No 3D tracker needed: motion learning fixes trajectory errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1476,"prompt_tokens":918,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":467}},"tokens_in":534,"tokens_out":558,"duration_ms":4301,"temperature":1.0,"reasoning_tokens":467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:36:31.513663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a Waymo sequence with known vehicle poses, run the pipeline with the 2D tracker's masks, and compare the estimated trajectory $T_t$ against the ground-truth poses frame by frame. The central claim would be falsified if frames accepted by the 30%/10% overlap rule systematically carry translation errors above a few tens of centimeters, or if removing the motion loss after 40% of training fails to improve dynamic-region rendering on occluded frames compared with keeping trajectory supervision throughout.","supporting_citations":[{"cited_title":"General object foundation model for images and videos at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the 2D foundation-model tracker whose segmentation masks seed all 3D vehicle trajectories."},{"cited_title":"4d gaussian splatting for real-time dynamic scene rendering","cited_arxiv_id":null,"evidence_quote":"The 4DGS deformation framework whose HexPlane decoders predict motion and color changes."},{"cited_title":"Street gaussians: Modeling dynamic urban scenes with gaussian splatting","cited_arxiv_id":null,"evidence_quote":"The 3D-tracker-based Street Gaussians baseline that the paper argues is limited by tracker generalization."}],"review_version":1}